Add simple, long-lived API tokens

Add a /profile/developer/ page where users can generate and re-generate
their API token. This token can be used as a Bearer token in the
Authorization header in API requests instead of using one of the more
complex and short-lived JWT tokens that the client uses to authenticate
API requests.

The tokens are just randomly generated opaque strings, each one
associated with one user account. There's 0 or 1 token per user, and the
user can regenerate their token at any time. The tokens are stored in a
`token` table in the db that just maps token values to userids.

Notes:

- Our authentication policy now calls the new API token validator first
  for API requests. If this validator does not accept the token, then it
  passes it to the legacy JWT validator (which is still used by our
  client).

  The idea is that if we add more types of API token in the future, the
  authentication policy will have a list of different validator
  functions for different token types, and will try each validator in
  turn until either one of them accepts the token or it runs out of
  validators.

  The use of a type prefix string at the beginning of tokens means that
  validators can usually reject tokens without a db lookup, so we won't
  end up with one db lookup per validator.

- The new tokens always start with u"6879-". If a token sent by a user
  doesn't start with this prefix then the token validator can reject it
  out of hand, without doing a database lookup.

  An opaque number is used for this prefix because we want users to
  treat API tokens as opaque, rather than using a human-readable prefix
  that makes it obvious what type of token you're looking at. (But the type
  of the token is not "secret" in any real sense.)

  In the future it's intended that we'll have different types of tokens
  identified by different prefixes, and different types of token might
  (for example) give access to different capabilities.

- The legacy JWT tokens (still used by our own client) do not have any
  such prefix. If all other token validators (currently just the one new
  API tokens validator) reject a token, then the auth policy will fall
  back to the legacy JWT token validator.

  I think it might be possible that a JWT by random chance could begin
  with u"6879-". If that happens then the new API tokens validator would
  not reject the token based on its prefix and would do a db lookup, but
  the lookup would return nothing and the validator would then reject
  the token, and the auth policy would then move on to the legacy JWT
  validator which would accept the token. So it would be okay.

- There's no foreign key constraint from the token table's userid column
  to the user table, because the user table belongs to the app whereas
  the token table belongs to the api.

docs/api.rst

@@ -6,6 +6,29 @@ developers interested in integrating functionality from Hypothesis into their
 own applications.
 
 
+Authorization
+-------------
+
+Some of the API URLs documented below require a valid API token.
+To use these API URLs you should:
+
+1. Generate yourself an API token on your
+   `Hypothesis developer page <https://hypothes.is/profile/developer>`_
+   (you must be signed in to Hypothesis to get to this page).
+
+2. Put the API token in the ``Authorization`` header in your requests to the
+   API.
+
+**Example request**::
+
+   GET /api
+   Host: hypothes.is
+   Accept: application/json
+   Authorization: Bearer 6879-31d62c13b0099456de5379de90f90395
+
+(Replace ``6879-31d62c13b0099456de5379de90f90395`` with your own API token.)
+
+
 root
 ----
 
@@ -225,15 +248,15 @@ create
 
 .. http:post:: /api/annotations
 
-   Create a new annotation. Requires a valid authentication token.
+   Create a new annotation. Requires a valid API token.
 
    **Example request**::
 
       POST /api/annotations
       Host: hypothes.is
       Accept: application/json
       Content-Type: application/json;charset=UTF-8
-      Authorization: Bearer eyJhbGc[...]mbl_YBM
+      Authorization: Bearer 6879-31d62c1[...]0f90395
 
       {
           "uri": "http://example.com/",
@@ -265,32 +288,31 @@ create
    :param id: annotation's unique id
    :reqheader Accept: desired response content type
    :reqheader Content-Type: request body content type
-   :reqheader Authorization: JWT authentication token
+   :reqheader Authorization: the API token
    :resheader Content-Type: response content type
    :>json string id: unique id of new annotation
    :>json datetime created: created date of new annotation
    :>json datetime updated: updated date of new annotation (same as `created`)
    :statuscode 200: no error
    :statuscode 400: could not create annotation from your request (bad payload)
-   :statuscode 401: no auth token was provided
-   :statuscode 403: auth token provided does not convey "create" permissions
+   :statuscode 401: no API token was provided
+   :statuscode 403: API token provided does not convey "create" permissions
 
 
 update
 ------
 
 .. http:put:: /api/annotations/(string:id)
 
-   Update the annotation with the given `id`. Requires a valid authentication
-   token.
+   Update the annotation with the given `id`. Requires a valid API token.
 
    **Example request**::
 
       PUT /api/annotations/AUxWM-HasREW1YKAwhil
       Host: hypothes.is
       Accept: application/json
       Content-Type: application/json;charset=UTF-8
-      Authorization: Bearer eyJhbGc[...]mbl_YBM
+      Authorization: Bearer 6879-31d62c1[...]0f90395
 
       {
           "uri": "http://example.com/foo",
@@ -312,14 +334,14 @@ update
    :param id: annotation's unique id
    :reqheader Accept: desired response content type
    :reqheader Content-Type: request body content type
-   :reqheader Authorization: JWT authentication token
+   :reqheader Authorization: the API token
    :resheader Content-Type: response content type
    :>json datetime updated: updated date of annotation
    :statuscode 200: no error
    :statuscode 400: could not update annotation from your request (bad payload)
-   :statuscode 401: no auth token was provided
+   :statuscode 401: no API token was provided
    :statuscode 403:
-      auth token provided does not convey "update" permissions for the
+      API token provided does not convey "update" permissions for the
       annotation with the given `id`
    :statuscode 404: annotation with the given `id` was not found
 
@@ -329,15 +351,14 @@ delete
 
 .. http:delete:: /api/annotations/(string:id)
 
-   Delete the annotation with the given `id`. Requires a valid authentication
-   token.
+   Delete the annotation with the given `id`. Requires a valid API token.
 
    **Example request**::
 
       DELETE /api/annotations/AUxWM-HasREW1YKAwhil
       Host: hypothes.is
       Accept: application/json
-      Authorization: Bearer eyJhbGc[...]mbl_YBM
+      Authorization: Bearer 6879-31d62c1[...]0f90395
 
    **Example response**::
 
@@ -351,13 +372,13 @@ delete
 
    :param id: annotation's unique id
    :reqheader Accept: desired response content type
-   :reqheader Authorization: JWT authentication token
+   :reqheader Authorization: the API token
    :resheader Content-Type: response content type
    :>json boolean deleted: whether the annotation was deleted
    :>json string id: the unique `id` of the deleted annotation
    :statuscode 200: no error
-   :statuscode 401: no auth token was provided
+   :statuscode 401: no API token was provided
    :statuscode 403:
-      auth token provided does not convey "update" permissions for the
+      API token provided does not convey "update" permissions for the
       annotation with the given `id`
    :statuscode 404: annotation with the given `id` was not found

h/accounts/test/views_test.py

@@ -12,6 +12,7 @@
 
 from h import accounts
 from h.accounts import views
+from h.api.models.token import API_TOKEN_PREFIX
 
 
 class DummyRequest(testing.DummyRequest):
@@ -952,6 +953,80 @@ def test_post_with_valid_data_redirects(self,
         assert isinstance(result, httpexceptions.HTTPFound)
 
 
+@pytest.mark.usefixtures('models')
+class TestDeveloperController(object):
+
+    def test_get_gets_token_for_authenticated_userid(self, models):
+        request = testing.DummyRequest()
+
+        views.DeveloperController(request).get()
+
+        models.Token.get_by_userid.assert_called_once_with(
+            request.authenticated_userid)
+
+    def test_get_returns_token(self, models):
+        token = API_TOKEN_PREFIX + u'abc123'
+        models.Token.get_by_userid.return_value.value = token
+
+        data = views.DeveloperController(testing.DummyRequest()).get()
+
+        assert data.get('token') == token
+
+    def test_get_with_no_token(self, models):
+        models.Token.get_by_userid.return_value = None
+
+        assert views.DeveloperController(testing.DummyRequest()).get() == {}
+
+    def test_post_gets_token_for_authenticated_userid(self, models):
+        request = testing.DummyRequest()
+
+        views.DeveloperController(request).post()
+
+        models.Token.get_by_userid.assert_called_once_with(
+            request.authenticated_userid)
+
+    def test_post_calls_regenerate(self, models):
+        """If the user already has a token it should regenerate it."""
+        views.DeveloperController(testing.DummyRequest()).post()
+
+        models.Token.get_by_userid.return_value.regenerate.assert_called_with()
+
+    def test_post_inits_new_token_for_authenticated_userid(self, models):
+        """If the user doesn't have a token yet it should generate one."""
+        models.Token.get_by_userid.return_value = None
+        request = testing.DummyRequest(db=mock.Mock())
+
+        views.DeveloperController(request).post()
+
+        models.Token.assert_called_once_with(request.authenticated_userid)
+
+    def test_post_adds_new_token_to_db(self, models):
+        """If the user doesn't have a token yet it should add one to the db."""
+        models.Token.get_by_userid.return_value = None
+        request = testing.DummyRequest(db=mock.Mock())
+
+        views.DeveloperController(request).post()
+
+        request.db.add.assert_called_once_with(models.Token.return_value)
+
+        models.Token.assert_called_once_with(request.authenticated_userid)
+
+    def test_post_returns_token_after_regenerating(self, models):
+        """After regenerating a token it should return its new value."""
+        data = views.DeveloperController(testing.DummyRequest()).post()
+
+        assert data['token'] == models.Token.get_by_userid.return_value.value
+
+    def test_post_returns_token_after_generating(self, models):
+        """After generating a new token it should return its value."""
+        models.Token.get_by_userid.return_value = None
+        request = testing.DummyRequest(db=mock.Mock())
+
+        data = views.DeveloperController(request).post()
+
+        assert data['token'] == models.Token.return_value.value
+
+
 @pytest.fixture
 def pop_flash(request):
     patcher = mock.patch('h.accounts.views.session.pop_flash', autospec=True)
@@ -1006,3 +1081,11 @@ def mailer(request):
     module = patcher.start()
     request.addfinalizer(patcher.stop)
     return module
+
+
+@pytest.fixture
+def models(request):
+    patcher = mock.patch('h.accounts.views.models', autospec=True)
+    module = patcher.start()
+    request.addfinalizer(patcher.stop)
+    return module

h/accounts/views.py

@@ -565,6 +565,39 @@ def _user_notifications(self):
             self.request.authenticated_userid)
 
 
+@view_defaults(route_name='profile_developer',
+               renderer='h:templates/accounts/developer.html.jinja2',
+               effective_principals=security.Authenticated)
+class DeveloperController(object):
+
+    def __init__(self, request):
+        self.request = request
+
+    @view_config(request_method='GET')
+    def get(self):
+        """Render the developer page, including the form."""
+        token = models.Token.get_by_userid(self.request.authenticated_userid)
+        if token:
+            return {'token': token.value}
+        else:
+            return {}
+
+    @view_config(request_method='POST')
+    def post(self):
+        """(Re-)generate the user's API token."""
+        token = models.Token.get_by_userid(self.request.authenticated_userid)
+
+        if token:
+            # The user already has an API token, regenerate it.
+            token.regenerate()
+        else:
+            # The user doesn't have an API token yet, generate one for them.
+            token = models.Token(self.request.authenticated_userid)
+            self.request.db.add(token)
+
+        return {'token': token.value}
+
+
 def activation_email(request, user):
     """Return the data for an 'activate your account' email for the given user.
 
@@ -633,6 +666,7 @@ def includeme(config):
     config.add_route('reset_password_with_code', '/reset_password/{code}')
     config.add_route('profile', '/profile')
     config.add_route('profile_notifications', '/profile/notifications')
+    config.add_route('profile_developer', '/profile/developer')
     config.add_route('dismiss_sidebar_tutorial',
                      '/app/dismiss_sidebar_tutorial')
     config.scan(__name__)

h/api/auth.py

@@ -6,6 +6,9 @@
 from jwt import InvalidTokenError
 from pyramid import security
 
+from h import models
+from h.api.models.token import API_TOKEN_PREFIX
+
 
 def translate_annotation_principals(principals):
     """
@@ -25,7 +28,7 @@ def translate_annotation_principals(principals):
     return list(result)
 
 
-def generate_bearer_token(request, expires_in):
+def generate_jwt(request, expires_in):
     """Return a signed JSON Web Token for the given request.
 
     The token can be used in the Authorization header in subsequent requests to
@@ -60,28 +63,24 @@ def generate_bearer_token(request, expires_in):
                       algorithm='HS256')
 
 
-def userid_from_bearer_token(request):
-    """Return the userid authenticated by the given request's JWT bearer token.
+def userid_from_jwt(token, request):
+    """Return the userid authenticated by the given JWT.
 
-    If the request contains a token generated by generate_bearer_token() above
-    in its Authorization header then return the userid authenticated by the
-    token.
+    If token is a token generated by generate_jwt() above then return the
+    userid authenticated by the token.
 
-    If the request contains no token, or the token is invalid, return None.
+    If the token is not a valid JWT bearer token, return None.
 
-    :param request: the request whose Authorization header to check
-    :type request: pyramid.request.Request
+    Note: this assumes that the prefix "Bearer " has already been stripped
+    from token.
+
+    :param token: the JWT to check
+    :type request: unicode
 
-    :returns: the userid authenticated by the request's JWT or None
+    :returns: the userid authenticated by the JWT or None
     :rtype: unicode or None
 
     """
-    if 'Authorization' not in request.headers:
-        return None
-
-    # The part of the Authorization header following "Bearer " is the JWT.
-    token = request.headers['Authorization'][len('Bearer '):]
-
     try:
         return jwt.decode(token,
                           key=request.registry.settings['h.client_secret'],
@@ -90,3 +89,29 @@ def userid_from_bearer_token(request):
                           algorithms=['HS256']).get('sub')
     except InvalidTokenError:
         return None
+
+
+def userid_from_api_token(token):
+    """
+    Return the userid authenticated by the given API token.
+
+    If token is a valid API token then return the corresponding userid,
+    otherwise return None.
+
+    Note: this assumes that the prefix "Bearer " has already been stripped
+    from token.
+
+    :param token: the token to check
+    :type request: unicode
+
+    :returns: the userid authenticated by the token, or None
+    :rtype: unicode or None
+    """
+    if not token.startswith(API_TOKEN_PREFIX):
+        return None
+
+    token_obj = models.Token.get_by_value(token)
+    if token_obj:
+        return token_obj.userid
+    else:
+        return None

h/api/models/__init__.py

@@ -10,3 +10,6 @@
 direct to the submodules of this package, but rather through the helper
 functions in `h.api.storage`.
 """
+from h.api.models.token import Token
+
+__all__ = ('Annotation', 'Document', 'Token')