Add journalist interface API #3619
Conversation
And an initial (pytest-based) unit test
We have logic in __init__.py here to ensure that a developer does not accidentally forget to protect a route with a @login_required decorator. Instead of the decorator, we have a list of insecure views. We rework this to allow us to use a decorator for the API routes.
This is primarily for the API, but it turns out that we actually make use of abort(403) to prevent an admin from deleting themselves, so instead of seeing a Flask error page, they will see a nice error page in the style of the rest of the journalist interface.
redshiftzero
force-pushed
the
journalist-api-0.9.0
branch
2 times, most recently
from
51cd067
to
6a2b8aa
Compare
Again, mostly for the journalist API, but users who are logged into the webapp will now see a custom error page with the regular SecureDrop styling instead of the default Flask page
Covers cases where: * Source does not exist (404 response) * Star is successfully added (201 response)
HTTP DELETE will remove the star
This was a bare except, but I think instead we want to handle only itsdangerous.BadData exceptions, which is a general exception that includes a bad signature and an expired one.
We need to also have a nice error handler for method not allowed, that should return JSON for the API and an HTML page for users of the regular web application.
This is a workaround for some unimplemented features (ALTER) in SQLite, e.g. the ability to modify constraints on a column after it has been created, see: https://stackoverflow.com/questions/30378233/sqlite-lack-of-alter-support-alembic-migration-failing-because-of-this-solutio http://alembic.zzzcomputing.com/en/latest/batch.html#batch-mode-with-autogenerate miguelgrinberg/Flask-Migrate#61
Just an off by one error
(cherry picked from commit 1ce5455)
During testing, I ran into an issue where there was a failure in the following case: reverted_schema was: CREATE TABLE "sources" ( id INTEGER NOT NULL, filesystem_id VARCHAR(96), journalist_designation VARCHAR(255) NOT NULL, flagged BOOLEAN, last_updated DATETIME, pending BOOLEAN, interaction_count INTEGER NOT NULL, PRIMARY KEY (id), CHECK (flagged IN (0, 1)), CHECK (pending IN (0, 1)), UNIQUE (filesystem_id) ) and original_schema was: CREATE TABLE sources ( id INTEGER NOT NULL, filesystem_id VARCHAR(96), journalist_designation VARCHAR(255) NOT NULL, flagged BOOLEAN, last_updated DATETIME, pending BOOLEAN, interaction_count INTEGER NOT NULL, PRIMARY KEY (id), UNIQUE (filesystem_id), CHECK (flagged IN (0, 1)), CHECK (pending IN (0, 1)) ) which fails for two reasons: * The unique constraint on filesystem_id is not at the same line in the CREATE TABLE statement. * The table name is quoted in one CREATE TABLE statement, but not in the other. In order to make our tests a little more lenient in this case (and not produce spurious test failures), we should: * Compare sorted lists consisting of the lines in each CREATE TABLE statement * Strip commas and double quotes for each element in the aforementioned lists
To prevent confusion, we also rename `uuid` to `source_uuid`.
Note that Flask's send_file does include ETags by default,
but they are not hashes, so less useful for verifying downloads
were not corrupted after fetching over Tor. The ETag in Flask is:
```
rv.set_etag('%s-%s-%s' % (
os.path.getmtime(filename),
os.path.getsize(filename),
adler32(
filename.encode('utf-8') if isinstance(filename, text_type)
else filename
) & 0xffffffff
))
```
https://github.com/pallets/flask/blob/161c43649d8c362c8359e0b79aeca40c754c5b51/flask/helpers.py#L616
redshiftzero
force-pushed
the
journalist-api-0.9.0
branch
from
207b4fa
to
dc21b3c
Compare
I'm intentionally not using test_source in the test_submissions fixture as it is a bit messy/spaghetti for saving 1 LOC
|
fixed, squashed (some of them), re-pushed 🚂 |
| seconds=TOKEN_EXPIRATION_MINS * 60) | ||
| response = jsonify({'token': journalist.generate_api_token( | ||
| expiration=TOKEN_EXPIRATION_MINS * 60), | ||
| 'expiration': token_expiry.isoformat() + 'Z'}) |
There was a problem hiding this comment.
As @redshiftzero mentioned, the timezone is always in UTC, the expiration value will have sometime like '2018-07-18T13:39:34.072044Z'.
| @wraps(f) | ||
| def decorated_function(*args, **kwargs): | ||
| try: | ||
| auth_header = request.headers['Authorization'] |
There was a problem hiding this comment.
It would be nice to have a one line comment on what that header looks like after this line.
| return abort(403, 'API token not found in Authorization header.') | ||
|
|
||
| if auth_header: | ||
| auth_token = auth_header.split(" ")[1] |
There was a problem hiding this comment.
What all can be valid values in first part before the space?
| source = get_or_404(Source, source_uuid, column=Source.uuid) | ||
| utils.make_star_false(source.filesystem_id) | ||
| db.session.commit() | ||
| return jsonify({'message': 'Star removed'}), 200 |
There was a problem hiding this comment.
In my test, I removed a start from a source, I got back this reply: {'message': 'Star removed'}. After this when I am trying to get all the sources or that particular source again, I can still see 'is_starred': True for that source.
There was a problem hiding this comment.
very good catch! filing followup ticket to address
| with journalist_app.app_context(): | ||
| source = Source.query.first() | ||
| with pytest.raises(NotImplementedError): | ||
| source.public_key = 'a curious developer tries to set a pubkey!' |
There was a problem hiding this comment.
Overall this looks fantastic @redshiftzero , I did not encounter any major issues. I separated my testing in 2 parts, the app/api testing in a development VM, and upgrade testing in staging VMs.
For the application portion, I tested the authentication, rate limiting, and went through all the various methods. Everything works as expected. I've observed the following behavior:
- POST with parameters to an API endpoint generates a error:
500: no JSON object could be decoded - Request body format: When making API calls with
raw/text, I get the following error only with the message reply endpoint:please send texts in valid json. Switching to the preferredapplication/jsonworks find, but curious thatraw/textworks for other methods. - There is no logout functionality. How complex would it be to implement some fort of logout functionality? For example, attaching the token to a user's session.
For the upgrade part of the review, I provisioned staging on develop, then build debs on this branch, and vagrant provision on this API branch. The code was updated in /var/www/securedrop, the migrations were successful, I observed the UUID column in the database, and the API is accessible over the authenticated Tor hidden service.
| return user | ||
|
|
||
|
|
||
| def token_required(f): |
There was a problem hiding this comment.
Not sure if this is a good idea, but instead of calling the token_required decorator, we could use the before_request decorator (http://flask.pocoo.org/docs/1.0/api/#flask.Flask.before_request) on the login function and explicitly mark the public endpoints.
| s = TimedJSONWebSignatureSerializer(current_app.config['SECRET_KEY']) | ||
| try: | ||
| data = s.loads(token) | ||
| except BadData: |
There was a problem hiding this comment.
👍 , BadData catches all errors, including BadSignature
|
RE: @emkll
A Flask |
Sources should only be exposed to journalists when they have submitted something
There was a problem hiding this comment.
I did another round of testing and everything looks good to me 👍. I will address the comment from my previous review (invalid/empty JSON in POST returns 500 from #3619 (review))
Thanks @redshiftzero and thanks @heartsucker for the thorough review.
Status
Ready for review
Description of Changes
Fixes #1761
Changes proposed in this pull request:
Testing
Follow the docs in this branch and try to use the API. From the perspective of a user consuming this API, is anything not intuitive or could be clearer (in the docs or in the API responses itself)?
Are there endpoints that are important for an initial API that are not here?
Are there error cases that are not gracefully handled? Apologies for the lack of detailed test plan here, but exploratory testing / attempts to break this is really what this needs.
Are there security improvements that should be made?
Test database migrations
The database migrations should occur without issue and you should be able to use the API (or direct database access) to verify the UUID on the source and submission now exist.
Deployment
Will be deployed in
securedrop-app-codepackageChecklist
If you made non-trivial code changes:
If you made changes to documentation:
make docs-lint) passed locally