make file submissions dis/allowable

[wbaid]

Status

Ready for review

• See next section for tests outstanding

Status of requested changes (since b5d493c)

1. fix: "Error strings [...], and also in the inline comment
   securedrop/source_templates/lookup.html, which could be confusing
   to sources." (1, 2)

2. fix: "Bug: textarea width on Source Interface"

3. refactor: "Proposed alternative language" for "Instance Configuration" page

   1. fix: "UX issue: 'Update' button on Instance Configuration page"
   2. fix: "UX issue: 'Document Uploads', 'Allow' language on Instance
      Configuration page"

4. refactor: versioned instance_config

   1. "one configuration option per column"
   2. migration to add column sets default value
   3. "use the configuration option that has null valid_until. When we
      update a config, we store the historical configuration entry by
      setting valid_until=datetime.datetime.utcnow(), then store the new
      configuration with null valid_until."
   • test: migrations

5. test: integration test

Description of Changes

1. InstanceConfig versioned key-value store as outlined

   • load_instance_config() sets each app's app.instance_config
     (via @before_request)

2. Source interface checks
   app.instance_config.allow_document_uploads; if False:

   1. /lookup hides the file input and
   2. changes its heading to "Submit Messages" (rather than "Submit
      Files or Messages") and other strings accordingly; and
   3. /submit skips processing of request.files.
   4. /metadata returns this setting as allow_document_uploads.

3. Journalist interface:

   1. /admin/config adds a section "Submission Preferences"; and
   2. /admin/update-submission-preferences updates
      InstanceConfig.allow_document_uploads.

Testing

Test by toggling the Prevent sources from uploading documents
checkbox in the "Submission Preferences" section of the "Instance
Config" view.

1. If unchecked (default): Observe no changes from current behavior.

2. If checked: Observe the changes described in (2) and (3) above.

Deployment

No action required. After upgrading, administrators may use the
"Submission Preferences" section of the "Instance Config" view.

Checklist

If you made changes to the server application code:

• Linting (make lint) and tests (make test) pass in the
  development container
  • Fix tests/test_i18n.py::test_verify_default_locale_en_us_if_not_defined_in_config (expected by end of day Monday, October 21)

[lgtm-com]

This pull request introduces 1 alert when merging 3b028ac into 94e4a9d - view on LGTM.com

new alerts:

• 1 for Implicit string concatenation in a list

[lgtm-com]

This pull request introduces 1 alert when merging 3b4777f into 94e4a9d - view on LGTM.com

new alerts:

• 1 for Implicit string concatenation in a list

[lgtm-com]

This pull request introduces 1 alert when merging 55bda5b into 94e4a9d - view on LGTM.com

new alerts:

• 1 for Implicit string concatenation in a list

[zenmonkeykstop]

Thanks for this PR! It looks like a valuable feature. There might be issues safely applying the config change (or lack thereof) to older instances - one way around this would be to make this feature configurable via the Journalist Interface (similar to the custom logo option). The SD core team's unavailable right now, but we'll discuss and review early next week and add more feedback at that point.

[zenmonkeykstop]

Hi @wbaid - it looks like having this configured via the web interface is the preferred option for the core team. We'll be entering feature freeze for v1.1.0 tomorrow so will be unlikely to be able to suggest or incorporate any changes for that version, but this should be on the list of changes for the following v1.2.0 release.

If you have availability and would like to talk about the change, the Securedrop gitter forum is active at https://gitter.im/freedomofpress/securedrop , and we host weekday public standup meetings at 10AM Pacific/1PM Eastern at https://meet.google.com/ekb-kkhf-mrk

[wbaid]

Thanks, @zenmonkeykstop. We'll be happy to revise this pull request for v1.2. We took the approach of modifying "config.py" via the Ansible workflow because there did not seem to be an obvious precedent in the journalist interface's "admin.py" for configuration changes, as opposed to writes to definite filesystem paths (the logo) or database updates (users). Would you like the Web-based setting to take the form of: 1. an update to a simple key-value store defined as a SQLAlchemy model; 2. a write to (e.g.) an optional "config.json" or "config.yml"; or 3. something else we may be overlooking?

[redshiftzero]

I agree with @zenmonkeykstop that this seems like a useful feature. Doing 1 (adding a new table called e.g. instance_config that we can use to store key/value config values) I think is the best way to go here. In that scenario the update process from the admin's perspective is better since:

1. immediate, i.e. no Ansible playbook run required,
2. has nicer UX since using the CLI is not needed,
3. and it's easier to coordinate changes in the case of multiple admins.

[wbaid]

Thanks, @redshiftzero. We'll revise accordingly. We've marked this pull request back to "work in progress" and will comment here once it's been updated for the team to revisit.

[eloquence]

Thanks @wbaid -- we've set aside time in our current sprint (10/9 to 10/23) to help get this over the finish line if you have time to work on it from your end.

[wbaid]

@eloquence, we'll aim for the middle of next week. Appreciate your making time for it in the sprint.

[wbaid]

With our apologies for the delay, this should be ready for review by Monday.

[lgtm-com]

This pull request introduces 3 alerts when merging 4a95f8d into 486853c - view on LGTM.com

new alerts:

• 2 for Unused import
• 1 for Testing equality to None

[wbaid]

See new HEAD, with changes summarized (and revisions outlined) above. Other notes: 1. Migration tests and an integration test will be forthcoming this week, pending any further feedback. 2. We have appended these revisions as new commits, on the assumption that once approved this pull request will be squash-merged. Let us know if you'd like us to rebase things more logically for review purposes. 3. Even with the inversion of the admin UX to "*Prevent* sources...", we have kept the *setting* as "allow_document_uploads" so that conditionals elsewhere remain intuitive: e.g. "if allow..." rather than "if not prevent...". 4. app.instance_config is now completely distinct from app.{config,sdconfig}: a separate object/attribute rather than an extra set of keys to be inserted at runtime.

[lgtm-com]

This pull request introduces 2 alerts when merging 745009c into 486853c - view on LGTM.com

new alerts:

• 1 for Testing equality to None
• 1 for Unused import

[wbaid]

FYI: We won't be available to work further on this pull request until the week of November 18. We look forward to your feedback.

[wbaid]

@emkll or @eloquence, will the team still be able to review this pull request in time for inclusion in v1.2.0? We saw the announcement of today's feature-level freeze and can work with you to finalize this as quickly as possible this week, pending your feedback.

[redshiftzero]

heads up @wbaid it looks like there are some alembic tests failures occurring in CI related to this diff

[redshiftzero]

i just took a spin through this diff and tested in a dev container, the previous UX feedback is incorporated and the versioned instance_config table we discussed is working as advertised:

redshiftzero@8dd9552eef95:/var/lib/securedrop$ sqlite3 db.sqlite
SQLite version 3.11.0 2016-02-15 17:29:24
Enter ".help" for usage hints.
sqlite> select * from instance_config;
1|2019-11-19 19:05:02.421767|1
2|2019-11-19 19:17:31.742951|0
3|2019-11-19 19:17:45.151717|1
4||0

i think once we get the tests passing and add the functional test coverage this should be good to go (btw if you want to make the alembic migrations a bit easier to address you can just squash the two migrations into one)

to be honest, this is unlikely to make it into the merge window for 1.2.0 but is very likely to make it into 1.3.0

[wbaid]

Many thanks, @redshiftzero. We've squashed the Alembic migrations and fixed the failing migration tests. Thank you for pointing out this oversight on our part. We wanted to wait for this high-level confirmation of the new instance_config functionality before we add final tests for the allow_document_uploads setting. Would you like to see an integration test (per @emkll[1]), a functional test, or both? We can have either or both tests pushed tonight in order to meet the EOD deadline. [1]: #4879 (review)

[lgtm-com]

This pull request introduces 1 alert when merging 1435d51 into ed014be - view on LGTM.com

new alerts:

• 1 for Testing equality to None

[redshiftzero]

oh nice thanks for the fast update! Just the integration and unit coverage is required for merge given all the work you've put in here. Let's consider the functional/selenium based tests as nice to have and we can add as a followup issue for another contributor

[wbaid]

Thanks, @redshiftzero! We've just pushed integration tests for allow_document_uploads={True,False} to accompany the unit tests in "test_journalist.py" and "test_source.py". We appreciate your looking at this today.

[lgtm-com]

This pull request introduces 1 alert when merging ab25a8f into ed014be - view on LGTM.com

new alerts:

• 1 for Testing equality to None

[eloquence]

Thanks for making all the UX changes previously discussed, @wbaid -- as @redshiftzero noted, it looks great from a UX perspective! The only thing we may want to track as a follow-up enhancement is adding a flashed message after you click "Update Submission Preferences", to let the user know that the settings have been successfully modified. But this is clearly not a blocker for landing this feature.

[redshiftzero]

new integration tests are good, thank you for those and the contribution @wbaid!

This PR looks ready to me so I'm going to approve and merge this now - so it will be included in 1.2.0 (final release planned for December 3rd).

[wbaid]

Thanks to all---@redshiftzero, @eloquence, @emkll, @ninavizz, @zenmonkeykstop---for your feedback and your time in getting this ready for merge.