build: package only translations explicitly marked as supported #1497

cfm · 2022-05-25T18:21:18Z

Description

Closes #1438 by adding in MANIFEST.in (rather than setup.py as originally proposed):

a prune directive to exclude translation assets present in the tree by default; and
instructions for adding graft directives for explicitly including translation assets only for languages considered supported.

Also:

removes make supported-languages, obsoleted by this approach per Add list of supported languages; disable unsupported languages #1438 (comment);
removes the Weblate CLI (wlc), which is no longer needed; and
updates development dependencies along the way.
- ebb5336 fixes an unrelated typing error caught by (4)'s update of mypy 0.940 → 0.960. Let me know if a stack of pull requests is preferable for separate review of (3)‒(4) and then these (1)‒(2).

Test Plan

By default, no translations are packaged:

(.venv) user@sd-dev:~/securedrop-client$ python setup.py sdist 2> /dev/null | grep "locale"
(.venv) user@sd-dev:~/securedrop-client$

Pick a language from securedrop_client/locale, e.g.:

(.venv) user@sd-dev:~/securedrop-client$ export LANG=de

An unpackaged translation is still loadable:

(.venv) user@sd-dev:~/securedrop-client$ python -m securedrop_client  # appears in $LANG

A language marked for inclusion is packaged:

(.venv) user@sd-dev:~/securedrop-client$ echo "graft securedrop_client/locale/$LANG" >> MANIFEST.in
(.venv) user@sd-dev:~/securedrop-client$ python setup.py sdist 2> /dev/null | grep "locale"
creating securedrop-client-0.7.0/securedrop_client/locale
creating securedrop-client-0.7.0/securedrop_client/locale/de
creating securedrop-client-0.7.0/securedrop_client/locale/de/LC_MESSAGES
copying securedrop_client/locale/de/LC_MESSAGES/messages.mo -> securedrop-client-0.7.0/securedrop_client/locale/de/LC_MESSAGES
copying securedrop_client/locale/de/LC_MESSAGES/messages.po -> securedrop-client-0.7.0/securedrop_client/locale/de/LC_MESSAGES

Checklist

If these changes modify code paths involving cryptography, the opening of files in VMs or network (via the RPC service) traffic, Qubes testing in the staging environment is required. For fine tuning of the graphical user interface, testing in any environment in Qubes is required. Please check as applicable:

I have tested these changes in the appropriate Qubes environment
I do not have an appropriate Qubes OS workstation set up (the reviewer will need to test these changes)
These changes should not need testing in Qubes

If these changes add or remove files other than client code, the AppArmor profile may need to be updated. Please check as applicable:

I have updated the AppArmor profile
No update to the AppArmor profile is required for these changes

These assets are covered by:

securedrop-client/files/usr.bin.securedrop-client

Line 21 in 1c5482b

/opt/venvs/securedrop-client/** r,

I don't know and would appreciate guidance

If these changes modify the database schema, you should include a database migration. Please check as applicable:

I have written a migration and upgraded a test database based on main and confirmed that the migration applies cleanly
I have written a migration but have not upgraded a test database based on main and would like the reviewer to do so
I need help writing a database migration
No database schema changes are needed

cfm · 2022-05-25T18:48:00Z

Marking as ready for review while I investigate why an unrelated test has started to fail in CI.

cfm · 2022-05-26T16:37:12Z

Once CI has been placated on main, I'll rebase this and see if anything's still unhappy here.

gonzalo-bulnes · 2022-05-26T18:13:49Z

You should be good to go whenever you want now @cfm -CI is fixed | latest nightly build 🙂

Translations are excluded from the source distribution by default by pruning the "locale" directory. Translations for supported languages must be explicitly grafted back in. All translations present in the tree remain loadable during development via (e.g.): $ LANG=es python -m securedrop_client

Per #1438, language support will be a maintainers' decision, rather than an automated check based on translation coverage in Weblate.

wlc was only used by "make supported-languages", which we've removed.

cfm · 2022-06-22T21:42:38Z

Investigating unrelated failing check remaining after rebase from main:

securedrop_client/queue.py:95: error: Argument 1 to "append" of "list" has incompatible type "Optional[QueueJob]"; expected "QueueJob"  [arg-type]

Caught by mypy 0.940 → 0.960.

cfm · 2022-06-22T22:27:31Z

Ready for review with CI (finally) passing. Along the way, ebb5336 fixes an unrelated typing error caught by new mypy 0.940 → 0.960.

This is an unfortunate side effect of not pinning development dependencies. Removing a dependency X from a requirements.in still entails regenerating the requirements.txt for all dependencies X'. And new versions of mypy often ~~break~~ reveal preexisting bugs in things!

Let me know if a stack of pull requests is preferable for separate review.

sssoleileraaa

lgtm - test plan checks out

sssoleileraaa · 2022-07-05T18:28:09Z

MANIFEST.in

+#
+#     graft securedrop_client/locale/$LANG
+#
+# Please keep this list alphabetized.


sssoleileraaa · 2022-07-05T18:30:15Z

securedrop_client/queue.py

@@ -92,7 +92,8 @@ def _check_for_duplicate_jobs(self, job: QueueJob) -> bool:
        stored on self.current_job. We check that the job to be added is not among them.
        """
        in_progress_jobs = [in_progress_job for priority, in_progress_job in self.queue.queue]
-        in_progress_jobs.append(self.current_job)
+        if self.current_job is not None:
+            in_progress_jobs.append(self.current_job)


I see this was caught by mypy update to 0.960

cfm added the i18n label May 25, 2022

cfm added this to In Development in SecureDrop Team Board May 25, 2022

cfm marked this pull request as ready for review May 25, 2022 18:47

cfm requested a review from a team as a code owner May 25, 2022 18:47

cfm moved this from In Development to Ready for Review in SecureDrop Team Board May 25, 2022

cfm force-pushed the 1438-supported-languages branch 2 times, most recently from 5840426 to fb85e48 Compare May 27, 2022 06:22

cfm mentioned this pull request Jun 3, 2022

Add support for community localization #239

Closed

10 tasks

gonzalo-bulnes self-requested a review June 8, 2022 05:59

cfm self-assigned this Jun 22, 2022

cfm moved this from Ready for Review to In Development in SecureDrop Team Board Jun 22, 2022

cfm added 3 commits June 22, 2022 14:31

build: remove "supported-languages" target

7c076f2

Per #1438, language support will be a maintainers' decision, rather than an automated check based on translation coverage in Weblate.

chore: remove Weblate CLI (wlc); update development dependencies

8f2761b

wlc was only used by "make supported-languages", which we've removed.

cfm force-pushed the 1438-supported-languages branch from fb85e48 to 94b2a50 Compare June 22, 2022 21:33

cfm added 2 commits June 22, 2022 15:15

fix(_check_for_duplicate_jobs): don't add current job if None

ebb5336

Caught by mypy 0.940 → 0.960.

docs: add localization steps to release checklist

57e9b40

cfm force-pushed the 1438-supported-languages branch from 2ed1bc4 to 57e9b40 Compare June 22, 2022 22:16

cfm removed their assignment Jun 22, 2022

cfm moved this from In Development to Ready for Review in SecureDrop Team Board Jun 22, 2022

sssoleileraaa approved these changes Jul 5, 2022

View reviewed changes

sssoleileraaa merged commit 69c3d7a into main Jul 5, 2022

SecureDrop Team Board automation moved this from Ready for Review to Done Jul 5, 2022

sssoleileraaa deleted the 1438-supported-languages branch July 5, 2022 18:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: package only translations explicitly marked as supported #1497

build: package only translations explicitly marked as supported #1497

cfm commented May 25, 2022 •

edited

cfm commented May 25, 2022

cfm commented May 26, 2022

gonzalo-bulnes commented May 26, 2022

cfm commented Jun 22, 2022 •

edited

cfm commented Jun 22, 2022

sssoleileraaa left a comment •

edited

sssoleileraaa Jul 5, 2022

sssoleileraaa Jul 5, 2022

build: package only translations explicitly marked as supported #1497

build: package only translations explicitly marked as supported #1497

Conversation

cfm commented May 25, 2022 • edited

Description

Test Plan

Checklist

cfm commented May 25, 2022

cfm commented May 26, 2022

gonzalo-bulnes commented May 26, 2022

cfm commented Jun 22, 2022 • edited

cfm commented Jun 22, 2022

sssoleileraaa left a comment • edited

Choose a reason for hiding this comment

sssoleileraaa Jul 5, 2022

Choose a reason for hiding this comment

sssoleileraaa Jul 5, 2022

Choose a reason for hiding this comment

cfm commented May 25, 2022 •

edited

cfm commented Jun 22, 2022 •

edited

sssoleileraaa left a comment •

edited