Update dev container browser software and validation #5472

rmol · 2020-08-31T20:57:20Z

Status

Ready for review

Description of Changes

Mozilla's keyserver went down, and they might just leave it that way. Their release signing key is available on another one, but that doesn't seem to be the result of any official policy I can find. The closest thing to guidance comes from a blog post instructing us to obtain the key from the archive server along with the software. This changes the Dockerfile to do that.

Recent versions of geckodriver are being signed with the release key, so we'll now use it to verify that package too.

We were also using an expired version of the Tor release signing key, kept in our repo, so I'm following their instructions for obtaining the key instead.

Finally, this updates all three packages.

Finally finally, this includes a workaround for the recent setuptools breakage.

Fixes #5470.
Fixes #5471.

Testing

Check out the 5470-keystone-kops branch.
Start the dev server with make dev. Confirm that the development container is built and started properly.

Deployment

Dev/test only.

Checklist

If you made changes to the server application code:

Linting (make lint) and tests (make test) pass in the development container

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

Mozilla's keyserver went down, and they might just leave it that way. Their release signing key is available on another one, but that doesn't seem to be the result of any official policy I can find. The closest thing to guidance comes from a blog post instructing us to obtain the key from the archive server along with the software[1]. This changes the Dockerfile to do that. Recent versions of geckodriver are being signed with the release key, so we'll now use it to verify that package too. We were also using an expired version of the Tor release signing key, kept in our repo, so I'm following their instructions[2] for obtaining the key instead. Finally, this updates all three packages. [1] https://blog.mozilla.org/security/2019/06/13/updated-firefox-gpg-key/ [2] https://support.torproject.org/tbb/how-to-verify-signature/

emkll

Thanks for the fixes @rmol , good to merge when CI passes

securedrop/dockerfiles/xenial/python3/Dockerfile

kushaldas

We have to find a way to use the older (46.0.0) setuptools here :(

install_files/ansible-base/roles/build-securedrop-app-code-deb-pkg/tasks/main.yml

kushaldas · 2020-09-01T12:41:52Z

I am going to push a separate PR just to test one idea, if that is good, then we can bring in the changes here.

Install dh-virtualenv from Debian Sid, switch to its alternative build system, and tell it to use our pinned version of setuptools.

I've pushed a change that does pin setuptools for dh-virtualenv, using a newer version of it with the alternative build system.

sssoleileraaa · 2020-09-01T21:04:03Z

Looks like all checks have passed so this is ready for another review. Hoping to get this merged today, so wondering if someone is already reviewing?

conorsch · 2020-09-01T21:40:20Z

@creviera Thanks for pinging. I'll take a pass through the test plan today, but given the complexity of the package build changes, this may not land in develop. So I recommend you branch locally if you're blocked at the moment!

@rmol Since this PR also addresses #5471, I'm going to extend the test plan with some package comparisons.

rmol · 2020-09-01T21:48:44Z

@conorsch Sounds good. Thanks.

sssoleileraaa · 2020-09-01T22:23:14Z

@creviera Thanks for pinging. I'll take a pass through the test plan today, but given the complexity of the package build changes, this may not land in develop. So I recommend you branch locally if you're blocked at the moment!

ah i thought it just updated the keyserver string - that's why our PRs are failing in the SDK repo. there's an SDK PR that allows us to merge a couple client PRs. at this point everything is tested so don't need to branch from this - just want to be able to merge BUT no rush. there plennnnty of other things to work on.

sssoleileraaa · 2020-09-01T22:25:07Z

ah right! it also includes the setuptools in xenial workaround

conorsch · 2020-09-02T00:44:08Z

There are significant changes in the resulting deb packages for Xenial. Some are appear superficial:

For all bundled Python dependencies, the directory path for metadata uses the "dist" format rather than the "<python_version>-egg" format, e.g.

/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi-4.6.7-py3.5.egg-info/
becomes
/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/mod_wsgi-4.6.7.dist-info/

Fortunately there's only package metadata in those directories, not source code for the libraries. The AppArmor profile
also liberally grants access to /opt/venvs/securedrop-app-code/lib/python3.5/, so no need to adjust grants there.

More concerning, the postinst for securedrop-app-code is several hundred lines longer, due to injection of systemd helper code. Perhaps the override_dh_systemd_* targets are no longer honored?

The virtualenv path declared in /opt/venvs/securedrop-app-code/bin/activate has changed from

VIRTUAL_ENV="/opt/venvs/securedrop-app-code" to
VIRTUAL_ENV="/tmp/securedrop-app-code_1.6.0~rc1+xenial_amd64/build/opt/venvs/securedrop-app-code"

The latter path is an emphemeral build dir, and does not exist on the App Server, nor is it bundled in the deb package. Fortunately our WSGI settings hardcode /opt/venvs/securedrop-app-code as the PythonHome, so the app still works well in a staging environment. The fact that build-time artifacts have slipped into the prod package suggests we may want further refinements to the build logic given the upgrade to dh-virtualenv.

That said, the app does work, based on some interactive spot-checking I tried in a local staging env, and CI is passing again, so I'm going to proceed with merge here.

conorsch

See comment listing some potentials areas for follow-up. Build is passing, though, so let's get it into develop. Thanks for plowing through multiple fixes to unbreak builds, @rmol!

rmol · 2020-09-02T14:20:03Z

More concerning, the postinst for securedrop-app-code is several hundred lines longer, due to injection of systemd helper code. Perhaps the override_dh_systemd_* targets are no longer honored?

What were you comparing it to? I grabbed the 1.5.0 package from apt.freedom.press and compared to the local deb, and the systemd content was the same.

The virtualenv path declared in /opt/venvs/securedrop-app-code/bin/activate has changed

I'm looking into this. dh-virtualenv's other path correction logic (shebangs in other scripts) seems to be working, just not the activate adjustment. Still obviously a showstopper.

conorsch · 2020-09-02T15:20:05Z

What were you comparing it to? I grabbed the 1.5.0 package from apt.freedom.press and compared to the local deb, and the systemd content was the same.

Good point! Looks like I used 1.5.0~rc1 debs, but dated 2020-08-10, meaning they were out of date—left over from reviewing Focal build logic changes. Comparing with the 1.5.0 prod debs, the diff is much smaller, exactly what we want. Thanks for checking, @rmol !

rmol · 2020-09-03T14:02:14Z

Just a note for future us: the alternative build system lacks the path correction of the virtualenv activation scripts. It's only in the dh-virtualenv Python script, and would need to be replicated in the debhelper Perl module, or our own code: hacks in rules or the like.

rmol requested review from kushaldas and redshiftzero as code owners August 31, 2020 20:57

Work around this week's setuptools breakage

74b6250

rmol requested review from conorsch and emkll as code owners August 31, 2020 22:08

emkll previously approved these changes Aug 31, 2020

View reviewed changes

securedrop/dockerfiles/xenial/python3/Dockerfile Show resolved Hide resolved

securedrop/dockerfiles/xenial/python3/Dockerfile Show resolved Hide resolved

securedrop/dockerfiles/xenial/python3/Dockerfile Show resolved Hide resolved

eloquence added this to Ready for Review in SecureDrop Team Board Sep 1, 2020

kushaldas previously requested changes Sep 1, 2020

View reviewed changes

install_files/ansible-base/roles/build-securedrop-app-code-deb-pkg/tasks/main.yml Outdated Show resolved Hide resolved

Install newer dh-virtualenv in Xenial builder

1773732

Install dh-virtualenv from Debian Sid, switch to its alternative build system, and tell it to use our pinned version of setuptools.

rmol dismissed emkll’s stale review via 1773732 September 1, 2020 18:01

conorsch approved these changes Sep 2, 2020

View reviewed changes

conorsch merged commit 2777a9f into develop Sep 2, 2020

SecureDrop Team Board automation moved this from Ready for Review to Done Sep 2, 2020

conorsch mentioned this pull request Sep 2, 2020

Patch dhvirtualenv for setuptools DO NOT MERGE #5473

Closed

7 tasks

rmol mentioned this pull request Sep 3, 2020

Build defenses against setuptools #5480

Closed

conorsch mentioned this pull request Sep 4, 2020

Revert to old dh-virtualenv, use built-in venv module #5484

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update dev container browser software and validation #5472

Update dev container browser software and validation #5472

rmol commented Aug 31, 2020 •

edited

Loading

emkll left a comment

kushaldas left a comment

kushaldas commented Sep 1, 2020

sssoleileraaa commented Sep 1, 2020

conorsch commented Sep 1, 2020

rmol commented Sep 1, 2020

sssoleileraaa commented Sep 1, 2020

sssoleileraaa commented Sep 1, 2020

conorsch commented Sep 2, 2020

conorsch left a comment

rmol commented Sep 2, 2020

conorsch commented Sep 2, 2020

rmol commented Sep 3, 2020

Update dev container browser software and validation #5472

Update dev container browser software and validation #5472

Conversation

rmol commented Aug 31, 2020 • edited Loading

Status

Description of Changes

Testing

Deployment

Checklist

If you made changes to the server application code:

If you made non-trivial code changes:

emkll left a comment

Choose a reason for hiding this comment

kushaldas left a comment

Choose a reason for hiding this comment

kushaldas commented Sep 1, 2020

sssoleileraaa commented Sep 1, 2020

conorsch commented Sep 1, 2020

rmol commented Sep 1, 2020

sssoleileraaa commented Sep 1, 2020

sssoleileraaa commented Sep 1, 2020

conorsch commented Sep 2, 2020

conorsch left a comment

Choose a reason for hiding this comment

rmol commented Sep 2, 2020

conorsch commented Sep 2, 2020

rmol commented Sep 3, 2020

rmol commented Aug 31, 2020 •

edited

Loading