Install libapache2-mod-wsgi-py3, which can only be installed if libapache2-mod-wsgi is not there

[kushaldas]

Description

For Python3 we will have to install libapache2-mod-wsgi-py3 which can only be installed if libapache2-mod-wsgi is not there. Means, we can not just autoupgrade to Python3 easily,

User Stories

I am SecureDrop admin, and I want to move to the latest Python3 based SecureDrop smoothly without any userinput.

[conorsch]

This issue is currently blocking progress on #4355. We cannot manage package state within postinst, given that the dpkg lock is active when those scripts run, so we must handle the dependency resolution outside the context of individual packages such as securedrop-app-code.

The best bet we have to leverage the cron-apt upgrade logic, similar to https://github.com/freedomofpress/securedrop/blob/e3da51c7a0a6e72103fdd4f39af4db8781095716/install_files/securedrop-config/etc/cron-apt/action.d/9-remove. The cron-apt logic is currently written to be agnostic to server roles, however, and the package removal of libapache2-mod-wsgi is only relevant to the Application Server. Ensuring removal of the package prior to the normal upgrade logic would result in an uninstall/reinstall flow for the subsequent release, thereby unblocking a future release to update the package dependencies to use libapache2-mod-wsgi-py3.

[conorsch]

The cron-apt tweaks are sufficient to ensure that libapache2-mod-wsgi is removed, then automatically reinstalled given its status as a dependency of securedrop-app-code. However, during removal of libapache2-mod-wsgi, the securedrop-app-code package is uninstalled, as well—which may create problems for database state. We'll have to test the alembic migrations carefully to ensure that they don't break against a previously existing database when the package is first installed.

[conorsch]

Proceeded with a test of the two-step upgrade logic, first modifying the cron-apt configs, second pulling in the new py3 dependencies via a rebuilt securedrop-app-code deb. Discovered that the libapache2-mod-wsgi-py3 package is served from the universe repo (https://packages.ubuntu.com/xenial/httpd/libapache2-mod-wsgi-py3).

Problems there are two-fold:

1. Universe packages do not receive security updates (view distro warning in /etc/apt/sources.list: "N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu team. Also, please note that software in universe WILL NOT receive any review or updates from the Ubuntu security team."
2. The cron-apt config ignores updates from universe (due to lack of security updates).

If we decide it's worthwhile to depend on the universe channel, we'll need to update the cron-apt config logic to support automatic installation. If we decide to preserve the existing update channels, then we'll need to find another way to unblock Python3 WSGI.

Let's discuss as a team which is the preferable option here. Once we have clarity, we can resume functional testing to evaluate db state behavior.

[rmol]

After digging into mod_wsgi's installation options and talking to @conorsch, it looks like there are at least the following solutions to choose from:

1. System Apache + bundled mod_wsgi

Replace the stock Ubuntu libapache2-mod-wsgi package with a pip-installed mod_wsgi included in the securedrop-app-code package. This requires the cron-apt tweak described above to purge libapache2-mod-wsgi, along with removal of libapache2-mod-wsgi as a dependency of securedrop-app-code, then changing /etc/apache2/mods-available/wsgi.load to point to our module. Otherwise, the Apache configuration would remain the same. We'd need to thoroughly test the cron-apt upgrade to see how the dependency is handled.

Advantages

• Requires the least change to Apache configuration.

Disadvantages

• The apt dependencies could mean a fraught transition; breakage would require manual correction.

2. System Apache, modified to proxy to new mod_wsgi-express Apache instances for source/journalist interfaces.

Here we don't have to deal with libapache2-mod-wsgi at all, other than removing it as a dependency from securedrop-app-code. There would however be a lot of change to /etc/sites-available/{journalist,source}.conf to remove mod_wsgi, add mod_proxy, and determine how to split up the remaining config between the system Apache and mod_wsgi-express. The mod_wsgi-express source configuration would be simpler as we wouldn't have to construct it by parsing the existing configuration files, where clients serve the source interface over HTTPS.

Advantages

• Avoids libapache2-mod-wsgi dependency problem.

Disadvantages

• Editing the Apache config will be risky, again probably requiring manual correction if we break it.
• A lot more work configuring mod_wsgi-express, supervisord, AppArmor, etc.

3. Decommission system Apache altogether; point Tor to new custom Apache instances for source/journalist interfaces.

Leave libapache2-mod-wsgi installed. Embed mod_wsgi in securedrop-app-code. Run the source and journalist interfaces in their own Apache instances with mod_wsgi-express under supervisord control. We could run them on the same ports, to avoid tweaking the Tor configuration, or move them to new ports and only run as www-data. This does support SSL configuration for sites using HTTPS on the source interface, but that would require parsing the certificate filenames from the old system Apache config, so it's riskier than just adjusting the mod_wsgi Apache module configuration in the first option above.

Advantages

• Avoids libapache2-mod-wsgi dependency problem.
• Doesn't involve the risk of changing the existing Apache configuration.
• Offers the possibility of running source interface as www-data.

Disadvantages

• A lot more work configuring mod_wsgi-express, supervisord, AppArmor, etc.
• Will be tricky to make sure we've reproduced the old Apache configuration in the mod_wsgi-express configuration, particularly source HTTPS.

It's important to note that all of these options involve the same ongoing security concern. We'll be responsible for getting a new securedrop-app-code package out whenever there's a vulnerability in mod_wsgi.

Also, in exploring this, it's become clear that though our application code is now Python 3 compatible, we still have a fair amount of work to do to base the securedrop-app-code package on Python 3; our development setup still assumes Python 2; and Molecule requires Python 2.7 or 3.6+ -- so we'll have to use Python 3.6+ for development, but be deploying to Python 3.5.x on Xenial.

I'm sure there are still pitfalls and obstacles I haven't thought of, but @conorsch does that accurately capture what we've considered so far?

[conorsch]

Great summary, @rmol! Modest preference for option 1, given the minimal updates required to the apache vhost configs. One clarifying question on this passage:

Replace the stock Ubuntu libapache2-mod-wsgi package with a pip-installed mod_wsgi included in the securedrop-app-code package. This requires the cron-apt tweak described above to purge libapache2-mod-wsgi, along with removal of libapache2-mod-wsgi as a dependency of securedrop-app-code, then changing /etc/apache2/mods-available/wsgi.load to point to our module.

What if we:

1. Remove the libapache2-mod-wsgi dependency from the securedrop-app-code package control file
2. Include the pip-installed mod_wsgi in the securedrop-app-code package
3. Modify /etc/apache2/mods-available/wsgi.load to point to the pip-installed mod_wsgi via postinst
4. Ship updates to cron-apt logic that remove libapache2-mod-wsgi during nightly runs

Then we'd have neatly sidestepped the dependency madness, and on the subsequent nightly run—after the servers have been upgraded to the latest version including the changes listed above—then the no-longer-used libapache2-mod-wsgi package will be absent going forward. That'd still leave ~24h window when upstream updates to the libapache2-mod-wsgi package could clobber our config—assuming libapache2-mod-wsgi writes to /etc/apache2/mods-available/wsgi.load, via postinst or otherwise—but it's clear sailing after that.

Does the above solution match your observations of the how the service must be defined when using the pip-installed wsgi? If so, then we can discuss the acceptability of the ~24h window of updates. If we decide we'd like to avoid it altogether, we could be able to pin a specific upstream version of libapache2-mod-wsgi in the securedrop-app-code dependencies, to ensure updates won't clobber, then remove it altogether in a subsequent release.

[rmol]

I think that flow should work nicely. I'd been (still am, honestly) unclear on conffile handling, but some quick testing suggests that we can modify /etc/apache2/mods-available/wsgi.* and if we're just removing, not purging, in the /etc/cron-apt/action.d entry, they'll be left in place and we should be fine.

[eloquence]

This was resolved via #4622.