Automatically update dom0 and VM configs over time

[conorsch]

Overview

Installs a new script to dom0 called securedrop-update, and symlinks it into cron.daily, so it runs regularly without manual activation. The script will:

• pull in all new dom0 updates
• start each TemplateVM and update its packages internally (included apt-test-qubes deb packages)
• display clunky feedback about the update process with a pretty SD logo

Closes #24.

Screenshots

Here's what the tool looks like in action:

Metrics

The script takes ~10m to run. (If you haven't updated your TemplateVMs in a while, then it could take longer, ~30m, the first time you run it!) It will take less if you don't currently have the Workstation VMs configured (e.g. if you've recently run make clean).

[user@dom0 securedrop-workstation]$ time sudo securedrop-update

[...snip...]

local:
    ----------
whonix-gw-14: OK
fedora-28: OK
openvpn: OK
sd-svs-disp-template: OK
signal-template: OK
debian-9: OK
whonix-ws-14: OK
sd-svs-template: OK
sd-journalist-template: OK
sd-whonix-template: OK
sd-workstation-template: OK
securedrop-workstation: OK

real	10m3.111s
user	0m37.405s
sys	1m1.834s

Next steps

As with other dom0-related configs, we'll need to package these as an RPM (#171) eventually.

[conorsch]

@emkll Does the update-tor command resolve #148? If so, might be best to submit as a follow-up PR... I gather the appropriate testing workflow for resolution there would be to uninstall the Whonix templates from dom0, then re-apply. Agree?

[emkll]

After having run the make target over 48h ago, I can confirm the cron logic works as expected, and all templates are updated roughly daily. Here are a few comments. I think the first one (update of the AppVMs) as well as the inline comment should be addressed by this PR. The rest are mostly observations, and might warrant follow-up tickets.

Updating AppVMs (state.highstate accross the board)

To benefit from Template updates, AppVMs must be rebooted (and associated templates shut down, more on that later). Given that rebooting arbitrary AppVMs has the potential to disrupt end-user flow. I see two options:

1. For package updates, a notification (e.g, notify-send, like in this PR) repeated every 5 minutes or so requesting the user reboot the workstation (heavy handed, but will restart all AppVMs, including sys-net, sys-firewall and sys-usb without fail).
2. A small PyQt widget or dock widget running in dom0 that would tell the users updates are required and reboot the VMs that need updating (templates, then AppVMs) and apply any configuration change. (I have not figured out an obvious way to get information on via CLI, need more docs research). I've looked into the Journalist updater code, and unfortunately the QtWidget class used by that appears to be PyQt5-only, my local dom0 only has PyQt4. This will be required to apply configuration changes on the AppVms.

Updating the configuration of the AppVMs likely requires more concrete details on how we plan on shipping the configuration logic.

Resource usage

With at least 9 templates: debian, fedora, sd-workstation-template, sd-svs, sd-svs-disp, sd-journalist, sd-whonix, whonix-gw, whonix-ws (with the last 4 updating over tor) on a default workstation install, this will take a significant amount of time, and there may be a chance that not all templates are updated within that period.

For example, my last update run (I had updated the day before) took just over 30 minutes to update 14 templates. Further user testing is required, but a concurrency of 2 seems sensible in the workstation scenario.

Qubes errors while running

During the update process, I occasionally experienced the same QubesPropertyAccessError errors that occur during provisioning, if the Qubes-Manager app is open while performing the upgrades.

When doing I would occasionally experience errors such cannot connext to qrexec agent for 60 seconds, which happens when I've exhausted CPU/IO . It also causes the update task on the template to FAIL instead of OK and appears to not update the template.

Developer environment

I assume some will use the same machine for development and testing, and by design the cron job will almost certainly run at the worst possible time and slow down the workstation considerably. Perhaps in the future it would be nice to have a config option to set concurrency.

[emkll]

Perhaps due to the configuration of my Qubes machine, my user in dom0 is not user

[conorsch]

Ouch, that's a good flag. We still need to drop privileges in here; or we could dig more in the notify-send settings. Off the cuff, inspecting /home/ for a single dirname should give us whatever the name of the (single) custom user is. Make sense?

[conorsch]

We can safely assume that the normal user configured at install time has uid 1000; so:

id -nu 1000

Then we su to that user to run the notify-send commands.

[conorsch]

Discussed review changes with @emkll; to summarize, next steps for this PR are:

• implement custom "user" handling for notify-send command
• use state.highstate to enforce VM settings (not just packages inside VMs)
• drop --templates flags and qubes-dom0-update cmd (as @marmarek helpfully pointed out is superfluous here)
• add a GUI notification recommending reboot of the full workstation after updates are applied.

The GUI notification recommending a reboot has UX implications; but it's better to warn than reboot the AppVMs while they may be in use. After this PR is stable and merged, we can ping @ninavizz (and @eloquence) for assistance evaluating the UX impact and recommended improvements.

For the update-tor fix, we'll handle separate, in a discrete PR targeting #148.

[conorsch]

All outstanding items addressed. Please have another look, @emkll! Would love your thoughts as well, @joshuathayer .

[marmarek]

Oh, sorry, I think I wasn't clear. My previous comment was about the ordering only, not which actions are performed. If you want to update templates, you still need --templates. If you want to apply configuration to other vms (non-templates), then you need --all.

[conorsch]

That would explain the behavior I'm seeing locally, @marmarek; many thanks for your guidance here!

[redshiftzero]

I've looked into the Journalist updater code, and unfortunately the QtWidget class used by that appears to be PyQt5-only, my local dom0 only has PyQt4

Note we can without too much difficulty migrate PyQt5 -> PyQt4 such that we can reuse that GUI application

[emkll]

I've addressed the feedback with the actions performed in securedrop-update (update dom0, templates, and AppVMs), and made some changes due to (new) flake8 failures. This now looks good to merge from my pespective.
Since I was the last one to push to this branch, would either @joshuathayer or @conorsch mind taking one last look at this PR?

[conorsch]

Thanks for handling the flake8 fixes, @emkll! Changes LGTM. Let's give @joshuathayer a chance to test prior to merging.

[joshuathayer]

Regarding UX for altering users about the need to update: an another option, we could probably add some notification to the client application UI, via a qubes-rpc job from dom0 to sd-svs which would twiddle some state in the securedrop-client DB.

In terms of review... I've been running the update script now and it's been running at least 2 hours. I've not observed any errors yet but that seems significantly longer than other people's experience, eh?

[emkll]

adding the --clean option here will refresh dnf cache, which might be useful in some cases.

I just had an issue where qubes-dom0-update was complaining of an unsigned package, due to me attempting to download an older whonix template in an effort to reproduce #122 (comment)

[conorsch]

Agreed, probably worth adding here, lest we forget to circle back—feel free to append, @emkll.

[joshuathayer]

Minor nit: "SecureDrop:" is added in securedrop-update-feedback(), so isn't needed in the message here.

[joshuathayer]

I eventually manually killed my first run of the update script, which seemed to be hung after running for 3+ hours. Rerunning it succeeded in just a few minutes.

I don't have a good report of where or how it hung, sadly. But the newly updated system works well for me, and if the process worked well for others, I'm happy to blame some non-SD situation on my machine for the problem I had. So, lgtm!

[conorsch]

I eventually manually killed my first run of the update script, which seemed to be hung after running for 3+ hours.

I've seen this happen only once in about a dozen runs—I suspect it's the Whonix VM updates, since those updates often crawl. Let's keep an eye on it over time. Since updates happen in the background, we can count on the updates being installed in a timely fashion—but if Whonix updates require hours, that may require a special fix, e.g. preventing proxying of Whonix updates over Tor.

[kushaldas]

I eventually manually killed my first run of the update script, which seemed to be hung after running for 3+ hours. Rerunning it succeeded in just a few minutes.

This happens most of the time in the whonix templates for me, the apt-get update even gets stuck (I tried to do that manually) to fetch any file.

[emkll]

Some packages are not updating for me after running the updater, seems to be limited to the packages served by the qubes repo. Can anyone reproduce this on a template that was newly updated by the securedrop-update script?

$ sudo apt list --upgradable 
Listing... Done
libvchan-xen/unknown 4.0.5-1+deb9u1 amd64 [upgradable from: 4.0.3-1+deb9u1]
libxen-4.8/unknown 2001:4.8.4-5+deb9u1 amd64 [upgradable from: 2001:4.8.4-2+deb9u1]
qubes-gui-agent/unknown 4.0.17-1+deb9u1 amd64 [upgradable from: 4.0.16-1+deb9u1]
xen-utils-common/unknown 2001:4.8.4-5+deb9u1 all [upgradable from: 2001:4.8.4-2+deb9u1]
xserver-xorg-input-qubes/unknown 4.0.17-1+deb9u1 amd64 [upgradable from: 4.0.16-1+deb9u1]
xserver-xorg-video-dummyqbs/unknown 4.0.17-1+deb9u1 amd64 [upgradable from: 4.0.16-1+deb9u1]

[conorsch]

Fantastic catch, @emkll ! I was indeed able to reproduce your results. I've pushed up two commits, one with tests to guard against regressions (using the apt list command you provided), and another with a patch to the updater logic. Please pull down the changes and try again.

Not yet fully confident in these changes because they're still running on my machine—and it appears it's going to take a while. Note also that the test logic checking for packages being up to date assumes the AppVMs have been rebooted; that'll require a fresh cycle (or rebuild) or the VMs after updates are actually installed. As it stands, I haven't yet observed the tests passing—will try running them when the upgrade logic finishes.

[conorsch]

One more tweak, squashed into prior commits. Tests are passing for me. Take it for a spin, @emkll.

Details in the commit messages, but looks like the new approach only works against Debian VMs. That's fine for merge, but something we should be aware of.

[emkll]

Thanks for your patience, @conorsch ! This does indeed fix the issue described above for Debian-based templates -- all packages are now correctly updated by the securedrop-update script/cron. I will create a follow-up ticket to track if the script correctly updates fedora-based templates.