Update base templates to buster #352

emkll · 2019-11-27T14:35:03Z

Closes #345

For new installs: Buster templates by default
For existing installs: In-place update to buster

Note that the upgrade logic will shut down sd-svs and sd-gpg (normally only a single time)

Testing

Clean scenario

make all and all tests should pass

Upgrade scenario

make all and all tests should pass

conorsch

Ran through a make all locally, and I'm pleased to report all tests were passing! The cleanup logic is sound. There are certainly conflicts with the outstanding changes in #351, so let's coordinate where changes should land.

The specific request I have is that the new cleanup script logic be called via Salt (but not ported to Salt tasks!). If it's called via salt, then our coalescing unattended-upgrade plans will be able to trigger these VM migrations without any Admin action, same as we did in #331.

conorsch · 2019-11-27T18:54:36Z

scripts/prep-upgrade

+# DispVM templates.
+
+# sd-svs, we simply shutdown the machine as we want to preserve the data
+qvm-check sd-svs > /dev/null 2>&1


Nit: qvm-check --quiet will omit the user-facing messages without masking all errors.

You can also collapse the conditional into:

if qvm-check --quiet sd-svs; then

and skip checking the exit code separately.

conorsch · 2019-11-27T18:54:57Z

scripts/prep-upgrade

+	if [[ ! $BASE_TEMPLATE =~ "buster" ]]; then
+		qvm-check --running sd-svs > /dev/null 2>&1
+		if [[ $? -eq 0 ]]; then
+			qvm-shutdown sd-svs && sleep 5


Nit: use qvm-shutdown --wait to block, rather than sleeping.

conorsch · 2019-11-27T19:45:43Z

scripts/prep-upgrade

+		qvm-shutdown sd-gpg
+		sleep 5
+	fi
+fi


At the end of this script seems to be the appropriate time to clean up the old VMs. That, or in a subsequent Salt task. Hardcoding the VMs to remove seems to be what we want. This Python one-liner describes the logic:

python3 -c 'from qubesadmin import Qubes; q = Qubes(); sdw_vms = [x.name for x in q.domains \ if "sd-workstation" in x.tags and x.klass == "TemplateVM" \ and "buster" not in x.name \ and not x.installed_by_rpm ]; print("\n".join(sdw_vms))'

I don't recommend we use that convoluted one-liner, although it does illustrate how useful the Admin API can be for use cases like this. For now, we can simply hardcode the list of "old" VMs in a Salt list and ensure they're gone.

Unfortunately, we cannot set them as part of this script, because the old (stretch) templates are still being used: we need to wait for the state of the templates to be applies, so that the stretch templates are no longer being used by an appvm

conorsch · 2019-11-27T19:48:00Z

scripts/provision-all

@@ -7,7 +7,7 @@ set -o pipefail

 # Format list of all VMs comma-separated, for use as qubesctl target
 all_sdw_vms_target="$(./scripts/list-vms | perl -npE 's/\n/,/g' | perl -npE 's/,$//' )"
-
+templates_to_remove="$(./scripts/list-vms old-templates | perl -npE 's/\n/ /g')"


N.B. that #351 removes the list-vms script, in favor of the new qvm-ls --tags support. As above, if we perform the removal in Salt, we can execute in parallel against all of them, as long as we wait until the bash script has finished.

conorsch · 2019-11-27T19:49:11Z

scripts/provision-all

+# This is required only when swapping base templates, see scripts/prep-upgrade
+# for more details.
+echo "Prep for upgrade, shut down AppVMs that require reboot for template swap"
+./scripts/prep-upgrade


Recommend moving the script call out of the provision-all script and into a Salt state, reason being that then we have the ability to trigger the change via unattended-upgrades, e.g. via dom0 state.highstate. Currently only developers run this bash script, so let's not depend on it overmuch to enforce updates we want to land on pilot/prod instances.

conorsch · 2019-11-27T19:49:45Z

scripts/provision-all

@@ -27,3 +32,6 @@ echo "Provision all SecureDrop Workstation VMs with service-specific configs"
 # The max concurrency reduction (4->2) was required to avoid "did not return clean data"
 # errors from qubesctl. It may be possible to raise this again.
 sudo qubesctl --show-output --max-concurrency 2 --targets "$all_sdw_vms_target" state.highstate
+
+echo "Cleaning up old templates"
+./scripts/destroy-vm "$templates_to_remove"


Same as above: let's port the removal to Salt, so it's enforced even during highstate runs when the provision-all bash script isn't called.

conorsch · 2019-11-27T19:50:01Z

tests/test_dom0_rpm_repo.py

@@ -26,7 +26,7 @@ def test_rpm_repo_config(self):
            "gpgcheck=1",
            "gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-securedrop-workstation-test",  # noqa
            "enabled=1",
-            "baseurl=https://dev-bin.ops.securedrop.org/dom0-rpm-repo/",
+            "baseurl=https://yum-test.securedrop.org/workstation/dom0/f25",


Thank you for fixing this! Heads up, fix also exists in #351.

redshiftzero · 2019-12-02T17:13:07Z

i'm seeing concurrent.futures as uninstalled for python 2 in the buster base template - my understanding is that salt is using this - can someone explain?

conorsch · 2019-12-02T17:21:31Z

@redshiftzero Judging by QubesOS/qubes-issues#4272 (comment), we should re-evaluate whether we must ensure the python futures apt package is present.

conorsch · 2019-12-02T18:04:04Z

After some code spelunking by @rmol, we're satisfied that the Qubes salt mgmt logic is using python3 on the target VMs, meaning that concurrent is no longer necessary.

For all SecureDrop-workstation provisioned VMs, with the exception of Whonix-based templates. This will not replace templates in-place for existing installs. Update tests to reflect new kernel version in buster template (4.14.151) and localname (-workstation suffix).

This is required because sd-proxy uses Whonix-14 (Stretch) based VMs. This should be removed once we move sd-proxy to Whonix-15

This will allow for in-place upgrades for all VMs based on this template.

…mplate swapped, by ensuring the following: 1. The AppVMs to be upgraded are shutdown 2. The AppVMs to be upgraded are not DispVM templates 3. To satisfy (2), the AppVMs that are DispVM templates cannot be the default DispVM template in qubes-prefs

Shut down and remove the following Debian Stretch templates: * sd-svs-template * sd-svs-disp-template * sd-export-template * securedrop-workstation Adds test to ensure the templates have been properly deleted.

Ensures the changes are applied to the correct template

emkll · 2019-12-02T18:57:40Z

rebased on latest master, which now includes #351

This reverts commit 5bc913f.

Leveraging Salt "grains" (equivalent of Ansible "facts") to detect Buster/Stretch intelligently. Whichever the target system is using will be substituted in-place. Saves us the headache of maintaining different Salt state files for backwards compatibility as we drive toward the Stretch -> Buster conversion for all SDW templates.

conorsch

Ran through another test round, changes look great. I have some minor comments in-line that warrant your attention, @emkll, but most are superficial.

I'm requesting changes mostly to block merge so that you can review two commits (one reversion, one addition) related to the apt repo logic. If you agree that's what we want, then up to you to squash/rebase as you see fit.

dom0/fpf-apt-test-repo-stretch.sls

dom0/fpf-apt-test-repo.sls

scripts/handle-upgrade

scripts/prep-salt

tests/test_vms_platform.py

dom0/sd-svs-disp.sls

tests/test_dom0_config.py

emkll · 2019-12-03T18:21:24Z

Thanks @conorsch your changes look good to me, consider those approved. They have been tested in both install and upgrade scenarios: make all and make test pass in both cases (including additions described below)

I've resolved all but one review comment in b9f07e1, commented on that final comment (set -e).

I've also appended two further more commits, for your review, to ensure test coverage of the conditional apt-test logic, as well as improvements to the cleanup logic.

* Move handle-upgrade to dom0/securedrop-handle-upgrade * Replace buster with sd-buster * Invert test logic for template deletion * set -e on securedrop-handle-upgrade script

…/salt * Delete files prefixed by securedrop- and fpf in /srv/salt/

Ensures the securedrop_workstation.list apt source is correctly populated.

emkll · 2019-12-03T18:49:39Z

Just pushed some final changes (see https://github.com/freedomofpress/securedrop-workstation/compare/469a3fe..7e56fca) addressing the comments in #352 (comment)

This is now ready for re-review

conorsch · 2019-12-03T19:08:29Z

Roger that. Proceeding with upgrade test. In preparation, I provisioned from latest master (75de802), and will apply these changes without a clean to verify the upgrade-in-place logic.

conorsch · 2019-12-03T20:01:48Z

All tests passing for the upgrade-in-place scenario. Overall, the process is remarkably smooth. Thanks for the hard work on this one, @emkll.

emkll requested review from conorsch and kushaldas November 27, 2019 14:36

emkll added this to Ready for Review in SecureDrop Team Board Nov 27, 2019

conorsch reviewed Nov 27, 2019

View reviewed changes

emkll mentioned this pull request Nov 28, 2019

More comprehensive "make clean" action #351

Merged

emkll moved this from Ready for Review to In Development in SecureDrop Team Board Nov 28, 2019

emkll force-pushed the 345-debian-buster branch 3 times, most recently from 33b6a1a to 127667a Compare December 2, 2019 16:10

emkll marked this pull request as ready for review December 2, 2019 16:13

emkll moved this from In Development to Ready for Review in SecureDrop Team Board Dec 2, 2019

emkll added 8 commits December 2, 2019 13:56

Temporary: duplicate fpf-apt-test-repo for stretch

5bc913f

This is required because sd-proxy uses Whonix-14 (Stretch) based VMs. This should be removed once we move sd-proxy to Whonix-15

Rename sd-export-template to sd-export-buster-template

0aac7fe

This will allow for in-place upgrades for all VMs based on this template.

Rename sd-svs-disp-template to sd-svs-disp-buster-template

d6edcae

This will allow for in-place upgrades for all VMs based on this template.

Rename sd-svs-template to sd-svs-buster-template

b1975ba

This will allow for in-place upgrades for all VMs based on this template.

Remove old templates after upgrade to buster templates

35c1a20

Shut down and remove the following Debian Stretch templates: * sd-svs-template * sd-svs-disp-template * sd-export-template * securedrop-workstation Adds test to ensure the templates have been properly deleted.

Update buster template names in sd-workstation topfile

b28db2e

Ensures the changes are applied to the correct template

emkll force-pushed the 345-debian-buster branch from 127667a to b28db2e Compare December 2, 2019 18:56

Conor Schaefer added 2 commits December 2, 2019 15:33

Revert "Temporary: duplicate fpf-apt-test-repo for stretch"

49a42e7

This reverts commit 5bc913f.

conorsch suggested changes Dec 3, 2019

View reviewed changes

emkll force-pushed the 345-debian-buster branch from 007c4fd to 469a3fe Compare December 3, 2019 18:13

Address review comments:

58a3a34

* Move handle-upgrade to dom0/securedrop-handle-upgrade * Replace buster with sd-buster * Invert test logic for template deletion * set -e on securedrop-handle-upgrade script

emkll added 2 commits December 3, 2019 13:29

Ensure all securedrop-workstation related files are cleaned from /srv…

099bfd3

…/salt * Delete files prefixed by securedrop- and fpf in /srv/salt/

Add test for conditional apt logic

7e56fca

Ensures the securedrop_workstation.list apt source is correctly populated.

emkll force-pushed the 345-debian-buster branch from 469a3fe to 7e56fca Compare December 3, 2019 18:46

conorsch approved these changes Dec 3, 2019

View reviewed changes

conorsch merged commit 7c19c63 into master Dec 3, 2019

SecureDrop Team Board automation moved this from Ready for Review to Done Dec 3, 2019

emkll mentioned this pull request Dec 4, 2019

Address apt vulnerability CVE-2019-3462 at install-time #249

Closed

conorsch mentioned this pull request Dec 6, 2019

Updates Whonix-based templates 14 -> 15 #358

Merged

emkll deleted the 345-debian-buster branch December 17, 2019 16:14

conorsch mentioned this pull request Oct 7, 2020

Support enforcing state for other VMs #567

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update base templates to buster #352

Update base templates to buster #352

emkll commented Nov 27, 2019 •

edited

conorsch left a comment

conorsch Nov 27, 2019

conorsch Nov 27, 2019

conorsch Nov 27, 2019

conorsch Nov 27, 2019

emkll Nov 29, 2019

conorsch Nov 27, 2019

conorsch Nov 27, 2019

conorsch Nov 27, 2019 •

edited

conorsch Nov 27, 2019

redshiftzero commented Dec 2, 2019

conorsch commented Dec 2, 2019 •

edited

conorsch commented Dec 2, 2019

emkll commented Dec 2, 2019

conorsch left a comment

emkll commented Dec 3, 2019

emkll commented Dec 3, 2019

conorsch commented Dec 3, 2019

conorsch commented Dec 3, 2019

Update base templates to buster #352

Update base templates to buster #352

Conversation

emkll commented Nov 27, 2019 • edited

Testing

Clean scenario

Upgrade scenario

conorsch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

conorsch Nov 27, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redshiftzero commented Dec 2, 2019

conorsch commented Dec 2, 2019 • edited

conorsch commented Dec 2, 2019

emkll commented Dec 2, 2019

conorsch left a comment

Choose a reason for hiding this comment

emkll commented Dec 3, 2019

emkll commented Dec 3, 2019

conorsch commented Dec 3, 2019

conorsch commented Dec 3, 2019

emkll commented Nov 27, 2019 •

edited

conorsch Nov 27, 2019 •

edited

conorsch commented Dec 2, 2019 •

edited