Initial support for Debian Bullseye templates #24

conorsch · 2022-04-13T17:25:15Z

Overview

Towards freedomofpress/securedrop-workstation#733

The work here constitutes a spike for Bullseye (and 4.1) compatibility. As usual, there were some minor touchups required to the build script, to ensure compatibility with the upstream qubes-builder logic (https://github.com/QubesOS/qubes-builder). It's important to note this is just a "spike" because full support will require additional changes, noted below.

Not implemented

rebuilding all component packages for Bullseye (specifically for python3.9, rather than python3.7), and hosting those packages in the FPF apt repo.
adding a prod-signed tag to this repo (thus the scary "TEMPORARY: disables signing verification" commit)
rehosting paxctld package in FPF apt repo (it was pulled from Debian Stable; see Migrate to Bullseye securedrop-workstation#733 (comment) for discussion about options)

Test plan

Confirm that you can build a Bullseye-based template RPM using the steps below. Since there are several changes in other repos, you'll need to build a few packages elsewhere, first.

Check out this kernel builder PR: Add support for 5.15 securedrop-workstation builds kernel-builder#20
Run make securedrop-workstation 5.15, and go grab a coffee. When the build finishes, store the tarball for safe keeping. Copy the linux-image and linux-headers packages to your sd-template-builder VM.
Checkout this packaging PR: [SPIKE] Update workstation-grsec for Bullseye securedrop-builder#301
Update the metapackage version to match the version of the kernel that you built above (it'll be newer than 5.15.33, documented here).
Run PKG_VERSION=5.15.<YOUR_VERSION> PKG_PLATFORM=bullseye make securedrop-workstation-grsec. Copy that metapackage to the sd-template-builder VM.
Fetch the latest version of paxctld from the grsecurity website: https://grsecurity.net/download/ . Copy that package to sd-template-builder.
Check out Initial support for Debian Bullseye templates #24 locally. Edit the package versions and paths in securedrop-workstation/04_install_qubes_post.sh to match what you have locally.
Run make template, grab another coffee.

🟥 Ensure that signatures are verified! That is, remove the commit "TEMPORARY: disable signing verification" (43ecc16).

After the build completes, you'll have a fresh RPM named something like:
qubes-template-securedrop-workstation-bullseye-4.0.6-202204132331.noarch.rpm.

Copy that RPM to dom0: qvm-run -p sd-kernel-builder path/to/qubes-template-securedrop-workstation-bullseye-4.0.6-202204132331.noarch.rpm > qubes-template-securedrop-workstation-bullseye-4.0.6-202204132331.noarch.rpm
Install it: sudo dnf install ./qubes-template-securedrop-workstation-bullseye-4.0.6-202204132331.noarch.rpm
Modify prefs: qvm-prefs securedrop-template-bullseye virt_mode hvm && qvm-prefs securedrop-template-bullseye kernel ''.
Boot VM. Confirm via uname -a that it's using the expected custom kernel version. Run apt-get update and apt-get upgrade`.

Next steps

There aren't any SDW deb packages ready for Bullseye yet, so the virtualenvs for packages like securedrop-client won't work, even if you install them. More work required on that front.

For the Template VM, once we're happy with the logic, we'll need to remove the temporary commits that disable signing verification on this PR, and push a new prod-signed tag. (N.B. great opportunity to use a keyring of SD maintainers, rather than the Release Key!) Then make template will work after verifying the integrity of all the resources it pulls in, including this repo.

sssoleileraaa · 2022-05-10T23:03:26Z

Check out this kernel builder PR: Add support for 5.15 securedrop-workstation builds kernel-builder#20

Run make securedrop-workstation 5.15, and go grab a coffee. When the build finishes, store the tarball for safe keeping. Copy the linux-image and linux-headers packages to your sd-template-builder VM.

Checkout this packaging PR: [SPIKE] Update workstation-grsec for Bullseye securedrop-builder#301

Update the metapackage version to match the version of the kernel that you built above (it'll be newer than 5.15.33, documented here).

Run PKG_VERSION=5.15.<YOUR_VERSION> PKG_PLATFORM=bullseye make securedrop-workstation-grsec. Copy that metapackage to the sd-template-builder VM.

Fetch the latest version of paxctld from the grsecurity website: https://grsecurity.net/download/ . Copy that package to sd-template-builder.

Check out Initial support for Debian Bullseye templates #24 locally. Edit the package versions and paths in securedrop-workstation/04_install_qubes_post.sh to match what you have locally.

So far, so good...

Run make template, grab another coffee.

Error:

scripts/test-sane-mount: line 10: ./test-dev-null: Permission denied
*******************************************************************************
***                               ERROR                                      ***
*** Cannot create chroot because the current filesystem is mounted as nodev. ***
*** Build Qubes on a different filesystem, or run 'make remount' to remount  ***
*** /home with dev option.
***                                                                          ***
*******************************************************************************
make[2]: *** [Makefile.generic:159: generic-prepare-chroot] Error 1
make[1]: *** [Makefile:265: mgmt-salt-vm] Error 1
make[1]: Leaving directory '/home/user/qubes-template-securedrop-workstation/qubes-builder'
make: *** [Makefile:4: template] Error 2

So, I switched over from my fedora-34 qube in 4.1 to my fedora-33 qube in 4.0, and tried again:

Run make template, grab another coffee.

✔️ It worked!

Tomorrow I can move on to the next step:

🟥 Ensure that signatures are verified! That is, remove the commit "TEMPORARY: disable signing verification" (43ecc16).

sssoleileraaa · 2022-05-11T23:26:46Z

I tested this before creating a signed tag and it worked:

Copy that RPM to dom0: qvm-run -p sd-kernel-builder path/to/qubes-template-securedrop-workstation-bullseye-4.0.6-202204132331.noarch.rpm > qubes-template-securedrop-workstation-bullseye-4.0.6-202204132331.noarch.rpm
Install it: sudo dnf install ./qubes-template-securedrop-workstation-bullseye-4.0.6-202204132331.noarch.rpm
Modify prefs: qvm-prefs securedrop-template-bullseye virt_mode hvm && qvm-prefs securedrop-template-bullseye kernel ''.
Boot VM. Confirm via uname -a that it's using the expected custom kernel version. Run apt-get update and apt-get upgrade`.

Confirm via uname -a that it's using the expected custom kernel version. Run apt-get update and apt-get upgrade`.

As described in the test plan, I removed the temporary commits. As described in the commit message, I removed the temporary code changes added here:
d141a30#diff-1c2d8ec1d2b91a056bfae8b35c278c3e942c73caf9c485820674ab80d2dbc5b5R67-R76

So this pr should be cleaned up at this point. I think we can tag the last commit, prod-sign it, and rebuild with signing verification turned on so that I can finish up the test plan:

🟥 Ensure that signatures are verified! That is, remove the commit "TEMPORARY: disable signing verification" (43ecc16).

If there is an issue during this part of the test plan, we can delete the tag, fix, and retry. @eloquence- since @conorsch is away, does this plan make sense to you?

eloquence · 2022-05-12T00:19:10Z

That sounds right, though I'll highlight Conor's comment from the PR:

(N.B. great opportunity to use a keyring of SD maintainers, rather than the Release Key!)

Since this isn't actually a shipped artifact but just a tag used to verify the build, as I understand it, we could sign the prod tag with any maintainer key.

sssoleileraaa · 2022-05-12T00:20:50Z

Since this isn't actually a shipped artifact but just a tag used to verify the build, as I understand it, we could sign the prod tag with any maintainer key.

I didn't see support for this yet, so I figured we could add that feature later rather than block on it. I might be mistaken though! I can try signing with my own key to test and delete the tag if it fails verification.

eloquence · 2022-05-12T00:24:06Z

Based on the README

Import and trust the Qubes Master Key and the SecureDrop Release Signing Key to the local gpg keyring in your sd-template-builder AppVM.

I would guess that you would need to do the same with your developer public key. 🤞

sssoleileraaa · 2022-05-12T01:36:39Z

@eloquence- thanks for the pointers. I tried simply adding my fingerprint to this logic but still saw the following error:

---> No valid signed tag found!
---> One invalid tag: 822d270fe9ca502f089e72eb9fb0aca6574d098c

Just as a sanity check, I signed the tag with the release signing key, see:

[user@sd-template-builder qubes-template-securedrop-workstation]$ git tag -v 0.3.0
object 4768762921e77d3d88c6f9a1439448f015b9ccf5
type commit
tag 0.3.0
tagger Allie Crevier <allie@freedom.press> 1652315122 -0700

qubes-template-securedrop-workstation 0.3.0
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEs7w4of7eC4courm/EXOOCajrhkUFAmJ8U/IACgkQEXOOCajr
hkUYjg//SOGvvGGJmW/RM00PrQmGVmfeZwlQVSav1SBQYLOLcqnAA4gcJyUMVa8m
jCmTRzXqZGflIkCeZQTfDeSkRK3QZ29uEiGCrs59SBLceLVkgmLVi6tYgmAYwHJT
iNrDgT94ZNgu0gJT5Vsq2U3TIDn8yZI3jImW6QM/3YMsmbkRQN7wKOeu3a8IJDN7
yggSgr43nqvUdxAO9K0IRosbBFQQbtKk4157PgWoureVJ1+z6n4P5rJCyCl8HdKE
aUt36KZHHcYrI4E5G1xutz/v7UBcV8QEGK/+pCEn8LoK2ZsezcX23MdrIy4jlqhA
DS9U22w3OhhCnlKsta8bE0QoLQxqJMsbpHALPryi9fbpNzjAqGz2xOGFad5+JkRp
WlXkRKl3s9g3iVGcIG3Ue91nwv4ofOPGvpxcPfpVCLw1M8c6sGKAZWpuANYySgzv
/HSEN8C1Own2gZvyvqTR8bszuGVk1Q3dUtg2Aw7zFWZmUF/xFunoQSrotQUUs8aH
mIDkYDTtqMNwXkUmzOXE9ypliQ32wy76XLVR6c9UoMtSX5/5ea48sW1kuF/Tr0Ys
QiagFq1fqC4K8l6j9Dp7nMtq6/AMNFSCLVXCkbWufzd5xvl/pnJxfQLBcwvmLP7C
bvT4MGp9b4BO1g6afQXu3uRwNKpnukc4gHxGAgUNnV2eeszprbM=
=ZQsA
-----END PGP SIGNATURE-----
gpg: Signature made Wed May 11 21:10:01 2022 EDT
gpg:                using RSA key 2359E6538C0613E652955E6C188EDD3B7B22E6A3
gpg: Good signature from "SecureDrop Release Signing Key <securedrop-release-key-2021@freedom.press>" [ultimate]

And ran into the same issue. Here's my output:

...
gpg: key DDFA1A3E36879494: public key "Qubes Master Signing Key" imported
gpg: key 063938BA42CFA724: public key "Marek Marczykowski-Górecki (Qubes OS signing key) <marmarek@invisiblethingslab.com>" imported
gpg: key DA0434BC706E1FCF: public key "Simon Gaiser (Qubes OS signing key) <simon@invisiblethingslab.com>" imported
gpg: Total number processed: 3
gpg:               imported: 3
gpg: no ultimately trusted keys found
gpg: inserting ownertrust of 6
gpg: key DDFA1A3E36879494: "Qubes Master Signing Key" not changed
gpg: key 063938BA42CFA724: "Marek Marczykowski-Górecki (Qubes OS signing key) <marmarek@invisiblethingslab.com>" not changed
gpg: key DA0434BC706E1FCF: "Simon Gaiser (Qubes OS signing key) <simon@invisiblethingslab.com>" not changed
gpg: Total number processed: 3
gpg:              unchanged: 3
---> Good tag bec85e4d6e431ae75782efb1e7f3acbaf7ce4b8b
--> Switching branch from master branch to master
Reset branch 'master'
Your branch is up to date with 'origin/master'.

-> Updating sources for template-securedrop-workstation...
--> Fetching from https://github.com/freedomofpress/qubes-template-securedrop-workstation main (options: --depth=1)...
--> Verifying tags...
gpg: keybox '/home/user/qubes-template-securedrop-workstation/qubes-builder/keyrings/git/template-securedrop-workstation/pubring.kbx' created
gpg: /home/user/qubes-template-securedrop-workstation/qubes-builder/keyrings/git/template-securedrop-workstation/trustdb.gpg: trustdb created
gpg: key DDFA1A3E36879494: public key "Qubes Master Signing Key" imported
gpg: key 063938BA42CFA724: public key "Marek Marczykowski-Górecki (Qubes OS signing key) <marmarek@invisiblethingslab.com>" imported
gpg: key DA0434BC706E1FCF: public key "Simon Gaiser (Qubes OS signing key) <simon@invisiblethingslab.com>" imported
gpg: Total number processed: 3
gpg:               imported: 3
gpg: no ultimately trusted keys found
gpg: inserting ownertrust of 6
gpg: key DDFA1A3E36879494: "Qubes Master Signing Key" not changed
gpg: key 063938BA42CFA724: "Marek Marczykowski-Górecki (Qubes OS signing key) <marmarek@invisiblethingslab.com>" not changed
gpg: key DA0434BC706E1FCF: "Simon Gaiser (Qubes OS signing key) <simon@invisiblethingslab.com>" not changed
gpg: Total number processed: 3
gpg:              unchanged: 3
---> No valid signed tag found!
---> One invalid tag: 822d270fe9ca502f089e72eb9fb0aca6574d098c
make[1]: *** [Makefile:224: template-securedrop-workstation.get-sources] Error 1
make[1]: Leaving directory '/home/user/qubes-template-securedrop-workstation/qubes-builder'
make: *** [Makefile:4: template] Error 2

At this point, I'm realizing that I need to better understand the upstream tooling for building Qubes templates. This is something I can come back to tomorrow, and maybe get some help from @eaon on. I'll also reach out to the Qubes team in case they can offer help.

sssoleileraaa · 2022-05-17T00:58:44Z

Since the qubes-builder logic expects a signed tag on the most recent HEAD commit of the target branch, and I undid the target branch to point to this PR branch, that would explain the issue.

Time to test this hypothesis! (Update: that did the trick)

As usual, the upstream qubes-builder repo has changed somewhat. We adjust our scripts to match accordingly. Mostly this was simplifying the keyring logic. Added some umount commands, as well, so that multiple repeated builds in the same environment worked cleanly. Touched up the docs somewhat, as well. Signed-off-by: Allie Crevier <allie@freedom.press>

Signed-off-by: Allie Crevier <allie@freedom.press>

sssoleileraaa

Successfully build Bullseye template on Qubes 4.0 with signature validation turned on for the 0.3.0 tag
Successfully build Bullseye template on Qubes 4.1 with signature validation turned on for the 0.3.0 tag

At this point, this is good to merge, but waiting in case @zenmonkeykstop updates the release key again with a different expiration date (sounds like this is still under discussion).

Next up is to build securedrop-keyring with the new release key (see freedomofpress/securedrop-builder#319).

This was referenced Apr 13, 2022

[SPIKE] Update workstation-grsec for Bullseye freedomofpress/securedrop-builder#301

Closed

Add support for 5.15 securedrop-workstation builds freedomofpress/kernel-builder#20

Merged

conorsch mentioned this pull request Apr 25, 2022

Consider preferring Salt customizations to custom template RPM freedomofpress/securedrop-workstation#765

Closed

sssoleileraaa mentioned this pull request May 9, 2022

Cannot open files in disposable VM using SecureDrop Client freedomofpress/securedrop-workstation#766

Closed

sssoleileraaa added this to Under Review in SecureDrop Team Board May 10, 2022

sssoleileraaa mentioned this pull request May 11, 2022

4.1 support: sys-usb DispVM compatibility and sd-log qrexec loopback denial notification suppression freedomofpress/securedrop-workstation#764

Merged

7 tasks

sssoleileraaa force-pushed the bullseye-support-1 branch 2 times, most recently from 1b19cd2 to 4768762 Compare May 11, 2022 23:05

sssoleileraaa force-pushed the bullseye-support-1 branch from 4768762 to a52fc18 Compare May 17, 2022 02:09

Conor Schaefer and others added 2 commits May 16, 2022 19:12

Update release key with new expiration

6250a54

Signed-off-by: Allie Crevier <allie@freedom.press>

sssoleileraaa force-pushed the bullseye-support-1 branch from a52fc18 to 6250a54 Compare May 17, 2022 02:14

sssoleileraaa marked this pull request as ready for review May 17, 2022 02:14

sssoleileraaa approved these changes May 17, 2022

View reviewed changes

sssoleileraaa merged commit 5fb25d1 into main May 17, 2022

SecureDrop Team Board automation moved this from Under Review to Done May 17, 2022

sssoleileraaa deleted the bullseye-support-1 branch May 17, 2022 19:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial support for Debian Bullseye templates #24

Initial support for Debian Bullseye templates #24

conorsch commented Apr 13, 2022 •

edited by gonzalo-bulnes

sssoleileraaa commented May 10, 2022 •

edited

sssoleileraaa commented May 11, 2022 •

edited

eloquence commented May 12, 2022

sssoleileraaa commented May 12, 2022 •

edited

eloquence commented May 12, 2022 •

edited

sssoleileraaa commented May 12, 2022

sssoleileraaa commented May 17, 2022 •

edited

sssoleileraaa left a comment

Initial support for Debian Bullseye templates #24

Initial support for Debian Bullseye templates #24

Conversation

conorsch commented Apr 13, 2022 • edited by gonzalo-bulnes

Overview

Not implemented

Test plan

Next steps

sssoleileraaa commented May 10, 2022 • edited

sssoleileraaa commented May 11, 2022 • edited

eloquence commented May 12, 2022

sssoleileraaa commented May 12, 2022 • edited

eloquence commented May 12, 2022 • edited

sssoleileraaa commented May 12, 2022

sssoleileraaa commented May 17, 2022 • edited

sssoleileraaa left a comment

Choose a reason for hiding this comment

conorsch commented Apr 13, 2022 •

edited by gonzalo-bulnes

sssoleileraaa commented May 10, 2022 •

edited

sssoleileraaa commented May 11, 2022 •

edited

sssoleileraaa commented May 12, 2022 •

edited

eloquence commented May 12, 2022 •

edited

sssoleileraaa commented May 17, 2022 •

edited