Build and test Workstation base template RPM with updated securedrop-keyring package #887

rocodes · 2023-05-17T18:36:41Z

Once the Bullseye version of securedrop-keyring is available as per freedomofpress/securedrop-builder#443, a new base template should be built and tested containing it.

zenmonkeykstop · 2023-06-05T16:22:35Z

first steps:

try existing build process with qubes-builder 1.0
if build not good, change up build process to be more like Qubes team's current process.

rocodes · 2023-06-08T15:30:52Z

Initial update:

Building using our qubes-template-securedrop-workstation convenience script does not quite work; the build gets stuck and errors out at the create chroot stage, and following the suggested steps to fix does not resolve the issue while using our script. (The error that appears is also an older error, first noticed in 2020 and ostensibly remedied in the qubes-builder repo already..)

log

make[1]: Leaving directory '/home/user/qubes-template-securedrop-workstation/qubes-builder'
make[1]: Entering directory '/home/user/qubes-template-securedrop-workstation/qubes-builder'
Makefile:223: target 'builder-debian.get-sources' given more than once in the same rule
Makefile:223: target 'mgmt-salt.get-sources' given more than once in the same rule
Makefile:225: target 'builder-debian.get-sources-extra' given more than once in the same rule
Makefile:225: target 'mgmt-salt.get-sources-extra' given more than once in the same rule
Makefile:264: target 'builder-debian-vm' given more than once in the same rule
Makefile:264: target 'mgmt-salt-vm' given more than once in the same rule
Makefile:272: target 'builder-debian-dom0' given more than once in the same rule
Makefile:272: target 'mgmt-salt-dom0' given more than once in the same rule
Makefile:473: target 'mgmt-salt.clean' given more than once in the same rule
Makefile:628: target 'builder-debian.grep' given more than once in the same rule
Makefile:628: target 'mgmt-salt.grep' given more than once in the same rule
Currently installed dependencies:
createrepo_c-0.21.1-1.fc37.x86_64
createrepo_c-0.21.1-1.fc37.x86_64
debootstrap-1.0.127-2.fc37.noarch
devscripts-2.22.2-3.fc37.x86_64
dialog-1.3-44.20220526.fc37.x86_64
dnf-plugins-core-4.4.1-1.fc37.noarch
dpkg-dev-1.21.21-1.fc37.noarch
git-2.40.1-1.fc37.x86_64
perl-Digest-MD5-2.58-489.fc37.x86_64
perl-Digest-SHA-6.03-1.fc37.x86_64
python3-pyyaml-6.0-5.fc37.x86_64
python3-sh-1.14.2-7.fc37.noarch
rpm-build-4.18.1-2.fc37.x86_64
rpmdevtools-9.6-2.fc37.noarch
systemd-container-251.14-2.fc37.x86_64
scripts/test-sane-mount: line 10: ./test-dev-null: Permission denied
*******************************************************************************
***                               ERROR                                      ***
*** Cannot create chroot because the current filesystem is mounted as nodev. ***
*** Build Qubes on a different filesystem, or run 'make remount' to remount  ***
*** /home with dev option.
***                                                                          ***
*******************************************************************************
make[2]: *** [Makefile.generic:159: generic-prepare-chroot] Error 1
make[1]: *** [Makefile:265: mgmt-salt-vm] Error 1
make[1]: Leaving directory '/home/user/qubes-template-securedrop-workstation/qubes-builder'
make: *** [Makefile:4: template] Error 2
++ __vte_prompt_command
+++ HISTTIMEFORMAT=
+++ history 1
+++ sed 's/^ *[0-9]\+ *//'
++ local 'command=make template'

I was able to make more progress however by continuing with the qubes-builder (v1) repo directly, since by this point our configuration files had already been put in place. I did the following:

run our make template script til it fails
cd qubes-builder, and run make remount as suggested in the error that was printed. (The rest of the steps are also from within the qubes-builder directory)
grep for securedrop-workstation to assure yourself that the ws files (really the .conf file) have already been copied into the qubes-builder directory
make install-deps (they should already be installed, this completes quickly)
make get-sources (they should already be downloaded, this completes quickly)
make qubes-vm (will take a while)
confirm make qubes-vm completes successfully, then:
make template

I am still running into issues with the make template command, which also seem related to the chroot/file permissions, so maybe they will be easy to solve:

log

[snip]
Makefile:628: target 'mgmt-salt.grep' given more than once in the same rule
-> Preparing buster reprepro environment (install reprepro on the host to avoid this)
Extracting templates from packages: 100%
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (19: No such device)
Moving old data out of the way
Running in chroot, ignoring request.
invoke-rc.d: policy-rc.d denied execution of start.
Adding system-user for exim (v4)
ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
exim: DB upgrade, deleting hints-db
Created symlink /etc/systemd/system/multi-user.target.wants/atd.service -> /lib/systemd/system/atd.service.
Running in chroot, ignoring request.
invoke-rc.d: policy-rc.d denied execution of start.
Building database of manual pages ...
Created symlink /etc/systemd/system/timers.target.wants/man-db.timer -> /lib/systemd/system/man-db.timer.
Running in chroot, ignoring request.
invoke-rc.d: policy-rc.d denied execution of start.
Initializing GnuTLS DH parameter file

Creating config file /etc/perl/XML/SAX/ParserDetails.ini with new version
Replacing config file /etc/perl/XML/SAX/ParserDetails.ini with new version
Replacing config file /etc/perl/XML/SAX/ParserDetails.ini with new version
Warning: apt-key output should not be parsed (stdout is not a terminal)
'qubes-mgmt-salt_4.1.16-1+deb11u1_amd64.buildinfo' looks like architecture 'amd64', but this is not listed in the Architecture-Header!
Ignoring as --ignore=surprisingarch given.
'qubes-mgmt-salt_4.1.16-1+deb11u1_amd64.buildinfo' looks like architecture 'amd64', but this is not listed in the Architecture-Header!
Ignoring as --ignore=surprisingarch given.
Skipping inclusion of 'qubes-mgmt-salt' '4.1.16-1+deb11u1' in 'bullseye|main|amd64', as it has already '4.1.16-1+deb11u1'.
Skipping inclusion of 'qubes-mgmt-salt-vm-connector' '4.1.16-1+deb11u1' in 'bullseye|main|amd64', as it has already '4.1.16-1+deb11u1'.
Skipping inclusion of 'qubes-mgmt-salt-dom0' '4.1.16-1+deb11u1' in 'bullseye|main|amd64', as it has already '4.1.16-1+deb11u1'.
Skipping inclusion of 'qubes-mgmt-salt-dom0-formulas' '4.1.16-1+deb11u1' in 'bullseye|main|amd64', as it has already '4.1.16-1+deb11u1'.
Skipping inclusion of 'qubes-mgmt-salt-config' '4.1.16-1+deb11u1' in 'bullseye|main|amd64', as it has already '4.1.16-1+deb11u1'.
Skipping inclusion of 'qubes-mgmt-salt' '4.1.16-1+deb11u1' in 'bullseye|main|source', as it has already '4.1.16-1+deb11u1'.
'qubes-mgmt-salt-base_4.1.5-1+deb11u1_amd64.buildinfo' looks like architecture 'amd64', but this is not listed in the Architecture-Header!
Ignoring as --ignore=surprisingarch given.
'qubes-mgmt-salt-base-topd_4.1.3-1+deb11u1_amd64.buildinfo' looks like architecture 'amd64', but this is not listed in the Architecture-Header!
Ignoring as --ignore=surprisingarch given.
'qubes-mgmt-salt-base-config_4.1.1-1+deb11u1_amd64.buildinfo' looks like architecture 'amd64', but this is not listed in the Architecture-Header!
Ignoring as --ignore=surprisingarch given.
Makefile:223: target 'builder-debian.get-sources' given more than once in the same rule
Makefile:223: target 'builder-debian.get-sources' given more than once in the same rule
Makefile:223: target 'template-securedrop-workstation.get-sources' given more than once in the same rule
Makefile:223: target 'builder-rpm.get-sources' given more than once in the same rule
Makefile:223: target 'builder-debian.get-sources' given more than once in the same rule
Makefile:223: target 'mgmt-salt.get-sources' given more than once in the same rule
Makefile:223: target 'mgmt-salt.get-sources' given more than once in the same rule
Makefile:225: target 'builder-debian.get-sources-extra' given more than once in the same rule
Makefile:225: target 'builder-debian.get-sources-extra' given more than once in the same rule
Makefile:225: target 'template-securedrop-workstation.get-sources-extra' given more than once in the same rule
Makefile:225: target 'builder-rpm.get-sources-extra' given more than once in the same rule
Makefile:225: target 'builder-debian.get-sources-extra' given more than once in the same rule
Makefile:225: target 'mgmt-salt.get-sources-extra' given more than once in the same rule
Makefile:225: target 'mgmt-salt.get-sources-extra' given more than once in the same rule
Makefile:264: target 'builder-debian-vm' given more than once in the same rule
Makefile:264: target 'builder-debian-vm' given more than once in the same rule
Makefile:264: target 'template-securedrop-workstation-vm' given more than once in the same rule
Makefile:264: target 'builder-rpm-vm' given more than once in the same rule
Makefile:264: target 'builder-debian-vm' given more than once in the same rule
Makefile:264: target 'mgmt-salt-vm' given more than once in the same rule
Makefile:264: target 'mgmt-salt-vm' given more than once in the same rule
Makefile:272: target 'builder-debian-dom0' given more than once in the same rule
Makefile:272: target 'builder-debian-dom0' given more than once in the same rule
Makefile:272: target 'template-securedrop-workstation-dom0' given more than once in the same rule
Makefile:272: target 'builder-rpm-dom0' given more than once in the same rule
Makefile:272: target 'builder-debian-dom0' given more than once in the same rule
Makefile:272: target 'mgmt-salt-dom0' given more than once in the same rule
Makefile:272: target 'mgmt-salt-dom0' given more than once in the same rule
Makefile:473: target 'template-securedrop-workstation.clean' given more than once in the same rule
Makefile:473: target 'mgmt-salt.clean' given more than once in the same rule
Makefile:473: target 'mgmt-salt.clean' given more than once in the same rule
Makefile:628: target 'builder-debian.grep' given more than once in the same rule
Makefile:628: target 'builder-debian.grep' given more than once in the same rule
Makefile:628: target 'template-securedrop-workstation.grep' given more than once in the same rule
Makefile:628: target 'builder-rpm.grep' given more than once in the same rule
Makefile:628: target 'builder-debian.grep' given more than once in the same rule
Makefile:628: target 'mgmt-salt.grep' given more than once in the same rule
Makefile:628: target 'mgmt-salt.grep' given more than once in the same rule
-> Building template bullseye (logfile: build-logs/template-bullseye.log)...
make: *** [Makefile:352: template-local-bullseye+securedrop-workstation] Error 1
[user@sd-template-builder qubes-builder]$ echo $?
2

I also looked at porting the buillder.conf file to the qubes builderv2 yaml format to test the builderv2 repo if these issues aren't resolvable (although it seems they should be). That is still in progress.

rocodes · 2023-06-08T15:41:06Z

This upstream issue gives us the biggest clue: QubesOS/qubes-issues#5263 (comment) Even though there is a make remount script in qubes builder now, it doesn't seem to have done the trick in our case. Trying the build again after remounting manually, will update :)

rocodes · 2023-06-12T16:05:14Z

I was not able to successfully build a template, still running into permissions issues that look to be chroot-related. @eaon has latest updates from last week.

eaon · 2023-06-16T16:55:45Z

After having had repeated mount related trouble in upstream dependencies, we pivoted by coming up with a non-standard procedure that for all intents and purposes should not be any different than a template that was updated in-flight.

Instead of building a new template from the ground up, we instead did the following:

Downloaded our own template
Unpacked its contents
Assembled the root image of the template
Mounted the root image
Used chroot to update the system contained within
Repackaged the updated image

Since this wasn't scripted but a series of commands, it doesn't really fit anywhere, so I created a gist describing what's being done and why.

Details of the actual build by @rocodes can be found in freedomofpress/build-logs@a3539dc

As I pointed out in my comment in freedomofpress/securedrop-yum-test#48, the installed packages for an updated in-flight template are exactly the same as what's contained in the package.

rocodes · 2023-06-16T18:07:50Z

Thank you so much for testing and instruction @eaon :) Just a heads up that when I tried to install the template on my machine, I ran into this upstream issue:
QubesOS/qubes-issues#6297

Won't affect vanilla SDW installs but for folks installing lots of other templates we should keep up with this issue from a support perspective.

eaon · 2023-06-20T16:00:18Z

Left a note in the issue you referenced so that the whole community may benefit from our new insights 🙂

rocodes · 2023-06-20T19:11:11Z

Here are some instructions for anyone testing the rpm, which is now on yum-test.securedrop.org

Test Plan

Hardware:
Scenario: Clean install
Prep:

Install Qubes 4.1.2 and perform system updates.
Next we'll be following a modified version of the prod boostrap (instructions). Download the staging dom0 config package, which is signed with the test key, verify it, and install it in dom0 using dnf install per prod instructions.
Finish the rest of the prod instructions for configuring your workstation (key fingerprint, onion particulars, etc).
Modify config.json to change the environment to staging.
Run sdw-admin --apply.

Testing

Installation completes successfully
/etc/yum.repos.d/securedrop-workstation.repo in dom0 is pointing to yum-test.securedrop.org (Just checking to make sure test template rpm is installed)
Basic functionality is available in qubes based off securedrop-workstation-bullseye templates (boot, shut down, transfer files between VMs where RPC policies allow, for example: logs from other VMs are entering sd-log; sd-log can copy files to any VM you tag with sd-receive-logs, provided you accept the RPC permissions prompt)

rocodes · 2023-06-21T00:07:00Z

I removed @nathandyer's assignment since there was actually one step missing from the test plan (manually edit config.json) and I didn't catch it quick enough, and Nathan's install was already underway (so he ended up with a prod install using the old template rpm, not a staging install using the new one).

Test plan

Hardware: Thinkpad Carbon X1 6gen
Install: clean install
Installation completes successfully
Verified new template is installed (keyring package in securedrop-workstation-bullseye is 0.2.1+bullseye)
Basic functionality is available/no obvious issues - will do more testing tomorrow

nathandyer · 2023-06-21T15:43:31Z

@rocodes Thanks! I'm happy to step through another reinstall with the edited test plan if that's at all helpful - it's not too far out of the way for me to leave things cooking in the background while I'm doing other things. Just let me know if that's of interest!

rocodes · 2023-06-23T01:15:54Z

@nathandyer Thanks for your offer, no need at present time - but don't worry, we'll make use of your prod setup for the dom0 config package update coming soon. I would never let a good prod workstation go to waste. ;)

rocodes · 2023-06-26T17:37:50Z

Now that freedomofpress/securedrop-yum-prod#43 is up, we will close this issue in favour of the release tracking issue:
#888

This was referenced May 17, 2023

Update the release pubkey to version with expiry date of 2024-07-08 freedomofpress/securedrop#6794

Closed

Release new Bullseye securedrop-keyring package freedomofpress/securedrop-builder#443

Closed

zenmonkeykstop transferred this issue from freedomofpress/securedrop May 18, 2023

rocodes mentioned this issue May 18, 2023

Release Workstation base template RPM with updated securedrop-keyring package #888

Closed

zenmonkeykstop assigned eaon and rocodes Jun 5, 2023

rocodes mentioned this issue Jun 6, 2023

Update release pubkey freedomofpress/qubes-template-securedrop-workstation#30

Closed

rocodes mentioned this issue Jun 16, 2023

Add qubes-template-securedrop-workstation-bullseye-4.1-20230615 freedomofpress/securedrop-yum-test#48

Merged

4 tasks

eaon assigned nathandyer Jun 20, 2023

rocodes unassigned nathandyer Jun 21, 2023

rocodes mentioned this issue Jun 23, 2023

Deploy updated qubes template to yum-qa freedomofpress/securedrop-yum-prod#43

Merged

5 tasks

rocodes closed this as completed Jun 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build and test Workstation base template RPM with updated securedrop-keyring package #887

Build and test Workstation base template RPM with updated securedrop-keyring package #887

rocodes commented May 17, 2023 •

edited by zenmonkeykstop

Loading

zenmonkeykstop commented Jun 5, 2023

rocodes commented Jun 8, 2023

rocodes commented Jun 8, 2023

rocodes commented Jun 12, 2023

eaon commented Jun 16, 2023

rocodes commented Jun 16, 2023 •

edited

Loading

eaon commented Jun 20, 2023

rocodes commented Jun 20, 2023 •

edited

Loading

rocodes commented Jun 21, 2023

nathandyer commented Jun 21, 2023

rocodes commented Jun 23, 2023

rocodes commented Jun 26, 2023

Build and test Workstation base template RPM with updated securedrop-keyring package #887

Build and test Workstation base template RPM with updated securedrop-keyring package #887

Comments

rocodes commented May 17, 2023 • edited by zenmonkeykstop Loading

zenmonkeykstop commented Jun 5, 2023

rocodes commented Jun 8, 2023

rocodes commented Jun 8, 2023

rocodes commented Jun 12, 2023

eaon commented Jun 16, 2023

rocodes commented Jun 16, 2023 • edited Loading

eaon commented Jun 20, 2023

rocodes commented Jun 20, 2023 • edited Loading

Test Plan

rocodes commented Jun 21, 2023

Test plan

nathandyer commented Jun 21, 2023

rocodes commented Jun 23, 2023

rocodes commented Jun 26, 2023

rocodes commented May 17, 2023 •

edited by zenmonkeykstop

Loading

rocodes commented Jun 16, 2023 •

edited

Loading

rocodes commented Jun 20, 2023 •

edited

Loading