fix: deflake //rs/tests/consensus/upgrade:upgrade_downgrade_old_nns_subnet_test_head_nns_colocate - #10569
Conversation
…ubnet_test The upgrade system tests assert that, after an upgrade, the previous boot's journal (`journalctl --boot=-1`) contains the orchestrator's "Orchestrator shut down gracefully" message. GuestOS uses A/B slots with separate encrypted /var partitions, and the target /var is wiped on upgrade. The previous boot's journal is carried across the reboot by `transfer_log_state()` in setup-var-encryption.sh, which copies the newest journal files from the old /var into the freshly-created new /var. That copy raced with the fstab-generated `var.mount` unit: both mount /dev/mapper/var_crypt. When var.mount won, the script's `mount /dev/mapper/var_crypt /mnt/var_new` failed with "already mounted or mount point busy", and because the script runs under `set -e` it aborted before `transfer_log_state` ran. The previous boot's journal (including the graceful-shutdown line) was therefore not preserved, and `journalctl --boot=-1` could not find it, failing the test. This was confirmed in the logs of all recent flaky runs (the failing node always logged the "already mounted" error). Make the journal transfer resilient to the race: try to mount var_crypt at /mnt/var_new, and if it is already mounted at /var (var.mount won), copy the logs there instead. Only unmount /mnt/var_new when we actually created that mount, and never abort the boot on a failed mount.
There was a problem hiding this comment.
Pull request overview
This PR deflakes a GuestOS upgrade system test by making the “previous boot” journal transfer resilient to a race between the setup script and the fstab-generated var.mount unit, ensuring the prior boot’s journal (read via journalctl --boot=-1) is preserved across A/B-slot upgrades.
Changes:
- Updates
transfer_log_stateto accept the destination/varmountpoint as an argument. - Makes mounting
/dev/mapper/var_cryptat/mnt/var_newbest-effort; if it’s already mounted at/var, copies logs there instead. - Only unmounts
/mnt/var_newwhen this script mounted it.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if mount /dev/mapper/var_crypt /mnt/var_new; then | ||
| var_new=/mnt/var_new | ||
| var_new_mounted_by_us=1 | ||
| elif awk '$2 == "/var" { found = 1 } END { exit !found }' /proc/mounts; then |
There was a problem hiding this comment.
Good catch — addressed in 61ccd55. I now confirm the device too, but compare the canonical device via readlink -f rather than matching the literal /dev/mapper/var_crypt string: the flaky-run logs show the system refers to it as var_crypt (dm-1) and the dm number varies across boots, so /proc/mounts may spell the source as either /dev/mapper/var_crypt or /dev/dm-N. Comparing canonical paths handles both spellings and avoids silently skipping the transfer (which would re-introduce the flake).
Address review feedback: the fallback that copies the previous boot's journal into the already-mounted /var only checked that *something* was mounted at /var, not that it was var_crypt. Verify the device too, comparing canonical device paths via `readlink -f` since /proc/mounts may name it /dev/mapper/var_crypt or /dev/dm-N.
|
Closed in favour of #10572. |
Summary
Deflakes the upgrade system tests that assert the orchestrator logs
Orchestrator shut down gracefullybefore an upgrade reboot (read back from theprevious boot via
journalctl --boot=-1).The orchestrator was not at fault — it logs the message every time. The
message was being lost from the persistent journal before the test could
read it.
Root cause (confirmed from flaky-run logs)
GuestOS uses A/B slots with separate encrypted
/varpartitions, and thetarget
/varis wiped on upgrade. The previous boot's journal is carried acrossthe reboot by
transfer_log_state()inic-os/components/guestos/init/setup-encryption/setup-var-encryption.sh, whichcopies the newest
.journalfiles from the old/varinto the freshly-creatednew
/varon first boot.That copy mounts
/dev/mapper/var_cryptat/mnt/var_new, which races withthe fstab-generated
var.mountunit that mounts the same device at/var(the device binding was loosened in #9984 via
x-systemd.device-bound=false,letting
var.mountfire as soon asvar_cryptappears — before the scriptreaches its copy step).
When
var.mountwins, the script's mount fails:Because the script runs under
set -e, it aborts beforetransfer_log_stateruns, so the previous boot's journal — including the graceful-shutdown line —
is never preserved.
journalctl --boot=-1then can't find the message and thetest fails.
This was confirmed across all 4 recent flaky runs: the failing node always
logged the
already mounted or mount point busyerror, while passingboots copied
system.journalsuccessfully.Fix
Make the journal transfer resilient to the race instead of aborting:
transfer_log_statenow takes the destination/varmountpoint as an argument.var_cryptat/mnt/var_new; if that fails becausevar.mountalready mounted it at/var, it copies the logs into/varinstead. It only unmounts
/mnt/var_newwhen it actually created that mount,and never aborts the boot on a failed mount.
This PR was created following the steps in
.claude/skills/fix-flaky-tests/SKILL.md.