feat(os): the migration hold — chain services start only after the slot commits - #883
Merged
Merged
Conversation
…ot commits The deadlock rule's automatic-fallback half (the floor guard shipped earlier): a data_migration bundle's forward-only lmdb migration must not run before the A/B commit decision, or a failed health check leaves a box that can neither commit nor cleanly fall back. os-update now leaves a version-stamped marker on /data when a flagged bundle installs (and clears a stale one on a plain install). On the next boot, pithead-boot sees a marker matching its own version and brings the stack up without the chain services — monerod, tari and their wallets, the lmdb holders and the only containers anything depends on together — via PITHEAD_HOLD_CHAIN=1, which subsets the compose service list. doctor reads the same marker and judges the deliberately-held chain containers by the miners' sync-hold rule, so the commit gate gates on everything that IS running instead of deadlocking on the hold it gates. Post-commit the marker is consumed and a plain up starts the chain: the migration runs on a slot a fallback can no longer leave. A mismatched marker is a fallback boot onto untouched data and holds nothing. Found live while deploying: the appliance image ships no unsquashfs, so rauc info could not read any bundle's manifest and every os-update guard was dead on the appliance — CI never saw it because the stack tests stub rauc. squashfs-tools joins the rootfs shell-out set. The KVM battery's provision phase grows a migration leg: build a flagged bundle, install through the real os-update, and prove from the boot journal and a podman poll that monerod never runs pre-commit, starts post-release, and the marker is consumed. The db_schema manifest field stays out until something reads it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first run of the leg failed exactly where the output was discarded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s it The tier-4 migration leg's first live run refuted os_bundle_meta: RAUC 1.11's --output-format=json omits the manifest's [meta.*] sections entirely, so every real bundle read as unstamped and os-update refused it as a possible downgrade. The stack tests passed because their fixtures were synthetic JSON in a shape real rauc never emits — the second dead layer under the missing-unsquashfs one, and exactly why the leg installs through the real os-update. --output-format=shell carries the meta (RAUC_META_PITHEAD_*) and still verifies the bundle signature before printing. Fixtures now mirror the real format, and both os-update harnesses sandbox the migration-marker path so a test run can never touch a real /data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Tier-4 evidence is in — the migration leg ran on the KVM bench (gouda) against the real image: The leg's first two runs each refuted a layer of |
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #851 (the automatic-fallback half; the floor guard shipped in #863)
What
A
data_migrationbundle's forward-only lmdb migration must not run before the A/B commit decision — otherwise a failed health gate leaves a box that can neither commit nor cleanly fall back (dual-distribution-plan.mdrisk #6). This lands the runner:os-updateleaves a version-stamped marker on/data(.os-migration-pending) when a flagged bundle installs; a non-migrating install clears a stale one (it supersedes a migrating install that never booted).pithead-boot, on a boot whose version matches the marker, brings the stack up without the chain services — monerod, tari and their wallets, the lmdb holders and the only containers anything depends on together (PITHEAD_HOLD_CHAIN=1subsets the compose service list). A mismatched marker is a fallback boot onto untouched data and holds nothing.doctorreads the same marker and judges the deliberately-held chain containers by the miners' sync-hold rule — the commit gate gates on everything that IS running instead of deadlocking on the very hold it gates. A running-but-unhealthy chain container still fails.upstarts the chain — the migration runs on a slot a fallback can no longer leave. If the chain start fails, the slot stays committed and the next (normal-path) boot retries a plainup.Found live on the bench, not by CI
Deploying today's develop-v2 tip through
pithead os-updatefailed outright: the appliance image ships nounsquashfs, sorauc infocould not extract any bundle's manifest — every guard #863 added was dead on the actual appliance (CI's stack tests stub rauc, so they can't see it).squashfs-toolsjoins the rootfs shell-out set, with the lesson recorded in the Dockerfile comment.Tests
os_update(written with the bundle version; cleared by a plain install), theos_migration_hold_activequery (match / mismatch / absent / empty), the verdict's chain-hold arm (down→ok, running-unhealthy→still fail, miners unaffected), and a black-boxpithead upunder the hold asserting the compose invocation starts tor/p2pool/dashboard and none of the four chain services. 2136 passed, 0 failed;make lintgreen.os-updateon the guest (exercising unsquashfs for real), and prove from the boot journal plus a podman poll that monerod never runs pre-commit, the release line lands, monerod starts after it, and the marker is consumed. This leg has not executed yet — it runs on gouda only; I'm running the battery next and will report before merging.The
db_schemamanifest field the plan lists stays out until something reads it — an unread field is a claim, not a contract (doc updated accordingly).Adversarially verified in a fresh context: fallback-path traces (gate-never-passes, crash between mark-good and marker removal — self-healing re-hold on an already-good slot), subset-grep exactness both directions, no
PITHEAD_HOLD_CHAINleak into operator commands, doctor deadlock check.🤖 Generated with Claude Code