baseosmgr/upgradeconverter/volumemgr: enable kvm↔k BaseOs upgrade with blob reuse - #6036
baseosmgr/upgradeconverter/volumemgr: enable kvm↔k BaseOs upgrade with blob reuse#6036eriknordmark wants to merge 5 commits into
Conversation
Adds five testdata files that exercise the EVE-kvm <-> EVE-k cross- flavor baseos upgrade path. The suite validates the EVE-side blob- reuse + volume-aware-gate work in lf-edge/eve#6036 end-to-end: update_eve_image_cross_hv.txt Base test: pushes the cross-HV upgrade with no volumes present and asserts the new BaseOs reaches PartitionState=active on the other flavor. update_eve_image_cross_hv_with_contenttree.txt Pre-stages a ContentTree via `content-tree-add` on EVE-kvm, triggers the upgrade, and asserts pillar's downloader pulled zero additional bytes on EVE-k (blob-reuse). update_eve_image_cross_hv_with_app_recreate.txt Operator-facing recipe: deploy an app on EVE-kvm, set `timer.defer.content.delete=86400`, delete the app, upgrade to EVE-k, redeploy the same app. Asserts no re-download. update_eve_image_cross_hv_app_reboot.txt EVE-k -> EVE-k reboot recovery test. Bringup must be done with HV=k (no cross-HV prelude). After app reaches RUNNING, issue `eden controller edge-node reboot` and assert the app returns to RUNNING within budget. Dumps longhorn/kubevirt diagnostics on failure for issue triage. update_eve_image_cross_hv_app_recreate_with_reboot.txt Combines the app_recreate flow with two consecutive reboots afterward — full reboot survivability after the upgrade. Helpers (snapshot-rcv-bytes.sh, wait-for-app-running.sh, wait-for-volumemgr-ready.sh, wait-for-reboot-and-back.sh, verify-app-functional.sh, etc.) live inline in each testdata file following eden's existing convention. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6036 +/- ##
==========================================
+ Coverage 22.64% 23.05% +0.41%
==========================================
Files 507 518 +11
Lines 93195 95204 +2009
==========================================
+ Hits 21103 21949 +846
- Misses 70367 71331 +964
- Partials 1725 1924 +199 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Adds five testdata files that exercise the EVE-kvm <-> EVE-k cross- flavor baseos upgrade path. The suite validates the EVE-side blob- reuse + volume-aware-gate work in lf-edge/eve#6036 end-to-end: update_eve_image_cross_hv.txt Base test: pushes the cross-HV upgrade with no volumes present and asserts the new BaseOs reaches PartitionState=active on the other flavor. update_eve_image_cross_hv_with_contenttree.txt Pre-stages a ContentTree via `content-tree-add` on EVE-kvm, triggers the upgrade, and asserts pillar's downloader pulled zero additional bytes on EVE-k (blob-reuse). update_eve_image_cross_hv_with_app_recreate.txt Operator-facing recipe: deploy an app on EVE-kvm, set `timer.defer.content.delete=86400`, delete the app, upgrade to EVE-k, redeploy the same app. Asserts no re-download. update_eve_image_cross_hv_app_reboot.txt EVE-k -> EVE-k reboot recovery test. Bringup must be done with HV=k (no cross-HV prelude). After app reaches RUNNING, issue `eden controller edge-node reboot` and assert the app returns to RUNNING within budget. Dumps longhorn/kubevirt diagnostics on failure for issue triage. update_eve_image_cross_hv_app_recreate_with_reboot.txt Combines the app_recreate flow with two consecutive reboots afterward — full reboot survivability after the upgrade. Helpers (snapshot-rcv-bytes.sh, wait-for-app-running.sh, wait-for-volumemgr-ready.sh, wait-for-reboot-and-back.sh, verify-app-functional.sh, etc.) live inline in each testdata file following eden's existing convention. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds five testdata files that exercise the EVE-kvm <-> EVE-k cross- flavor baseos upgrade path. The suite validates the EVE-side blob- reuse + volume-aware-gate work in lf-edge/eve#6036 end-to-end: update_eve_image_cross_hv.txt Base test: pushes the cross-HV upgrade with no volumes present and asserts the new BaseOs reaches PartitionState=active on the other flavor. update_eve_image_cross_hv_with_contenttree.txt Pre-stages a ContentTree via `content-tree-add` on EVE-kvm, triggers the upgrade, and asserts pillar's downloader pulled zero additional bytes on EVE-k (blob-reuse). update_eve_image_cross_hv_with_app_recreate.txt Operator-facing recipe: deploy an app on EVE-kvm, set `timer.defer.content.delete=86400`, delete the app, upgrade to EVE-k, redeploy the same app. Asserts no re-download. update_eve_image_cross_hv_app_reboot.txt EVE-k -> EVE-k reboot recovery test. Bringup must be done with HV=k (no cross-HV prelude). After app reaches RUNNING, issue `eden controller edge-node reboot` and assert the app returns to RUNNING within budget. Dumps longhorn/kubevirt diagnostics on failure for issue triage. update_eve_image_cross_hv_app_recreate_with_reboot.txt Combines the app_recreate flow with two consecutive reboots afterward — full reboot survivability after the upgrade. Helpers (snapshot-rcv-bytes.sh, wait-for-app-running.sh, wait-for-volumemgr-ready.sh, wait-for-reboot-and-back.sh, verify-app-functional.sh, etc.) live inline in each testdata file following eden's existing convention. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Adds five testdata files that exercise the EVE-kvm <-> EVE-k cross- flavor baseos upgrade path. The suite validates the EVE-side blob- reuse + volume-aware-gate work in lf-edge/eve#6036 end-to-end: update_eve_image_cross_hv.txt Base test: pushes the cross-HV upgrade with no volumes present and asserts the new BaseOs reaches PartitionState=active on the other flavor. update_eve_image_cross_hv_with_contenttree.txt Pre-stages a ContentTree via `content-tree-add` on EVE-kvm, triggers the upgrade, and asserts pillar's downloader pulled zero additional bytes on EVE-k (blob-reuse). update_eve_image_cross_hv_with_app_recreate.txt Operator-facing recipe: deploy an app on EVE-kvm, set `timer.defer.content.delete=86400`, delete the app, upgrade to EVE-k, redeploy the same app. Asserts no re-download. update_eve_image_cross_hv_app_reboot.txt EVE-k -> EVE-k reboot recovery test. Bringup must be done with HV=k (no cross-HV prelude). After app reaches RUNNING, issue `eden controller edge-node reboot` and assert the app returns to RUNNING within budget. Dumps longhorn/kubevirt diagnostics on failure for issue triage. update_eve_image_cross_hv_app_recreate_with_reboot.txt Combines the app_recreate flow with two consecutive reboots afterward — full reboot survivability after the upgrade. Helpers (snapshot-rcv-bytes.sh, wait-for-app-running.sh, wait-for-volumemgr-ready.sh, wait-for-reboot-and-back.sh, verify-app-functional.sh, etc.) live inline in each testdata file following eden's existing convention. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
|
Claude finished @rene's task in 5m 46s —— View job Code Review
Two findings posted as inline comments:
Everything else looks solid: the transitive GC-ref walk in |
4f811dc to
7d3b68d
Compare
Adds five testdata files that exercise the EVE-kvm <-> EVE-k cross- flavor baseos upgrade path. The suite validates the EVE-side blob- reuse + volume-aware-gate work in lf-edge/eve#6036 end-to-end: update_eve_image_cross_hv.txt Base test: pushes the cross-HV upgrade with no volumes present and asserts the new BaseOs reaches PartitionState=active on the other flavor. update_eve_image_cross_hv_with_contenttree.txt Pre-stages a ContentTree via `content-tree-add` on EVE-kvm, triggers the upgrade, and asserts pillar's downloader pulled zero additional bytes on EVE-k (blob-reuse). update_eve_image_cross_hv_with_app_recreate.txt Operator-facing recipe: deploy an app on EVE-kvm, set `timer.defer.content.delete=86400`, delete the app, upgrade to EVE-k, redeploy the same app. Asserts no re-download. update_eve_image_cross_hv_app_reboot.txt EVE-k -> EVE-k reboot recovery test. Bringup must be done with HV=k (no cross-HV prelude). After app reaches RUNNING, issue `eden controller edge-node reboot` and assert the app returns to RUNNING within budget. Dumps longhorn/kubevirt diagnostics on failure for issue triage. update_eve_image_cross_hv_app_recreate_with_reboot.txt Combines the app_recreate flow with two consecutive reboots afterward — full reboot survivability after the upgrade. Helpers (snapshot-rcv-bytes.sh, wait-for-app-running.sh, wait-for-volumemgr-ready.sh, wait-for-reboot-and-back.sh, verify-app-functional.sh, etc.) live inline in each testdata file following eden's existing convention. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
caa58cb to
0a9d6cc
Compare
c7670f8 to
caf1544
Compare
dfc896e to
943bd36
Compare
943bd36 to
5f52856
Compare
Rebuild on current master with the restructured PR stack (lf-edge#6121 squashed to 8 commits, lf-edge#6036 to 5, lf-edge#6063 reordered/squashed to 9). lf-edge#6120 has merged to master and is now inherited rather than merged explicitly.
5f52856 to
225c772
Compare
225c772 to
d4386ef
Compare
|
@andrewd-zededa please re-view. I've addressed your blocking comment. |
After zedagent restarts pubVolumeConfig, volumemgr has reconciled its on-disk recovery (populateExistingVolumesFormat*) with the controller- intended set into VolumeStatus. Signal restart on pubVolumeStatus at the tail of handleVolumeRestart so subscribers can use Restarted() as a gate on the published set being in a known state, rather than Synchronized() which only reflects the socket-level handshake. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cross-flavor BaseOs upgrade between EVE-k and non-EVE-k was unconditionally rejected. The rationale is that /persist/vault/volumes layout differs across flavors, so existing volumes can break across the switch. On a device with no volumes there is nothing to disturb, so the block is overly strict. Subscribe to VolumeConfig (from zedagent) and VolumeStatus (from volumemgr) in baseosmgr and only error out on the flavor mismatch when either set is non-empty. Wait at startup for both publications to signal restart so the first BaseOsConfig after boot evaluates against the true volume set; Synchronized() is not sufficient because it only confirms the socket handshake. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pillar's containerd client uses namespace eve-user-apps on EVE-kvm and k8s.io on EVE-k (pkg/pillar/containerd/containerd.go init()). The user containerd's content store at /persist/vault/containerd survives the kvm→k upgrade, but the bolt metadata DB records blob ownership per namespace, so pillar on EVE-k sees the store as empty under k8s.io and re-downloads every blob that EVE-kvm had previously fetched. Each blob's on-disk file gets overwritten with bit-identical content on a fresh content.WriteBlob; pre-staged ContentTrees burn full registry bandwidth on every upgrade. Add a post-vault converter that runs once on the first EVE-k boot to copy blob metadata records from eve-user-apps to k8s.io directly in the bolt DB at /persist/vault/containerd/io.containerd.metadata.v1.bolt. Metadata-only, no blob copies. The post-vault phase fits cleanly: the vault is decrypted but the user containerd has not yet been started by the kube container, so the bolt DB is idle and safe to mutate via bbolt (already vendored). A sentinel file at /persist/vault/ containerd/.eve-namespace-port-done short-circuits subsequent boots. No-op on EVE-kvm and on fresh-install EVE-k that has no prior eve-user-apps records. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
populateInitBlobStatus() reconstructs BlobStatus from the user containerd's content store on volumemgr startup so that blobs already on /persist (e.g. those pulled before an EVE-kvm → EVE-k upgrade) are recognized and don't trigger a re-download. On EVE-k it filters by the EVEDownloadedLabel because the user containerd is shared with k3s and we must not publish BlobStatus for k3s/longhorn/etc. pod images. Pillar's CAS writes EVEDownloadedLabel only on manifest-shaped blobs (and on the named image entry), not on the individual layer blobs the manifest references. The manifest carries containerd.io/gc.ref.content.* labels that point at every layer — the same set containerd's GC uses to keep layers alive while the manifest is reachable. Filtering by the label per-blob therefore rejects every layer with "Ignoring the blob ... not downloaded by eve", forcing volumemgr to re-fetch each layer even though they're physically present on /persist. Walk those labels transitively from each directly-labeled blob to recover the full set pillar considers its own. Foreign manifests (k3s pod images) don't carry the label so they don't seed the walk — no false positives for shared blobs. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
A device whose /persist is ZFS and that is converted in-field from EVE-kvm to EVE-k carries over a vault stored as a filesystem dataset, but EVE-k expects the vault as a zvol formatted ext4 alongside a separate etcd-storage zvol. The carried-over filesystem vault therefore never reached a mounted state on the first EVE-k boot, so attestation escrow could not complete and the device reboot-looped back to kvm. On the first EVE-k boot the unlock path now detects that the vault is still a filesystem dataset, mounts it so the device boots, and migrates it in place to the zvol layout: it stages a new zvol, copies the vault contents into it (containerd content store and metadata, downloader, verifier, and configs all live inside the vault, so blobs and content trees come across), then destroys the old dataset and renames the new zvol into place. An empty etcd-storage zvol is created since etcd starts fresh on EVE-k. The staging zvol survives until the rename and remount succeed, so an interrupted attempt is reconstructed on the next boot. The migration helper is parameterized on encryption: the TPM path creates the new zvols encrypted with the staged unlock key, the no-TPM SetupDefaultVault path creates them unencrypted (matching how no-TPM ZFS vaults are created), so both paths end up with the same zvol layout. This covers a ZFS device carrying content trees and blobs but no app volumes; app-volume conversion is handled separately. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
d4386ef to
c23959e
Compare
Rebuild on current lf-edge/eve master. Component heads refreshed to the restacked branches (lf-edge#6036, fork#6, lf-edge#6063, fork#4); the whole stack is layered on the same rebased bases so the merges are clean. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rebuild on current lf-edge/eve master. Component heads refreshed to the restacked branches (lf-edge#6036, fork#6, lf-edge#6063, fork#4); the whole stack is layered on the same rebased bases so the merges are clean. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Description
Enable cross-HV-flavor BaseOs upgrade (EVE-kvm ↔ EVE-k) on devices
with no live app volumes, make the post-upgrade BlobStatus
reconstruction work for the EVE-k variant, and migrate a carried-over
ZFS vault to the zvol layout EVE-k expects.
Note that this assumes that EVE-k fits in the IMGx partition otherwise it is rejected. Later work will add the partition resizing as part of an EVE-kvm to EVE-k upgrade.
The blocker was a four-part problem:
baseosmgr rejected the flavor change unconditionally. Now it
subscribes to
VolumeConfigandVolumeStatusand only rejectsthe cross-flavor upgrade when at least one of the two sets is
non-empty (i.e. there are volumes whose on-disk layout could
diverge between flavors). The wait for both
Restarted()isnecessary because
Synchronized()only signals the sockethandshake, not that zedagent and volumemgr have re-published.
volumemgrnow signals restart onVolumeStatusfor that waitto converge.
Boot-time
populateInitBlobStatusdiscarded the blobs. OnEVE-k the user-containerd metadata lives in the
k8s.ionamespace, but the CAS records the kvm-side
eve-user-appsnamespace from the prior BaseOs.
upgradeconverter's newpost-vault
containerd_namespace.gophase ports the relevantblob + image records across namespaces directly in
meta.db(read-only-style: writes to the new namespace, leaves the old
one in place) so the eve-k containerd sees the pre-upgrade
blobs at startup.
accepted-blobsGC ref was too narrow. The filter onlycounted blobs with the
EVEDownloadedLabel, which meanttransitively GC-referenced layers (parent manifest declares them
via
containerd.io/gc.ref.content.*) were discarded. The acceptrule now walks those refs so the full DAG survives the
namespace port.
A carried-over ZFS vault never mounted on EVE-k. On a device
whose
/persistis ZFS, EVE-kvm stores the vault as a filesystemdataset, but EVE-k expects it as an ext4-formatted zvol alongside
a separate etcd-storage zvol. The carried-over filesystem vault
therefore never reached a mounted state on the first EVE-k boot,
attestation escrow could not complete, and the device reboot-looped
back to kvm. The first-boot unlock path now detects the
filesystem-dataset vault, mounts it so the device boots, and
migrates it in place to the zvol layout: it stages a new encrypted
zvol, copies the vault contents into it (the containerd content
store and metadata, downloader, verifier, and configs all live
inside the vault, so blobs and content trees come across), destroys
the old dataset, and renames the new zvol into place. An empty
etcd-storage zvol is created since etcd starts fresh on EVE-k. The
staging zvol survives until the rename and remount succeed, so an
interrupted migration is reconstructed on the next boot. The no-TPM
boot path performs the same migration unencrypted, so a converted
no-TPM device also gains the etcd-storage zvol and zvol vault
layout. This covers a ZFS device carrying content trees and blobs
but no app volumes; app-volume conversion is handled separately.
PR dependencies
None. The six commits in this PR are self-contained.
How to test and validate this PR
Blob reuse across the flavor change (ext4 or ZFS
/persist). Astandalone eden test draft lives in
/tmp/eden-test-drafts/update_eve_image_cross_hv_with_app_recreate.txton the author's workstation and validates the full operator-facing
recipe end-to-end:
timer.defer.content.delete=86400(24h)./run/volumemgr/VolumeMgrStatus/volumemgr.jsonwith"Initialized":true, redeploy the SAME app on eve-k.the redeploy. The eclient blobs (~44 MiB) were reused from CAS
via the transitive-label-walk and namespace-port paths.
ZFS vault fs→zvol migration. On a device with ZFS
/persistcarrying content trees and blobs (no app volumes), run the kvm→k
conversion under eden + swtpm and confirm that after the first EVE-k
boot the vault comes up on a zvol (rather than reboot-looping back to
kvm) and that
VaultStatus.UnlockMethodreflects a successfulTPM-sealed unlock. This path was validated end-to-end under eden +
swtpm.
Unit coverage:
pkg/pillar/cmd/upgradeconverter/containerd_namespace_test.gocoversthe namespace port in isolation against a synthetic bbolt store.
pkg/pillar/cmd/volumemgr/blob_test.gocovers the transitivelabel walk for the GC-ref filter.
pkg/pillar/vault/handler_zfs_test.gocovers the ZFS vaultfs-dataset detection and zvol migration.
Changelog notes
An in-field EVE-kvm → EVE-k BaseOs upgrade on a device with no live
app volumes now succeeds without re-downloading already-present
container images, and a device with a ZFS
/persistmigrates itsvault to the layout EVE-k expects instead of reboot-looping back to
EVE-kvm.
NOTE that this will only work if the IMGx partitions are already large enough thus it is mostly useful if e.g., 17.0.0 EVE-kvm has been installed and the user want to move to 17.X.Y or later EVE-k. For older starting points than 17.0.0 EVE-kvm please wait for follow-on work which will resize partitions.
PR Backports
This is new EVE-k in-field conversion functionality that is not
present in the current LTS branches.
Checklist
And the last but not least:
check them.