Skip to content

baseosmgr/upgradeconverter/volumemgr: enable kvm↔k BaseOs upgrade with blob reuse - #6036

Open
eriknordmark wants to merge 5 commits into
lf-edge:masterfrom
eriknordmark:kvm-k-baseos-upgrade-blob-reuse
Open

baseosmgr/upgradeconverter/volumemgr: enable kvm↔k BaseOs upgrade with blob reuse#6036
eriknordmark wants to merge 5 commits into
lf-edge:masterfrom
eriknordmark:kvm-k-baseos-upgrade-blob-reuse

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Description

Enable cross-HV-flavor BaseOs upgrade (EVE-kvm ↔ EVE-k) on devices
with no live app volumes, make the post-upgrade BlobStatus
reconstruction work for the EVE-k variant, and migrate a carried-over
ZFS vault to the zvol layout EVE-k expects.

Note that this assumes that EVE-k fits in the IMGx partition otherwise it is rejected. Later work will add the partition resizing as part of an EVE-kvm to EVE-k upgrade.

The blocker was a four-part problem:

  1. baseosmgr rejected the flavor change unconditionally. Now it
    subscribes to VolumeConfig and VolumeStatus and only rejects
    the cross-flavor upgrade when at least one of the two sets is
    non-empty (i.e. there are volumes whose on-disk layout could
    diverge between flavors). The wait for both Restarted() is
    necessary because Synchronized() only signals the socket
    handshake, not that zedagent and volumemgr have re-published.
    volumemgr now signals restart on VolumeStatus for that wait
    to converge.

  2. Boot-time populateInitBlobStatus discarded the blobs. On
    EVE-k the user-containerd metadata lives in the k8s.io
    namespace, but the CAS records the kvm-side eve-user-apps
    namespace from the prior BaseOs. upgradeconverter's new
    post-vault containerd_namespace.go phase ports the relevant
    blob + image records across namespaces directly in meta.db
    (read-only-style: writes to the new namespace, leaves the old
    one in place) so the eve-k containerd sees the pre-upgrade
    blobs at startup.

  3. accepted-blobs GC ref was too narrow. The filter only
    counted blobs with the EVEDownloadedLabel, which meant
    transitively GC-referenced layers (parent manifest declares them
    via containerd.io/gc.ref.content.*) were discarded. The accept
    rule now walks those refs so the full DAG survives the
    namespace port.

  4. A carried-over ZFS vault never mounted on EVE-k. On a device
    whose /persist is ZFS, EVE-kvm stores the vault as a filesystem
    dataset, but EVE-k expects it as an ext4-formatted zvol alongside
    a separate etcd-storage zvol. The carried-over filesystem vault
    therefore never reached a mounted state on the first EVE-k boot,
    attestation escrow could not complete, and the device reboot-looped
    back to kvm. The first-boot unlock path now detects the
    filesystem-dataset vault, mounts it so the device boots, and
    migrates it in place to the zvol layout: it stages a new encrypted
    zvol, copies the vault contents into it (the containerd content
    store and metadata, downloader, verifier, and configs all live
    inside the vault, so blobs and content trees come across), destroys
    the old dataset, and renames the new zvol into place. An empty
    etcd-storage zvol is created since etcd starts fresh on EVE-k. The
    staging zvol survives until the rename and remount succeed, so an
    interrupted migration is reconstructed on the next boot. The no-TPM
    boot path performs the same migration unencrypted, so a converted
    no-TPM device also gains the etcd-storage zvol and zvol vault
    layout. This covers a ZFS device carrying content trees and blobs
    but no app volumes; app-volume conversion is handled separately.

PR dependencies

None. The six commits in this PR are self-contained.

How to test and validate this PR

Blob reuse across the flavor change (ext4 or ZFS /persist). A
standalone eden test draft lives in
/tmp/eden-test-drafts/update_eve_image_cross_hv_with_app_recreate.txt
on the author's workstation and validates the full operator-facing
recipe end-to-end:

  1. Deploy an eclient app on EVE-kvm; wait RUNNING + SSH-functional.
  2. Set timer.defer.content.delete=86400 (24h).
  3. Delete the app + wait for its volume to fully clear (~10 min).
  4. Push the kvm→k baseos upgrade.
  5. After eve-k boots and volumemgr publishes
    /run/volumemgr/VolumeMgrStatus/volumemgr.json with
    "Initialized":true, redeploy the SAME app on eve-k.
  6. Assert: pillar's downloader received 0 additional bytes for
    the redeploy. The eclient blobs (~44 MiB) were reused from CAS
    via the transitive-label-walk and namespace-port paths.
  7. SSH-probe the redeployed app (Ubuntu /etc/issue check).

ZFS vault fs→zvol migration. On a device with ZFS /persist
carrying content trees and blobs (no app volumes), run the kvm→k
conversion under eden + swtpm and confirm that after the first EVE-k
boot the vault comes up on a zvol (rather than reboot-looping back to
kvm) and that VaultStatus.UnlockMethod reflects a successful
TPM-sealed unlock. This path was validated end-to-end under eden +
swtpm.

Unit coverage:

  • pkg/pillar/cmd/upgradeconverter/containerd_namespace_test.go covers
    the namespace port in isolation against a synthetic bbolt store.
  • pkg/pillar/cmd/volumemgr/blob_test.go covers the transitive
    label walk for the GC-ref filter.
  • pkg/pillar/vault/handler_zfs_test.go covers the ZFS vault
    fs-dataset detection and zvol migration.

Changelog notes

An in-field EVE-kvm → EVE-k BaseOs upgrade on a device with no live
app volumes now succeeds without re-downloading already-present
container images, and a device with a ZFS /persist migrates its
vault to the layout EVE-k expects instead of reboot-looping back to
EVE-kvm.

NOTE that this will only work if the IMGx partitions are already large enough thus it is mostly useful if e.g., 17.0.0 EVE-kvm has been installed and the user want to move to 17.X.Y or later EVE-k. For older starting points than 17.0.0 EVE-kvm please wait for follow-on work which will resize partitions.

PR Backports

This is new EVE-k in-field conversion functionality that is not
present in the current LTS branches.

  • 16.0-stable: No.
  • 14.5-stable: No.
  • 13.4-stable: No.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

eriknordmark added a commit to eriknordmark/eden that referenced this pull request Jun 11, 2026
Adds five testdata files that exercise the EVE-kvm <-> EVE-k cross-
flavor baseos upgrade path. The suite validates the EVE-side blob-
reuse + volume-aware-gate work in lf-edge/eve#6036 end-to-end:

  update_eve_image_cross_hv.txt
    Base test: pushes the cross-HV upgrade with no volumes present
    and asserts the new BaseOs reaches PartitionState=active on the
    other flavor.

  update_eve_image_cross_hv_with_contenttree.txt
    Pre-stages a ContentTree via `content-tree-add` on EVE-kvm,
    triggers the upgrade, and asserts pillar's downloader pulled
    zero additional bytes on EVE-k (blob-reuse).

  update_eve_image_cross_hv_with_app_recreate.txt
    Operator-facing recipe: deploy an app on EVE-kvm, set
    `timer.defer.content.delete=86400`, delete the app, upgrade
    to EVE-k, redeploy the same app. Asserts no re-download.

  update_eve_image_cross_hv_app_reboot.txt
    EVE-k -> EVE-k reboot recovery test. Bringup must be done with
    HV=k (no cross-HV prelude). After app reaches RUNNING, issue
    `eden controller edge-node reboot` and assert the app returns
    to RUNNING within budget. Dumps longhorn/kubevirt diagnostics
    on failure for issue triage.

  update_eve_image_cross_hv_app_recreate_with_reboot.txt
    Combines the app_recreate flow with two consecutive reboots
    afterward — full reboot survivability after the upgrade.

Helpers (snapshot-rcv-bytes.sh, wait-for-app-running.sh,
wait-for-volumemgr-ready.sh, wait-for-reboot-and-back.sh,
verify-app-functional.sh, etc.) live inline in each testdata file
following eden's existing convention.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@codecov

codecov Bot commented Jun 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 34.78261% with 195 lines in your changes missing coverage. Please review.
✅ Project coverage is 23.05%. Comparing base (74ffe8e) to head (c23959e).
⚠️ Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/vault/handler_zfs.go 2.22% 88 Missing ⚠️
pkg/pillar/cmd/baseosmgr/baseosmgr.go 0.00% 44 Missing ⚠️
...illar/cmd/upgradeconverter/containerd_namespace.go 71.68% 20 Missing and 12 partials ⚠️
pkg/pillar/zfs/zfs.go 0.00% 13 Missing ⚠️
pkg/pillar/cmd/volumemgr/blob.go 72.41% 7 Missing and 1 partial ⚠️
pkg/pillar/cmd/baseosmgr/handlebaseos.go 0.00% 5 Missing ⚠️
pkg/pillar/cmd/volumemgr/handlevolume.go 0.00% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6036      +/-   ##
==========================================
+ Coverage   22.64%   23.05%   +0.41%     
==========================================
  Files         507      518      +11     
  Lines       93195    95204    +2009     
==========================================
+ Hits        21103    21949     +846     
- Misses      70367    71331     +964     
- Partials     1725     1924     +199     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

eriknordmark added a commit to eriknordmark/eden that referenced this pull request Jun 11, 2026
Adds five testdata files that exercise the EVE-kvm <-> EVE-k cross-
flavor baseos upgrade path. The suite validates the EVE-side blob-
reuse + volume-aware-gate work in lf-edge/eve#6036 end-to-end:

  update_eve_image_cross_hv.txt
    Base test: pushes the cross-HV upgrade with no volumes present
    and asserts the new BaseOs reaches PartitionState=active on the
    other flavor.

  update_eve_image_cross_hv_with_contenttree.txt
    Pre-stages a ContentTree via `content-tree-add` on EVE-kvm,
    triggers the upgrade, and asserts pillar's downloader pulled
    zero additional bytes on EVE-k (blob-reuse).

  update_eve_image_cross_hv_with_app_recreate.txt
    Operator-facing recipe: deploy an app on EVE-kvm, set
    `timer.defer.content.delete=86400`, delete the app, upgrade
    to EVE-k, redeploy the same app. Asserts no re-download.

  update_eve_image_cross_hv_app_reboot.txt
    EVE-k -> EVE-k reboot recovery test. Bringup must be done with
    HV=k (no cross-HV prelude). After app reaches RUNNING, issue
    `eden controller edge-node reboot` and assert the app returns
    to RUNNING within budget. Dumps longhorn/kubevirt diagnostics
    on failure for issue triage.

  update_eve_image_cross_hv_app_recreate_with_reboot.txt
    Combines the app_recreate flow with two consecutive reboots
    afterward — full reboot survivability after the upgrade.

Helpers (snapshot-rcv-bytes.sh, wait-for-app-running.sh,
wait-for-volumemgr-ready.sh, wait-for-reboot-and-back.sh,
verify-app-functional.sh, etc.) live inline in each testdata file
following eden's existing convention.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eden that referenced this pull request Jun 11, 2026
Adds five testdata files that exercise the EVE-kvm <-> EVE-k cross-
flavor baseos upgrade path. The suite validates the EVE-side blob-
reuse + volume-aware-gate work in lf-edge/eve#6036 end-to-end:

  update_eve_image_cross_hv.txt
    Base test: pushes the cross-HV upgrade with no volumes present
    and asserts the new BaseOs reaches PartitionState=active on the
    other flavor.

  update_eve_image_cross_hv_with_contenttree.txt
    Pre-stages a ContentTree via `content-tree-add` on EVE-kvm,
    triggers the upgrade, and asserts pillar's downloader pulled
    zero additional bytes on EVE-k (blob-reuse).

  update_eve_image_cross_hv_with_app_recreate.txt
    Operator-facing recipe: deploy an app on EVE-kvm, set
    `timer.defer.content.delete=86400`, delete the app, upgrade
    to EVE-k, redeploy the same app. Asserts no re-download.

  update_eve_image_cross_hv_app_reboot.txt
    EVE-k -> EVE-k reboot recovery test. Bringup must be done with
    HV=k (no cross-HV prelude). After app reaches RUNNING, issue
    `eden controller edge-node reboot` and assert the app returns
    to RUNNING within budget. Dumps longhorn/kubevirt diagnostics
    on failure for issue triage.

  update_eve_image_cross_hv_app_recreate_with_reboot.txt
    Combines the app_recreate flow with two consecutive reboots
    afterward — full reboot survivability after the upgrade.

Helpers (snapshot-rcv-bytes.sh, wait-for-app-running.sh,
wait-for-volumemgr-ready.sh, wait-for-reboot-and-back.sh,
verify-app-functional.sh, etc.) live inline in each testdata file
following eden's existing convention.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
eriknordmark added a commit to eriknordmark/eden that referenced this pull request Jun 15, 2026
Adds five testdata files that exercise the EVE-kvm <-> EVE-k cross-
flavor baseos upgrade path. The suite validates the EVE-side blob-
reuse + volume-aware-gate work in lf-edge/eve#6036 end-to-end:

  update_eve_image_cross_hv.txt
    Base test: pushes the cross-HV upgrade with no volumes present
    and asserts the new BaseOs reaches PartitionState=active on the
    other flavor.

  update_eve_image_cross_hv_with_contenttree.txt
    Pre-stages a ContentTree via `content-tree-add` on EVE-kvm,
    triggers the upgrade, and asserts pillar's downloader pulled
    zero additional bytes on EVE-k (blob-reuse).

  update_eve_image_cross_hv_with_app_recreate.txt
    Operator-facing recipe: deploy an app on EVE-kvm, set
    `timer.defer.content.delete=86400`, delete the app, upgrade
    to EVE-k, redeploy the same app. Asserts no re-download.

  update_eve_image_cross_hv_app_reboot.txt
    EVE-k -> EVE-k reboot recovery test. Bringup must be done with
    HV=k (no cross-HV prelude). After app reaches RUNNING, issue
    `eden controller edge-node reboot` and assert the app returns
    to RUNNING within budget. Dumps longhorn/kubevirt diagnostics
    on failure for issue triage.

  update_eve_image_cross_hv_app_recreate_with_reboot.txt
    Combines the app_recreate flow with two consecutive reboots
    afterward — full reboot survivability after the upgrade.

Helpers (snapshot-rcv-bytes.sh, wait-for-app-running.sh,
wait-for-volumemgr-ready.sh, wait-for-reboot-and-back.sh,
verify-app-functional.sh, etc.) live inline in each testdata file
following eden's existing convention.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
@eriknordmark
eriknordmark requested review from andrewd-zededa, rene and zedi-pramodh and removed request for OhmSpectator June 19, 2026 15:34
@eriknordmark
eriknordmark marked this pull request as ready for review June 19, 2026 15:35
@rene

rene commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

@claude

@claude

claude Bot commented Jun 19, 2026

Copy link
Copy Markdown

Claude finished @rene's task in 5m 46s —— View job


Code Review

  • Gather PR diff and context
  • Review baseosmgr.go changes
  • Review handlebaseos.go changes
  • Review containerd_namespace.go (new file)
  • Review upgradeconverter.go changes
  • Review blob.go changes
  • Review handlevolume.go changes
  • Post inline comments and summary

Two findings posted as inline comments:

  1. containerd_namespace.go:106 — a non-IsNotExist error from os.Stat (e.g. permission denied on the vault path) is silently swallowed; the code falls through to bolt.Open which produces a second, less informative error.

  2. handlevolume.go:208SignalRestarted is called unconditionally on every handleVolumeRestart invocation. If zedagent crashes and restarts, the handler fires again and increments pubVolumeStatus's restart counter a second time. Currently benign (baseosmgr's Restarted() is a one-way flag and no RestartHandler is registered on the subscriber), but a if restartCount == 1 guard would match the stated intent and avoid confusing any monitoring of the restart counter.

Everything else looks solid: the transitive GC-ref walk in transitivelyEVEDownloaded is correct, the bolt namespace port is ACID-safe (transaction rolls back on error; sentinel is written only on success; idempotent on re-run), and the hasVolumes OR condition correctly catches the transient state where VolumeConfig is empty but VolumeStatus is still draining a deletion. Test coverage for both the bolt porter and the transitive-walk filter is thorough.

Comment thread pkg/pillar/cmd/upgradeconverter/containerd_namespace.go
Comment thread pkg/pillar/cmd/volumemgr/handlevolume.go Outdated
Comment thread pkg/pillar/cmd/baseosmgr/handlebaseos.go
@eriknordmark
eriknordmark force-pushed the kvm-k-baseos-upgrade-blob-reuse branch from 4f811dc to 7d3b68d Compare June 20, 2026 18:44
@github-actions
github-actions Bot requested a review from OhmSpectator June 20, 2026 18:44
@eriknordmark
eriknordmark marked this pull request as draft June 22, 2026 09:54
eriknordmark added a commit to eriknordmark/eden that referenced this pull request Jul 1, 2026
Adds five testdata files that exercise the EVE-kvm <-> EVE-k cross-
flavor baseos upgrade path. The suite validates the EVE-side blob-
reuse + volume-aware-gate work in lf-edge/eve#6036 end-to-end:

  update_eve_image_cross_hv.txt
    Base test: pushes the cross-HV upgrade with no volumes present
    and asserts the new BaseOs reaches PartitionState=active on the
    other flavor.

  update_eve_image_cross_hv_with_contenttree.txt
    Pre-stages a ContentTree via `content-tree-add` on EVE-kvm,
    triggers the upgrade, and asserts pillar's downloader pulled
    zero additional bytes on EVE-k (blob-reuse).

  update_eve_image_cross_hv_with_app_recreate.txt
    Operator-facing recipe: deploy an app on EVE-kvm, set
    `timer.defer.content.delete=86400`, delete the app, upgrade
    to EVE-k, redeploy the same app. Asserts no re-download.

  update_eve_image_cross_hv_app_reboot.txt
    EVE-k -> EVE-k reboot recovery test. Bringup must be done with
    HV=k (no cross-HV prelude). After app reaches RUNNING, issue
    `eden controller edge-node reboot` and assert the app returns
    to RUNNING within budget. Dumps longhorn/kubevirt diagnostics
    on failure for issue triage.

  update_eve_image_cross_hv_app_recreate_with_reboot.txt
    Combines the app_recreate flow with two consecutive reboots
    afterward — full reboot survivability after the upgrade.

Helpers (snapshot-rcv-bytes.sh, wait-for-app-running.sh,
wait-for-volumemgr-ready.sh, wait-for-reboot-and-back.sh,
verify-app-functional.sh, etc.) live inline in each testdata file
following eden's existing convention.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
@eriknordmark
eriknordmark force-pushed the kvm-k-baseos-upgrade-blob-reuse branch 2 times, most recently from caa58cb to 0a9d6cc Compare July 3, 2026 03:31
@eriknordmark eriknordmark changed the title baseosmgr/upgradeconverter/volumemgr: enable kvm↔k BaseOs upgrade with blob reuse [WIP] baseosmgr/upgradeconverter/volumemgr: enable kvm↔k BaseOs upgrade with blob reuse Jul 6, 2026
@eriknordmark
eriknordmark force-pushed the kvm-k-baseos-upgrade-blob-reuse branch from c7670f8 to caf1544 Compare July 9, 2026 22:04
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 11, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 11, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 14, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 14, 2026
@eriknordmark
eriknordmark force-pushed the kvm-k-baseos-upgrade-blob-reuse branch from dfc896e to 943bd36 Compare July 14, 2026 23:13
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 14, 2026
@eriknordmark
eriknordmark force-pushed the kvm-k-baseos-upgrade-blob-reuse branch from 943bd36 to 5f52856 Compare July 15, 2026 00:53
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 15, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 15, 2026
Rebuild on current master with the restructured PR stack (lf-edge#6121 squashed
to 8 commits, lf-edge#6036 to 5, lf-edge#6063 reordered/squashed to 9). lf-edge#6120 has merged
to master and is now inherited rather than merged explicitly.
@eriknordmark
eriknordmark force-pushed the kvm-k-baseos-upgrade-blob-reuse branch from 5f52856 to 225c772 Compare July 15, 2026 05:21
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 15, 2026
@eriknordmark
eriknordmark force-pushed the kvm-k-baseos-upgrade-blob-reuse branch from 225c772 to d4386ef Compare July 16, 2026 23:17
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 17, 2026
@eriknordmark eriknordmark changed the title [WIP] baseosmgr/upgradeconverter/volumemgr: enable kvm↔k BaseOs upgrade with blob reuse baseosmgr/upgradeconverter/volumemgr: enable kvm↔k BaseOs upgrade with blob reuse Jul 20, 2026
@eriknordmark
eriknordmark marked this pull request as ready for review July 20, 2026 21:09
@eriknordmark

Copy link
Copy Markdown
Contributor Author

@andrewd-zededa please re-view. I've addressed your blocking comment.

eriknordmark and others added 5 commits July 20, 2026 15:06
After zedagent restarts pubVolumeConfig, volumemgr has reconciled its
on-disk recovery (populateExistingVolumesFormat*) with the controller-
intended set into VolumeStatus. Signal restart on pubVolumeStatus at the
tail of handleVolumeRestart so subscribers can use Restarted() as a gate
on the published set being in a known state, rather than Synchronized()
which only reflects the socket-level handshake.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cross-flavor BaseOs upgrade between EVE-k and non-EVE-k was
unconditionally rejected. The rationale is that /persist/vault/volumes
layout differs across flavors, so existing volumes can break across the
switch. On a device with no volumes there is nothing to disturb, so the
block is overly strict.

Subscribe to VolumeConfig (from zedagent) and VolumeStatus (from
volumemgr) in baseosmgr and only error out on the flavor mismatch when
either set is non-empty. Wait at startup for both publications to signal
restart so the first BaseOsConfig after boot evaluates against the true
volume set; Synchronized() is not sufficient because it only confirms
the socket handshake.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pillar's containerd client uses namespace eve-user-apps on EVE-kvm and
k8s.io on EVE-k (pkg/pillar/containerd/containerd.go init()). The user
containerd's content store at /persist/vault/containerd survives the
kvm→k upgrade, but the bolt metadata DB records blob ownership per
namespace, so pillar on EVE-k sees the store as empty under k8s.io
and re-downloads every blob that EVE-kvm had previously fetched.
Each blob's on-disk file gets overwritten with bit-identical content
on a fresh content.WriteBlob; pre-staged ContentTrees burn full
registry bandwidth on every upgrade.

Add a post-vault converter that runs once on the first EVE-k boot to
copy blob metadata records from eve-user-apps to k8s.io directly in
the bolt DB at /persist/vault/containerd/io.containerd.metadata.v1.bolt.
Metadata-only, no blob copies. The post-vault phase fits cleanly: the
vault is decrypted but the user containerd has not yet been started by
the kube container, so the bolt DB is idle and safe to mutate via
bbolt (already vendored). A sentinel file at /persist/vault/
containerd/.eve-namespace-port-done short-circuits subsequent boots.
No-op on EVE-kvm and on fresh-install EVE-k that has no prior
eve-user-apps records.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
populateInitBlobStatus() reconstructs BlobStatus from the user
containerd's content store on volumemgr startup so that blobs already
on /persist (e.g. those pulled before an EVE-kvm → EVE-k upgrade) are
recognized and don't trigger a re-download. On EVE-k it filters by the
EVEDownloadedLabel because the user containerd is shared with k3s and
we must not publish BlobStatus for k3s/longhorn/etc. pod images.

Pillar's CAS writes EVEDownloadedLabel only on manifest-shaped blobs
(and on the named image entry), not on the individual layer blobs the
manifest references. The manifest carries containerd.io/gc.ref.content.*
labels that point at every layer — the same set containerd's GC uses
to keep layers alive while the manifest is reachable. Filtering by the
label per-blob therefore rejects every layer with "Ignoring the blob
... not downloaded by eve", forcing volumemgr to re-fetch each layer
even though they're physically present on /persist.

Walk those labels transitively from each directly-labeled blob to
recover the full set pillar considers its own. Foreign manifests
(k3s pod images) don't carry the label so they don't seed the walk —
no false positives for shared blobs.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
A device whose /persist is ZFS and that is converted in-field from
EVE-kvm to EVE-k carries over a vault stored as a filesystem dataset,
but EVE-k expects the vault as a zvol formatted ext4 alongside a
separate etcd-storage zvol. The carried-over filesystem vault therefore
never reached a mounted state on the first EVE-k boot, so attestation
escrow could not complete and the device reboot-looped back to kvm.

On the first EVE-k boot the unlock path now detects that the vault is
still a filesystem dataset, mounts it so the device boots, and migrates
it in place to the zvol layout: it stages a new zvol, copies the vault
contents into it (containerd content store and metadata, downloader,
verifier, and configs all live inside the vault, so blobs and content
trees come across), then destroys the old dataset and renames the new
zvol into place. An empty etcd-storage zvol is created since etcd starts
fresh on EVE-k. The staging zvol survives until the rename and remount
succeed, so an interrupted attempt is reconstructed on the next boot.

The migration helper is parameterized on encryption: the TPM path
creates the new zvols encrypted with the staged unlock key, the no-TPM
SetupDefaultVault path creates them unencrypted (matching how no-TPM ZFS
vaults are created), so both paths end up with the same zvol layout.

This covers a ZFS device carrying content trees and blobs but no app
volumes; app-volume conversion is handled separately.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@eriknordmark
eriknordmark force-pushed the kvm-k-baseos-upgrade-blob-reuse branch from d4386ef to c23959e Compare July 20, 2026 22:07
@eriknordmark eriknordmark added the main-quest The fate of the project rests on this PR. Prioritise review to advance the storyline! label Jul 20, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 20, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 20, 2026
Rebuild on current lf-edge/eve master. Component heads refreshed to the
restacked branches (lf-edge#6036, fork#6, lf-edge#6063, fork#4); the whole stack is
layered on the same rebased bases so the merges are clean.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 20, 2026
Rebuild on current lf-edge/eve master. Component heads refreshed to the
restacked branches (lf-edge#6036, fork#6, lf-edge#6063, fork#4); the whole stack is
layered on the same rebased bases so the merges are clean.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

main-quest The fate of the project rests on this PR. Prioritise review to advance the storyline!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants