bootc status failed with error `Reading deployment metadata: Missing base image ref ostree/container/blob/e05a2`

[henrywang]

This issue is for anaconda installation only (bib anaconda-iso included), bootc install does not have this issue. Test result: https://gitlab.com/redhat/rhel/bifrost/tests/bootc-image/-/issues/219

Deploy bootc image with anaconda kickstart and boot the system. System can be booted without error. Check ostree status with rpm-ostree successful. But check with bootc status failed.

$ sudo bootc status
ERROR: Status: Computing status: Booted deployment: Reading deployment metadata: Missing base image ref ostree/container/blob/e05a222db71963792f1e74c3ca70a0009ae55e8794de45e8f8d7c9cea416936e"

$ rpm-ostree status
Deployments:
    ● ostree-unverified-registry:192.168.100.1:5000/hidden:22t5
                       Digest: sha256:d1aa7635c6fa50c70e8cfd110e229d1a2329a48609616e1bc585b9c8ed8bf3f4
                      Version: 9.20240920.0 (2024-09-20T21:59:00Z)

1. This issue has been found since bootc-202409192026.g85b2ca5256-1.el9.x86_64.
2. The bootc-202409191923.geea3996f8f-1.el9.x86_64 does not have this issue.
3. Check bootc commit from eea3996f8f to 85b2ca5256. The a8737d9 might be the root case.

[cgwalters]

Yes I saw this on one machine too, it's really concerning.

We may have broken the semantics for how we write refs as part of the upgrade in ostreedev/ostree-rs-ext#663

[cgwalters]

PR in ostreedev/ostree-rs-ext#666

[cgwalters]

And youki-dev/oci-spec-rs#229

[cgwalters]

Should be fixed by #804 - thanks for the report.

[henrywang]

Verified on bootc-202409242020.g34e104d217-1.el9.x86_64.

[cgwalters]

This issue is for anaconda installation only (bib anaconda-iso included),

To explain the reason for this (it's important) - it's because the way Anaconda works today we rely on the version of the ostree-container code which is embedded in the ISO (ref rhinstaller/anaconda#5197 ). So we have version skew.

But we really need to test the version-skew case in general, i.e. we want an "upgrade from stable" test that needs to gate PRs on this repo by default probably.

[henrywang]

But we really need to test the version-skew case in general, i.e. we want an "upgrade from stable" test that needs to gate PRs on this repo by default probably.

Ack. It's in my list. The test flow would be start vm from fedora/cs-bootc image, and upgrade bootc with dnf upgrade (after bootc usr-overlay), then try bootc status. Adding this test into integration test is more reasonable.

[henrywang]

To explain the reason for this (it's important) - it's because the way Anaconda works today we rely on the version of the ostree-container code which is embedded in the ISO (ref rhinstaller/anaconda#5197 ). So we have version skew.

Hi @cgwalters, do you mean the ostree-rs-ext version bumped by rpm-ostree (which is used by anadonda) is different from ostree-rs-ext version bumped by bootc in system. That caused this issue. Do I understand correct? Thanks.

[cgwalters]

Hi @cgwalters, do you mean the ostree-rs-ext version bumped by rpm-ostree (which is used by anadonda) is different from ostree-rs-ext version bumped by bootc in system. That caused this issue. Do I understand correct? Thanks.

Yep!