composefs-native: bootc retroactively broken after attempting to switch branch

[TheHooly]

Attempted to switch from ghcr.io/secureblue/cosmic-main-hardened:br-testing-44-uki to ghcr.io/secureblue/cosmic-main-hardend:uki, after rebooting bootc is rendered unusable on all deployments retroactively (rebooted into the old deployment), attempting to run status, upgrade or switch returns:

hooly@fedora-desktop:~$ run0 -i bootc upgrade
jq: parse error: Invalid numeric literal at line 1, column 6
error: Upgrading composefs: Getting composefs deployment status: Getting composefs deployment status: Getting composefs deployment status: Getting composefs deployment metadata: Reading image info for deployment 994112916dd9dcfd68c5dba5efc0c0585dff421f81a9a6dfdd69641756c3b94b8bd3c78114143bad8a618a78545ddcada68a147fdf8f89bcf893836720a68254: Opening OCI image for manifest sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d: Opening stream 'oci-manifest-sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d': Opening ref 'streams/oci-manifest-sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d': No such file or directory (os error 2)
[1]hooly@fedora-desktop:~$ run0 -i bootc status
jq: parse error: Invalid numeric literal at line 1, column 6
error: Status: Getting composefs deployment status: Getting composefs deployment status: Getting composefs deployment metadata: Reading image info for deployment 994112916dd9dcfd68c5dba5efc0c0585dff421f81a9a6dfdd69641756c3b94b8bd3c78114143bad8a618a78545ddcada68a147fdf8f89bcf893836720a68254: Opening OCI image for manifest sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d: Opening stream 'oci-manifest-sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d': Opening ref 'streams/oci-manifest-sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d': No such file or directory (os error 2)
[1]hooly@fedora-desktop:~$ run0 -i bootc switch ghcr.io/secureblue/cosmic-main-hardened:uki
jq: parse error: Invalid numeric literal at line 1, column 6
error: Switching: Composefs Switching: Getting composefs deployment status: Getting composefs deployment status: Getting composefs deployment status: Getting composefs deployment metadata: Reading image info for deployment 994112916dd9dcfd68c5dba5efc0c0585dff421f81a9a6dfdd69641756c3b94b8bd3c78114143bad8a618a78545ddcada68a147fdf8f89bcf893836720a68254: Opening OCI image for manifest sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d: Opening stream 'oci-manifest-sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d': Opening ref 'streams/oci-manifest-sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d': No such file or directory (os error 2)
[1]hooly@fedora-desktop:~$

Additional info:
secureblue's UKI images are sealed images with the composefs-native backend,
the jq error is motd failing because it gets no info from bootc status

[cgwalters]

Hmm, are there different versions of bootc in use here? We should probably add a bootc internals fsck --report or so that would just export all the data into a zip file (or OCI image) or so.

jq: parse error: Invalid numeric literal at line 1, column 6

This bit is really strange...I am pretty sure we don't have any usage of jq being forked from our stack. Is bootc a wrapper script for you?

[TheHooly]

jq: parse error: Invalid numeric literal at line 1, column 6

This bit is really strange...I am pretty sure we don't have any usage of jq being forked from our stack. Is bootc a wrapper script for you?

Disregard that part, it is just a failure of secureblue's motd (it queries rpm-ostree status or bootc status), run0 -i starts a fresh shell for root and root's shell gets the motd, too.

I will gladly provide more info if you can tell me what you need.

[Johan-Liebert1]

It's failing to get the manifest for ghcr.io/secureblue/cosmic-main-hardend:uki which is weird. Could you get the outputs of the following commands

1. RUST_LOG=trace bootc status
2. tree /sysroot/composefs/streams/refs
3. find /sysroot/state -type f -name '*.origin' -exec sh -c 'for f; do echo "==> $f <=="; cat "$f"; echo; done' _ {} +
4. podman image info ghcr.io/secureblue/cosmic-main-hardend:uki

[TheHooly]

It's failing to get the manifest for ghcr.io/secureblue/cosmic-main-hardend:uki which is weird. Could you get the outputs of the following commands

1. RUST_LOG=trace bootc status

[root@fedora-desktop hooly]# RUST_LOG=trace bootc status
TRACE starting bootc
DEBUG argv0="bootc"
TRACE exec: "findmnt" "-J" "-v" "--output=SOURCE,TARGET,MAJ:MIN,FSTYPE,OPTIONS,UUID" "--mountpoint" "/"
TRACE Verified uid 0 with CAP_SYS_ADMIN
DEBUG Re-executing current process for _ostree_unshared
TRACE starting bootc
DEBUG argv0="bootc"
TRACE exec: "findmnt" "-J" "-v" "--output=SOURCE,TARGET,MAJ:MIN,FSTYPE,OPTIONS,UUID" "--mountpoint" "/"
TRACE Verified uid 0 with CAP_SYS_ADMIN
DEBUG Already in a mount namespace
DEBUG Current security context is unconfined_u:system_r:install_t:s0-s0:c0.c1023
DEBUG We have install_t
TRACE exec: "findmnt" "-J" "-v" "--output=SOURCE,TARGET,MAJ:MIN,FSTYPE,OPTIONS,UUID" "--mountpoint" "."
TRACE exec: "lsblk" "-J" "-b" "-O" "/dev/mapper/root"
DEBUG Target . is a mountpoint, remounting rw
TRACE exec: "findmnt" "-J" "-v" "--output=SOURCE,TARGET,MAJ:MIN,FSTYPE,OPTIONS,UUID" "--mountpoint" "."
TRACE exec: "lsblk" "-J" "-b" "-O" "/dev/mapper/root"
TRACE exec: "lsblk" "-J" "-b" "-O" "--inverse" "/dev/mapper/root"
TRACE exec: "lsblk" "-J" "-b" "-O" "/dev/nvme1n1"
TRACE exec: "findmnt" "-J" "-v" "--output=SOURCE,TARGET,MAJ:MIN,FSTYPE,OPTIONS,UUID"
DEBUG Nothing to do
error: Status: Getting composefs deployment status: Getting composefs deployment status: Getting composefs deployment metadata: Reading image info for deployment 994112916dd9dcfd68c5dba5efc0c0585dff421f81a9a6dfdd69641756c3b94b8bd3c78114143bad8a618a78545ddcada68a147fdf8f89bcf893836720a68254: Opening OCI image for manifest sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d: Opening stream 'oci-manifest-sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d': Opening ref 'streams/oci-manifest-sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d': No such file or directory (os error 2)

2. tree /sysroot/composefs/streams/refs

[root@fedora-desktop hooly]# tree /sysroot/composefs/streams/refs
/sysroot/composefs/streams/refs
└── oci
    ├── localhost%2Fbootc-sha256:28be8301611e215699a4dc2f32cceb17714e8698bedc2ce0796111b90ed138c2 -> ../../oci-manifest-sha256:28be8301611e215699a4dc2f32cceb17714e8698bedc2ce0796111b90ed138c2
    └── localhost%2Fbootc-sha256:94fff0ff6a0122f563aabdab045ff511f6037492b7646baae5bf0a52a3c67448 -> ../../oci-manifest-sha256:94fff0ff6a0122f563aabdab045ff511f6037492b7646baae5bf0a52a3c67448

2 directories, 2 files

3. find /sysroot/state -type f -name '*.origin' -exec sh -c 'for f; do echo "==> $f <=="; cat "$f"; echo; done' _ {} +

[root@fedora-desktop hooly]# find /sysroot/state -type f -name '*.origin' -exec sh -c 'for f; do echo "==> $f <=="; cat "$f"; echo; done' _ {} +
==> /sysroot/state/deploy/1272b536c7b8d757f40be596d096690566a78a697d43c747d572ae8f2886ba63e347f60d411895fce48e763f3e6a0b80819cffee1cf8274a3a0a0a932bbfbd06/1272b536c7b8d757f40be596d096690566a78a697d43c747d572ae8f2886ba63e347f60d411895fce48e763f3e6a0b80819cffee1cf8274a3a0a0a932bbfbd06.origin <==
[origin]
container-image-reference = ostree-unverified-image:docker://ghcr.io/secureblue/cosmic-main-hardened:br-testing-44-uki

[boot]
boot_type = uki
digest = e097afd62a99e985ca62bae4fe198256f95a5f03deedcff9e37c91f2b7fdb60a

[image]
manifest_digest = sha256:94fff0ff6a0122f563aabdab045ff511f6037492b7646baae5bf0a52a3c67448

==> /sysroot/state/deploy/21bb598254ccdc8cce33834d90456f6c652ca2b24e0dd8d0b2b141fbd3f1bf55f4320b86ac354017448888cda60de03b24dd42fe7ef7b681d6f5c3aae4a4f514/21bb598254ccdc8cce33834d90456f6c652ca2b24e0dd8d0b2b141fbd3f1bf55f4320b86ac354017448888cda60de03b24dd42fe7ef7b681d6f5c3aae4a4f514.origin <==
[origin]
container-image-reference = ostree-unverified-image:docker://ghcr.io/secureblue/cosmic-main-hardened:br-testing-44-uki

[boot]
boot_type = uki
digest = 0f4f8b77879c32d386188d8c5c24c20aed09eacbf457ae9ba5cec695a5793941

[image]
manifest_digest = sha256:28be8301611e215699a4dc2f32cceb17714e8698bedc2ce0796111b90ed138c2

==> /sysroot/state/deploy/994112916dd9dcfd68c5dba5efc0c0585dff421f81a9a6dfdd69641756c3b94b8bd3c78114143bad8a618a78545ddcada68a147fdf8f89bcf893836720a68254/994112916dd9dcfd68c5dba5efc0c0585dff421f81a9a6dfdd69641756c3b94b8bd3c78114143bad8a618a78545ddcada68a147fdf8f89bcf893836720a68254.origin <==
[origin]
container-image-reference = ostree-unverified-image:docker://ghcr.io/secureblue/cosmic-main-hardened:uki

[boot]
boot_type = uki
digest = c7a62528899ffae37ae9d4e2db3b0d5ced1445e2f6311f170fb842248e634989

[image]
manifest_digest = sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d

4. podman image info ghcr.io/secureblue/cosmic-main-hardend:uki

[root@fedora-desktop hooly]# podman image info ghcr.io/secureblue/cosmic-main-hardened:uki
Error: unrecognized command `podman image info`
Try 'podman image --help' for more information
[root@fedora-desktop hooly]# podman image inspect ghcr.io/secureblue/cosmic-main-hardened:uki
[]
Error: ghcr.io/secureblue/cosmic-main-hardened:uki: image not known

[Johan-Liebert1]

Weird that we have three state directories, but only two refs inside streams/refs. That's where the error is originating from, it's trying to get manifest for sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d i.e. deployment 994112916dd9dcfd68c5dba5efc0c0585dff421f81a9a6dfdd69641756c3b94b8bd3c78114143bad8a618a78545ddcada68a147fdf8f89bcf893836720a68254, but is failing to find.

This seems like a bug in the GC code.

Can you try two more commands

1. tree /sysroot/composefs/images
2. bootc internals composefs-gc

[TheHooly]

Weird that we have three state directories, but only two refs inside streams/refs. That's where the error is originating from, it's trying to get manifest for sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d i.e. deployment 994112916dd9dcfd68c5dba5efc0c0585dff421f81a9a6dfdd69641756c3b94b8bd3c78114143bad8a618a78545ddcada68a147fdf8f89bcf893836720a68254, but is failing to find.

This seems like a bug in the GC code.

Can you try two more commands

1. tree /sysroot/composefs/images

[root@fedora-desktop hooly]# tree /sysroot/composefs/images
/sysroot/composefs/images
├── 1272b536c7b8d757f40be596d096690566a78a697d43c747d572ae8f2886ba63e347f60d411895fce48e763f3e6a0b80819cffee1cf8274a3a0a0a932bbfbd06 -> ../objects/12/72b536c7b8d757f40be596d096690566a78a697d43c747d572ae8f2886ba63e347f60d411895fce48e763f3e6a0b80819cffee1cf8274a3a0a0a932bbfbd06
├── 21bb598254ccdc8cce33834d90456f6c652ca2b24e0dd8d0b2b141fbd3f1bf55f4320b86ac354017448888cda60de03b24dd42fe7ef7b681d6f5c3aae4a4f514 -> ../objects/21/bb598254ccdc8cce33834d90456f6c652ca2b24e0dd8d0b2b141fbd3f1bf55f4320b86ac354017448888cda60de03b24dd42fe7ef7b681d6f5c3aae4a4f514
├── 8e4637d15d9cbaf7c7c1a1aa974464136d8b45fa7ab4977b2f13c351a91651e69ca43394a65ed10022f16624f5cd291a1eca4f90d09bbe3fc7d8b2d65a39a6a0 -> ../objects/8e/4637d15d9cbaf7c7c1a1aa974464136d8b45fa7ab4977b2f13c351a91651e69ca43394a65ed10022f16624f5cd291a1eca4f90d09bbe3fc7d8b2d65a39a6a0
└── cb6cbdbafefeb0908e19878ff47c98eb81b8c27739ad187f4336e2e53936dc813e84c1a46ec6ce8b531a3775d2098ec6cb8fbf75ca1d1095fb6dd55a950d6566 -> ../objects/cb/6cbdbafefeb0908e19878ff47c98eb81b8c27739ad187f4336e2e53936dc813e84c1a46ec6ce8b531a3775d2098ec6cb8fbf75ca1d1095fb6dd55a950d6566

1 directory, 4 files

2. bootc internals composefs-gc

[root@fedora-desktop hooly]# bootc internals composefs-gc
error: Running composefs garbage collection: Getting composefs deployment status: Getting composefs deployment status: Getting composefs deployment metadata: Reading image info for deployment 994112916dd9dcfd68c5dba5efc0c0585dff421f81a9a6dfdd69641756c3b94b8bd3c78114143bad8a618a78545ddcada68a147fdf8f89bcf893836720a68254: Opening OCI image for manifest sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d: Opening stream 'oci-manifest-sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d': Opening ref 'streams/oci-manifest-sha256:2d0432b651df916a9d16c62802cd7b4776a50c78bc43b8eaca1a099d38ca0e7d': No such file or directory (os error 2)

[Johan-Liebert1]

yeah, okay, so definitely a GC issue. The state directory for that particular deployment wasn't removed. I think this is reproducible. Thanks for the quick replies

[Johan-Liebert1]

If you want a quick fix, and this is hacky, you can try running

sudo unshare -m -- bash -c 'mount -orw,remount /sysroot && rm -rf /sysroot/state/deploy/994112916dd9dcfd68c5dba5efc0c0585dff421f81a9a6dfdd69641756c3b94b8bd3c78114143bad8a618a78545ddcada68a147fdf8f89bcf893836720a68254'