"Error setting up sysroot: Missing base image" error on a custom image

[castrojo]

Describe the bug
Not sure where to file custom image bugs so let me know if this is the wrong place. I made a custom image and it errors out with this, which I've been able to reproduce on two machines. Seems like it happens a day after I load the image when it's getting an update:

OS version:

rpm-ostree status -b
Job for rpm-ostreed.service failed because the control process exited with error code.
See "systemctl status rpm-ostreed.service" and "journalctl -xeu rpm-ostreed.service" for details.
× rpm-ostreed.service - rpm-ostree System Management Daemon
     Loaded: loaded (/usr/lib/systemd/system/rpm-ostreed.service; static)
     Active: failed (Result: exit-code) since Sun 2022-12-11 10:31:12 EST; 11ms ago
       Docs: man:rpm-ostree(1)
    Process: 3271 ExecStart=rpm-ostree start-daemon (code=exited, status=1/FAILURE)
   Main PID: 3271 (code=exited, status=1/FAILURE)
     Status: "error: Couldn't start daemon: Error setting up sysroot: Reading deployment 0: Missing base image ref"
        CPU: 15ms

Dec 11 10:31:12 hamilton systemd[1]: Starting rpm-ostreed.service - rpm-ostree System Management Daemon...
Dec 11 10:31:12 hamilton rpm-ostree[3271]: Reading config file '/etc/rpm-ostreed.conf'
Dec 11 10:31:12 hamilton rpm-ostree[3271]: error: Couldn't start daemon: Error setting up sysroot: Reading deployment 0: Missing b… image ref
Dec 11 10:31:12 hamilton systemd[1]: rpm-ostreed.service: Main process exited, code=exited, status=1/FAILURE
Dec 11 10:31:12 hamilton systemd[1]: rpm-ostreed.service: Failed with result 'exit-code'.
Dec 11 10:31:12 hamilton systemd[1]: Failed to start rpm-ostreed.service - rpm-ostree System Management Daemon.
Hint: Some lines were ellipsized, use -l to show in full.

Additional context

This breaks the rpm-ostree command, all subcommands appear to be broken now, including rollback.

[inffy]

Also hit this today, but only on my virtual machine install.

After it errors, rpm-ostree also breaks. Same errors as in Jorges post.

[cgwalters]

Looks like a dup of coreos/rpm-ostree#4185 which should be fixed by ostreedev/ostree-rs-ext#432

Updating in coreos/rpm-ostree#4203 - I'll post some bits on how to un-wedge things.

Can you both confirm that you were doing client-side package layering i.e. rpm-ostree install type stuff not on the build server?

[castrojo]

I can confirm that I was layering client side on top of a custom image.

[cyrv6737]

Confirming I also encountered this yesterday in a VM.

[inffy]

Yes i had few packages layered top of the image on the VM

[cgwalters]

OK, I did coreos/rpm-ostree#4204 and ostreedev/ostree-rs-ext#436 - will get these PRs merged soon and then we'll have a build in https://copr.fedorainfracloud.org/coprs/g/CoreOS/continuous/ that should be usable to un-wedge things in the next few hours.

Sorry about this, it's embarrassing this one got through our CI. I personally went all-in on custom images and only did basic sanity testing of local layering, not including across upgrades. We'll get that CI gap covered after this stuff merges because it definitely should work.

[cgwalters]

OK, got fresh new builds in the COPR, to apply do e.g.:

$ rpm-ostree usroverlay 
$ rpm -Uvh https://download.copr.fedorainfracloud.org/results/@CoreOS/continuous/fedora-37-x86_64/05123243-rpm-ostree/rpm-ostree-{libs-,}2022.16.56.gede3d55e-1.fc37.x86_64.rpm

That should get things unblocked. But of course, you'll want the fixed version in the new upgrade target too; so there are two ways to do that. One is to stop doing client side layering and add the COPR to a custom container build. The second is to do a persistent local override:

$ rpm-ostree override replace https://download.copr.fedorainfracloud.org/results/@CoreOS/continuous/fedora-37-x86_64/05123243-rpm-ostree/rpm-ostree-{libs-,}2022.16.56.gede3d55e-1.fc37.x86_64.rpm

In any case we'll turn the crank and (ideally with confirmation this fixes things for you all) that get a new release out this next week.

[castrojo]

I can confirm the commands and override work for me, I was able to reboot and this new rpm-ostree is functioning normally.

Thanks for taking the time on a weekend! 👍🏾

[inffy]

I also confirm this fixed my VM install, thank you!

[cgwalters]

(And the CI gap here is plugged by coreos/rpm-ostree#4217 )

[cgwalters]

This fix is now queued in https://bodhi.fedoraproject.org/updates/FEDORA-2022-4ad713eb82

[kramfs]

OK, got fresh new builds in the COPR, to apply do e.g.:

$ rpm-ostree usroverlay 
$ rpm -Uvh https://download.copr.fedorainfracloud.org/results/@CoreOS/continuous/fedora-37-x86_64/05123243-rpm-ostree/rpm-ostree-{libs-,}2022.16.56.gede3d55e-1.fc37.x86_64.rpm

That should get things unblocked. But of course, you'll want the fixed version in the new upgrade target too; so there are two ways to do that. One is to stop doing client side layering and add the COPR to a custom container build. The second is to do a persistent local override:

$ rpm-ostree override replace https://download.copr.fedorainfracloud.org/results/@CoreOS/continuous/fedora-37-x86_64/05123243-rpm-ostree/rpm-ostree-{libs-,}2022.16.56.gede3d55e-1.fc37.x86_64.rpm

In any case we'll turn the crank and (ideally with confirmation this fixes things for you all) that get a new release out this next week.

Following this workaround helped me get back to a working rpm-ostree. I was using @castrojo ublue-os/base