Container needs manual intervention to switch from Fedora Rawhide to Branched after branching

[garrett]

Describe the bug

When creating a Fedora 34 container, the repository sources are still set to rawhide, not Fedora 34.

(Yes, F34 is in beta, but this should have been fixed around branching time.)

Steps how to reproduce the behaviour

1. Create a Fedora 34 container with toolbox: toolbox create f34 -r 34
2. Enter the Fedora 34 container: toolbox enter f34
3. Attempt to install a package: sudo dnf install -y htop
4. See error; certificates are for F34, but the repo sources are still set to rawhide

Additionally, a workaround with swapping rawhide for fedora repo sources and doing a dnf distro-sync doesn't work, as dnf fails.

Expected behaviour

Creating a F34 container should have F34 ready-to-go. Packages should be installable by default and come from the F34 locations.

Actual behaviour

⬢[garrett@toolbox pipewire]$ sudo dnf install -y htop
Last metadata expiration check: 0:00:29 ago on Mon Mar 15 10:50:07 2021.
Dependencies resolved.
=====================================================================================================
 Package             Architecture          Version                      Repository              Size
=====================================================================================================
Installing:
 htop                x86_64                3.0.5-4.fc35                 rawhide                154 k

Transaction Summary
=====================================================================================================
Install  1 Package

Total download size: 154 k
Installed size: 354 k
Downloading Packages:
htop-3.0.5-4.fc35.x86_64.rpm                                         519 kB/s | 154 kB     00:00    
-----------------------------------------------------------------------------------------------------
Total                                                                168 kB/s | 154 kB     00:00     
warning: /var/cache/dnf/rawhide-2d95c80a1fa0a67d/packages/htop-3.0.5-4.fc35.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 9867c58f: NOKEY
Fedora - Rawhide - Developmental packages for the next Fedora releas 1.6 MB/s | 1.6 kB     00:00    
GPG key at file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-34-x86_64 (0x45719A39) is already installed
The GPG keys listed for the "Fedora - Rawhide - Developmental packages for the next Fedora release" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.. Failing package is: htop-3.0.5-4.fc35.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-34-x86_64
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED

Output of toolbox --version (v0.0.90+)

toolbox version 0.0.99.1

Toolbox package info (rpm -q toolbox)

toolbox-0.0.99.1-1.fc34.x86_64

Output of podman version

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.16
Built:        Mon Feb 22 15:08:57 2021
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)

podman-3.0.1-1.fc34.x86_64

Info about your OS

Fedora Silverblue 34 (beta)

34.20210314.n.0 (Silverblue Prerelease)

Additional context

This issue has started happening ever since the branch from rawhide to Fedora 34 beta. This has happened in previous releases, but was fixed within a few weeks. It's been a lot longer now.

It's a good idea to consider using official upstream containers instead, but, meanwhile, it's important to get this fixed before the F34 beta is tested by even more people.

[debarshiray]

Could it be that you had an old Fedora 34 image (ie., fedora-toolbox:34) on your system from the pre-branching days? Once an image has been cached locally, it doesn't get updated even if the registry has an updated copy.

If so, then try deleting the old image, and re-create the container.

I wonder why the usual mechanisms for moving a system across the branching point doesn't work for our containers that were created before the branching.

[garrett]

I have the latest image... at least according to a podman pull.

podman pull registry.fedoraproject.org/f34/fedora-toolbox:34
Trying to pull registry.fedoraproject.org/f34/fedora-toolbox:34...
Getting image source signatures
Copying blob defe698f583f skipped: already exists  
Copying blob 21358d8e4d2a [--------------------------------------] 0.0b / 0.0b
Copying config 7f6c60b1df done  
Writing manifest to image destination
Storing signatures
7f6c60b1df0279a59f0aef6ff50892ecefdceb77330fa7a29498cd113d8ee570

(I ran this earlier before filing the bug and it actually did grab a little bit of new content. But freshly create toolboxes still had the same issue.)

I can try deleting and trying again. Perhaps updating isn't working?

[HarryMichal]

The problem here is that you're pulling the old image. We recently changed the name of the images from f{ver}/fedora-toolbox to just fedora-toolbox. Here's a link to the registry.

So, instead of:

$ podman pull registry.fedoraproject.org/f34/fedora-toolbox:34

use

$ podman pull registry.fedoraproject.org/fedora-toolbox:34

The latest version of Toolbox should pull the proper image if it's not already pulled.

[martinpitt]

@garrett: No, you don't have an old image locally -- the official one was built on Feb 19, almost a month ago.

[garrett]

$ podman rmi fedora-toolbox:34
Untagged: registry.fedoraproject.org/f34/fedora-toolbox:34
Deleted: 7f6c60b1df0279a59f0aef6ff50892ecefdceb77330fa7a29498cd113d8ee570

$ toolbox create f34 -r 34
Image required to create toolbox container.
Download registry.fedoraproject.org/fedora-toolbox:34 (500MB)? [y/N]: y
Created container: f34
Enter with: toolbox enter f34

$ toolbox enter f34

⬢[garrett@toolbox ~]$ sudo dnf install -y htop
Fedora 34 - x86_64                                                                                                                                     12 MB/s |  73 MB     00:06    
[...]
Installed:
  htop-3.0.5-4.fc34.x86_64                                                                                                                                                            

Complete!

So removing and using the default container image fixed it. Thanks!

I wonder how many others are still using registry.fedoraproject.org/f34/fedora-toolbox:34 instead of registry.fedoraproject.org/fedora-toolbox:34.

Is there some way to redirect this or fix the URL on existing systems? Or at least message a fix?

[garrett]

...and I guess all my other toolboxes, including the F33 ones, are also based on the old location and are out of date? At least F33 is a bit older and updatable via dnf, so it's probably not as big of an issue.

[HarryMichal]

Toolbox images don't get updated over time. To get an update they have to be pulled again. So, redirecting the URL would be a solution only for users of an old version of Toolbox which should not happen for users of Fedora (considering users don't use EOL version). A warning message is a good thing to add for the next release, thank you for the suggestion.

[debarshiray]

I have the latest image... at least according to a podman pull.

Yes, that's what I meant by:

Once an image has been cached locally, it doesn't get updated even if
the registry has an updated copy.

This seems like an annoying limitation of the OCI stack that we have hit once in a while, and I haven't been able to find a good way to deal with it. Maybe I am missing something? Or someone knows something?

...and I guess all my other toolboxes, including the F33 ones, are also
based on the old location and are out of date? At least F33 is a bit older
and updatable via dnf, so it's probably not as big of an issue.

I wouldn't worry so much about it. Once an OCI container has been created out of an image, conceptually, it's a thing on its own. It needs to be updated or maintained like any package-based distro. I don't know of an easy way to force an update by replacing the old image with a new one.

It would be interesting if we could do that. :)

[HarryMichal]

I wouldn't worry so much about it. Once an OCI container has been created out of an image, conceptually, it's a thing on its own. It needs to be updated or maintained like any package-based distro. I don't know of an easy way to force an update by replacing the old image with a new one.

It would be interesting if we could do that. :)

For this we'd need a way to bootstrap the recreated toolboxes. Maybe even a user-provided script would be sufficient but how much can we trust that...

[debarshiray]

For this we'd need a way to bootstrap the recreated toolboxes.

Well, the container ultimately uses an overlaid filesystem. So one could imagine replacing the lower layer coming from the image. It's just that the available tooling doesn't seem to enable that.

[debarshiray]

As for the original issue, switching from Fedora Rawhide to Branched has always looked somewhat messy to me. Officially one is meant to do:

$ sudo dnf config-manager --set-disabled rawhide,rawhide-modular
$ sudo dnf config-manager --set-enabled fedora,fedora-modular,updates,updates-modular,updates-testing,updates-testing-modular
$ sudo dnf distrosync
$ sudo reboot

However, the reliability of those steps seems predicated on a bunch of variables like was your last package update before the branch point or after and so on. It always felt very flaky to me. :)

Toolbox containers aren't any different from Fedora hosts in this matter.

[garrett]

Right. Inside the container that thought it was F34, but still used an old rawhide, doing those steps failed at the distro-sync. But it's a container, so it can be thrown away and rebuilt much easier than installing to bare metal, at least. 😉

[HarryMichal]

Well, the container ultimately uses an overlaid filesystem. So one could imagine replacing the lower layer coming from the image. It's just that the available tooling doesn't seem to enable that.

Maybe it's time to go lower than Podman? :)

[oturpe]

Hello,
I hit a variant of this issue described in this comment. I had an old version of fedora-toolbox:34 image from pre-branching days:

$ podman image list | grep fedora-toolbox | grep "34 "
registry.fedoraproject.org/fedora-toolbox      34          80cdf4c9ee92  5 months ago   352 MB

That led to a Rawhide container and signature errors when trying to update a fresh toolbox container:

$ podman create --release 34 dev
$ podman enter
$ sudo dnf -y update
...
warning: /var/cache/dnf/rawhide-2d95c80a1fa0a67d/packages/cpio-2.13-10.fc35.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 9867c58f: NOKEY
Fedora - Rawhide - Developmental packages for the next Fedora release 1.6 MB/s | 1.6 kB     00:00    
GPG key at file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-34-x86_64 (0x45719A39) is already installed
The GPG keys listed for the "Fedora - Rawhide - Developmental packages for the next Fedora release" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.. Failing package is: cpio-2.13-10.fc35.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-34-x86_64
Public key for cracklib-dicts-2.9.6-25.fc34.x86_64.rpm is not installed. Failing package is: cracklib-dicts-2.9.6-25.fc34.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-34-x86_64
Public key for crypto-policies-scripts-20210621-1.gita0e819e.fc35.noarch.rpm is not installed. Failing package is: crypto-policies-scripts-20210621-1.gita0e819e.fc35.noarch
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-34-x86_64
...

As soon as I figured what is going on, this was easy to fix with podman pull registry.fedoraproject.org/fedora-toolbox:34. I was left wondering at these two points:

1. Why was the image called fedora-toolbox:f34 even before Fedora 34 was branched? It was not configured to use Fedora 34 repositories, which did not exist at the time. Would not fedora-toolbox:rawhide be a better name? That change would solve this issue, and be easier to understand.
2. When I do toolbox create --release fX, I always want the latest of fX. Should toolbox create start with podman pull registry.fedoraproject.org/fedora-toolbox:34 or similar, so that the latest & greatest is always used, not whatever happened to be available when the command was first used?

[vchernin]

I've run into this again with Fedora 35:
Trying the workaround listed above:

# remove all fedora toolbox 35 containers with toolbox rm <container>
podman rmi fedora-toolbox:35
toolbox create f35 -r 35
toolbox enter 35

Did not workaround the issue.
So I opened #869

[debarshiray]

I am tentatively closing this.

The OCI stack doesn't offer an easy way to update an image from underneath an existing container, and switching a Fedora environment from Rawhide to Branched is always a bit messy.

We could probably offer a way to prompt the user to manually switch. See #580