rpm-ostree rebase fedora/x86_64/coreos/stable from FCOS 40 images does not work, works fine from FCOS 39 images

[AdamWill]

Describe the bug

One of openQA's tests of FCOS is to deploy from the image being tested, rebase to something else, then roll back. Currently, on FCOS 40 images (including recent ones which are confused about whether they're really 40 or 41), the rebase step - rpm-ostree rebase fedora/x86_64/coreos/stable - fails, saying "error: Refspec 'fedora/x86_64/coreos/stable' not found". But the output of ostree remote refs fedora (which we run immediately beforehand) clearly shows a fedora:fedora/x86_64/coreos/stable entry, and we've never needed to include the leading fedora: in the refspec before. This same command works just fine from FCOS 39 images.

Reproduction steps

1. Deploy a system from any recent FCOS 40 image (e.g. https://builds.coreos.fedoraproject.org/prod/streams/rawhide/builds/40.20240215.91.0/x86_64/fedora-coreos-40.20240215.91.0-live.x86_64.iso )
2. Boot the installed system and run rpm-ostree rebase fedora/x86_64/coreos/stable

Expected behavior

An error "error: Refspec 'fedora/x86_64/coreos/stable' not found"

Actual behavior

A successful rebase

System details

See description. Clean install from any FCOS 40 image since Fedora-CoreOS-40.20231121.91.0 , which was the first time this failure occurred (sorry I didn't get around to reporting it sooner).

Butane or Ignition config

No response

Additional information

No response

[dustymabe]

Hmm. When is the last time this succeeded?

I would think this is related to the fact that we are now using deploy-via-container for our rawhide stream (see coreos/fedora-coreos-config#2711) and thus to rebase to an ostree remote (i.e. as opposed to a container registry remote) you would need to prepend with fedora:.

[AdamWill]

Hmm. When is the last time this succeeded?

I included that. "Clean install from any FCOS 40 image since Fedora-CoreOS-40.20231121.91.0 , which was the first time this failure occurred"

[AdamWill]

I see. the intent of the test is to check that rebasing and then reverting works. The rebase target just needs to be something that we can rely on to work if the rebasing mechanism itself works; logically, I guess, it should be the kind of rebase operation a user would be likely to do in the real world. So I would say whether we want to rebase to an OCI or ostree remote depends on what the 'default' is for the release we're rebasing to. If ostree was still the 'default' for whatever release 'stable' currently represents, rebasing to ostree is appropriate, I guess.

If I did want to rebase to a container remote, what would the command be? What's the equivalent of the ostree remote refs fedora command?

[travier]

If I did want to rebase to a container remote, what would the command be? What's the equivalent of the ostree remote refs fedora command?

https://coreos.github.io/rpm-ostree/container/#url-format-for-ostree-native-containers

FCOS is still ostree by default for Fedora 40 / for now.

[travier]

So probably:

$ rpm-ostree rebase ostree-remote-image:fedora:registry:quay.io/fedora/fedora-coreos:testing

https://quay.io/repository/fedora/fedora-coreos?tab=tags

[dustymabe]

FCOS is still ostree by default for Fedora 40 / for now.

No. We switched rawhide over on Nov 14, 2023: coreos/fedora-coreos-config#2711

[dustymabe]

Your test as it is isn't super great IMO. Here you are rebasing rawhide to stable, which can be significantly behind (for example rawhide is now F41 and stable is F39), which means that there's a high likelihood that you'll run into issues because of downgrading software.

Aside from that, for the rebase issue for now I would just modify the test to use fedora:fedora/x86_64/coreos/stable and that should work everywhere.

[AdamWill]

The test has limited options, we have to make the best of what's available to us. We have to rebase to something that we know will be available, and it has to work in various circumstances. I've flipped the targets around in the past when we've run into issues, but in fact it generally works okay.

[dustymabe]

@AdamWill did you get this resolved?

[AdamWill]

no, because it was the weekend. :D I can change the test to use fedora:fedora for now, that seems like the easiest choice.

[AdamWill]

OK, I've updated the test now and re-scheduled the tests for the most recent CoreOS build, will see if that works.

edit: yes it did!