RFC: A new continuous mechanical FCOS stream

[jlebon]

Describe the enhancement

In the Fedora Atomic days, we had continuous builds from git-main of all the important upstream repos via rpmdistro-gitoverlay. I've been trying on and off to replicate something similar for FCOS using Packit, though that stalled on https://pagure.io/releng/issue/9801.

In a discussion with @bgilbert, we realized a simpler approach is to just use the fcos-buildroot image to make && make install from all the repos we care about into overrides/rootfs and go from there. This should naturally work on all arches too. So with not too much effort, we could have a mechanical stream of FCOS tracking git-main of all our components. This would be useful for CI purposes only. We'd never release/promote the content there.

The primary negative is simply the added overhead of tracking failures in yet another stream. It's also not exactly the same as using RPMs because removed files won't actually be removed, which could lead to false positives. Also, the rpmdb would be incorrect.

Still, it's worth considering I think given the effort it'd require to set up.

[jlebon]

I guess maybe to clarify the framing, there are two separate questions:

1. Should we pursue the goal of a continuous FCOS stream tracking git-main of all our projects?
2. If so, what is the best way to achieve this goal? Should we just use overrides/rootfs, or try to push forward on the Packit path (or something else)?

[travier]

I like this. Some thoughts:

• Daily builds might be too much work tracking failures. Once or twice a week might be good enough?
• Pushing Packit support forward for all our projects is interesting on its own as it could help us move to source-git for CentOS Stream packages
• But I'm not opposed to using make install into overrides too if that's easier

[cgwalters]

Heh I had missed this issue. OK so I had worked on coreos/rpm-ostree#3032 which will hook into copr for this. My immediate motivation actually in this case is I want to create quay.io/coreos-assembler:continuous so I have my updated error prefixing PRs. In the interim I spun a custom image which was fine, but having a continually regenerated image would be clearly useful.

I think it's fine if short term a continuous build is only x86_64, which COPR supports fine.

[cgwalters]

One argument for using COPR is other Fedora editions using rpm-ostree would find it useful.

[jlebon]

One intermediate approach to this which doesn't involve a new stream and the overhead it involves is to target rawhide only and have the rawhide stream pick them up.

Even if we do a separate continuous stream which is based on testing-devel's releasever we should still do that IMO.

Obviously if we go the COPR/Packit route, we can still also build the RPMs for the current releasever too. That would be useful for FCOS developers to drop into overrides/rpm locally.

[lucab]

We briefly touched this topic in the last meeting. Lots of questions still open, it will be brought back in the next meeting.

[jlebon]

ostreedev/ostree#2534
coreos/ignition#1317
coreos/butane#323
coreos/coreos-installer#782
coreos/afterburn#704

[jlebon]

OK, so it turns out setting up COPR for other repos is pretty easy so it seems silly to go with the make install hack. I've now set up automated builds for ostree, butane, and ignition in https://copr.fedorainfracloud.org/coprs/g/CoreOS/continuous/packages/ (and rpm-ostree was already set up by Colin).

Will work on adding more packages to the list in the background, but my proposal for this now is:

• Let's add a new continuous stream matching testing-devel but which includes the COPR repo in its repo list.
• Let's not build all the artifacts for the continuous stream: maybe just QEMU, live artifacts, and AWS + single region upload? (Full builds can be requested on-demand.)
• Let's include the COPR repo in the repo list for the rawhide stream too.

Frequency-wise, it could just match rawhide for now. I'd like to have both of those rebuild more frequently, but we should really implement some GC first.

[cgwalters]

Let's not build all the artifacts for the continuous stream: maybe just QEMU, live artifacts, and AWS + single region upload? (Full builds can be requested on-demand.)

We could even just start with only pushing a container image; xref coreos/coreos-assembler#2685

[jlebon]

Let's not build all the artifacts for the continuous stream: maybe just QEMU, live artifacts, and AWS + single region upload? (Full builds can be requested on-demand.)

We could even just start with only pushing a container image; xref coreos/coreos-assembler#2685

Hmm, I think having at least QEMU and AWS images would make it more useful to developers/testers so they can quickly try it out. E.g. cosa buildfetch --stream continuous && kola spawn -p aws (or --artifact qemu && cosa run).

[dustymabe]

I have to admit that I think this would be useful, but I'm really not excited about looking at failures from another stream.

[dustymabe]

We discussed this in the community meeting yesterday.

12:05:35  dustymabe | #agreed While we see value in a continuous stream we aren't
                    | wholly convinced the benefits qualify adding another stream at
                    | this point. We'd like to continue discussion in the ticket and
                    | collect examples of issues we think would have been caught by a
                    | continuous stream before landing in testing-devel.

[cgwalters]

(related) PR in coreos/fedora-coreos-config#1710