URL for most recent artifact for platform+stream?

[mcsaucy]

Howdy,

I've been playing with some iPXE stuffs and Flatcar Container Linux. My setup involves pulling down the newest image at boot-time and then doing what you'd expect with it.

Looks like Flatcar (and CoreOS CL, which they forked from) expose some URL for grabbing the most recent artifact:

• flatcar ex: https://stable.release.flatcar-linux.net/amd64-usr/current/...
• coreos ex: https://stable.release.core-os.net/amd64-usr/current/...

FCOS exposes https://builds.coreos.fedoraproject.org/streams/stable.json, which allows for construction of artifact URLs (coreos-installer does this, I think), but that doesn't really mesh well with iPXE.

Is there any URL I can hit that will reliably get me the most up-to-date artifact for a given platform+stream?

[jlebon]

Is there any URL I can hit that will reliably get me the most up-to-date artifact for a given platform+stream?

We don't have stable URLs right now for our artifacts, but as you found we do have a stable interface to get to them. :)

I think there's been some hesitation in the past on exposing anything outside of that JSON file so that we retained the freedom to change things around as needed.

[bgilbert]

If you're caching the images locally, you can use coreos-installer download -f pxe to query the stream metadata and fetch the most recent images. coreos-installer download prints the resulting file paths on stdout, and you can use a wrapper script to rename the files to a well-known path on your PXE server.

For iPXE, at least, it might make sense for us to host a small web service that reads the stream metadata and redirects requests to the latest image.

[dustymabe]

We discussed this during the open floor of the community meeting today.

In general we'd like to do this in the future but we'd like to note that there is currently one very large benefit to using coreos-installer to download the artifacts versus using a latest download link: signature verification. coreos-installer will grab the latest artifact AND it will verify the signature making sure the image isn't corrupt or compromised. There are some use cases today where signature verification isn't possible (i.e. for direct iPXE booting or for creating an image from a URL in DigitalOcean) so in those cases having a latest link makes the most sense. For the other cases we'd want to be careful to still encourage people to use coreos-installer to download OR we'd want to make sure to properly educate them to also check the signature in their tooling. Either way we'd need to do some design to get the URLs right and also work with Fedora's Infrastructure team to stand up a small service that will parse the metadata and serve these URLs for people.

[mcsaucy]

17:19:21 anyone could actually write that, it doesn't have to be us
...
17:25:58 better to have a service that reads the single source of truth

Aight, I threw together https://github.com/mcsaucy/fcos-image-redirector.

It just parses the request URI to navigate the stream (which it fetches, parses, and caches for a bit) and then redirects. So as long as the stream structure remains stable, this should work.

So right now, https://latest-fcos.herokuapp.com/stable/artifacts/x86_64/metal/pxe/kernel -> https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200907.3.0/x86_64/fedora-coreos-32.20200907.3.0-live-kernel-x86_64

This will satisfy my needs and it seems like this isn't anything y'all are interested in owning atm, so I'll close this out.

[bgilbert]

This seems like something we might want to do, so let's keep open for now.

[dustymabe]

I think the easiest way to do this (without hosting a service) would be for us to just add a step to the release job that copies (plus a renames to remove the version from the filenames) the objects into a latest/ directory in s3 for that stream.

Another option is a redirector service, but we'd need to host it; which probably wouldnt' be too bad since we have the Fedora OpenShift instance. Though now that service would be exposed to the world.

[Nemric]

I did some work arround that subject, my 2 cents : https://github.com/Relativ-IT/Update-CoreOS
This won't help for ipxe over internet but in a local mirroring path

[yasminvalim]

We discussed this in FCOS Meeting today and agreed to continue this topic async.

To enable use cases like netboot.xyz (and iPXE in general) we think it would be beneficial to have stable links people can use. We're not sure 100% on which implementation would be most appropriate yet, though.

[dustymabe]

on implementation, two options come to mind:

1. simple copy of actual files in s3 from version specific folder to a latest folder or something, along with the version specific parts of the filenames stripped out

i.e. prod/streams/stable/builds/40.20240416.3.1/x86_64/fedora-coreos-40.20240416.3.1-live-kernel-x86_64 -> prod/streams/stable/builds/latest/x86_64/fedora-coreos-live-kernel-x86_64

• pro: simple
• con: wasted storage space

2. implement a redirector service that takes a URL and redirects to the actual latest artifact:

i.e. curl fcos.com/stable/x86_64/fedora-coreos-live-kernel-x86_64 redirects to https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20240416.3.1/x86_64/fedora-coreos-40.20240416.3.1-live-kernel-x86_64

• pro: no wasted space
• con: we need to implement a redirector service, figure out a URL, keep it up to date, etc.

[mcsaucy]

Option 1 seems way simpler. The time spent implementing option 2 almost certainly outweighs the storage cost.