[RFE] allow OEMs to configure URL where to download oem-sysext updates

[LittleFox94]

Current situation

oem-sysext image updates are downloaded from a URL provided by Nebraska or, as a fallback, from the release-server.

Impact

Neither of those download locations work for out-of-tree OEMs, for example for cloud providers just starting to develop an OEM sysext for flatcar (Hi).

Ideal future situation

Out-of-tree OEMs should be able to configure download locations for the oem-sysext image and to provide a set of public keys to verify those image updates.

OEMs should also be able to configure if the upstream sources should be tried first or if the OEM-specific location is to be preferred (or even the only one to try). Preferring the upstream download servers would allow a smooth migration from out-of-tree to in-tree, while requiring the oem-sysext images being loaded from the OEM-specific location could be an easy way to block flatcar updates when they are not (yet) compatible or untested with the OEM.

Implementation options

Both configuration flags (download location and download location preference) could be stored on the OEM partition in the oem-release file.

Public keys to verify the updated oem-sysext images could be provided in a GPG keyring file stored in the oem-sysext to verify signatures of those updated oem-sysext images. This would allow updating the keys over time (by providing new oem-sysext images). Having this as a keyring file instead of a just a key would allow having multiple keys, e.g. the "current" and "next" (with "next" being in cold-storage or something).

Additional information

We are currently working on building OEM support for Anexia, which is required to bring support cloud-init. I'm not yet happy enough with the way we support cloud-init here, so I hesitate to upstream our current state - but it is good enough to use for us right now, besides our Flatcar systems being unable to update due to this issue.

Several workarounds for this problem and alternative solutions to providing our cloud-init support were explored so far, but having the here-documented implementation option would just make the apparently nicest way (oem-sysext image) very easy for us and seems like it could be useful for other folks as well.

If this feature request and the implementation options given here are deemed reasonable, we are willing to contribute the actual implementation :)

[tormath1]

Hello, thanks for this detailed issue and happy to see the interest of using Flatcar on Anexia.

Usually, the providers OEM are maintained in the scripts repository (see there: https://github.com/flatcar/scripts/tree/main/sdk_container/src/third_party/coreos-overlay/coreos-base the oem-* directories).

Some questions:

• What, roughly, would be the content of the OEM?
• Is cloud-init mandatory for Anexia? Usually Flatcar is provisioned via Ignition (Ignition can fetch its configuration from a metadata service)

[LittleFox94]

Hi @tormath1 :)

I found where the OEMs are stored in the Flatcar repository, but currently not yet too happy with what we have to upstream it - hence thinking about improving the situation for out-of-tree OEMs. I'm pretty sure there are other cases where an out-of-tree OEM might be a sensible option, not just for cloud providers working on their initial OEM support.

The contents of our OEM would be a program to adapt our cloud-config volume to cloud-init - what we have there is (currently, I hope this can be changed but would not be short-term) very Canonical cloud-init-brained with some extra twists. It should at some point also contain a guest agent and similar stuff.

Right now we can only do cloud-init since several layers in our VM provisioning workflow touch that. We also (sadly, again, maybe I can push for changes) don't have a metadata service but deliver the cloud-config by attaching a disk to the VM (NoCloud Datasource Canonicals cloud-init) - but that could be added to ignition when adding our OEM there (once the rest of our platform allows for ignition).

[tormath1]

Hi @LittleFox94,

Something I have been thinking - since this OEM image would be out-of-tree, I think we could consider it as a regular sysext image and then leverage the systemd-sysupdate¹ mechanism to deliver updates.

You could provide a systemd-sysupdate configuration file to target your download location + key to use to verify the download. This is an example of sysupdate configuration file to automatically pull new Kubernetes sysext image from Github releases²:

[Transfer]
Verify=false
[Source]
Type=url-file
Path=https://github.com/flatcar/sysext-bakery/releases/latest/download/
MatchPattern=kubernetes-@v-%a.raw
[Target]
InstancesMax=3
Type=regular-file
Path=/opt/extensions/kubernetes
CurrentSymlink=/etc/extensions/kubernetes.raw

Footnotes

1. https://www.freedesktop.org/software/systemd/man/latest/sysupdate.d.html# ↩

2. https://github.com/flatcar/sysext-bakery ↩

[LittleFox94]

Hi @tormath1 and sorry for my late reply

Using a regular sysext and systemd-sysupdate for updating it was considered as one of the options, but that has the downside that it is not possible to have that sysext bound to be a specific Flatcar version, especially not to have two versions of the sysext around, matching the Flatcar versions currently installed in the two slots - the versioning is completely independent from Flatcar then.

A given oem-sysext might interface quite tightly with the whole system and might just have to be adapted for given Flatcar versions. The current oem-sysext mechanism is already bound to a given Flatcar version, which is also why it's updated together with Flatcar which, (addition to Current Situation) leads to Flatcar updates not being successful for custom oem-sysexts injected into the disk image manually.