How to ship cloud specific bits

[ajeddeloh]

Not to be confused with the discussion of agents in #12.

Clouds will need slightly different base/default Ignition configs and probably some extra config files for Ignition to use. Two questions arise: where do we ship them (initramfs? /? /boot?) and how do we ship them considering there are multiple clouds?

My proposal that I'm not super attached to:

• Ship the bits in the initramfs itself. They're not big and this means we don't have to deal with copying things over from the real root. At least do this for the Ignition configs.
• Ship all of the clouds' configs on for all the clouds (and bare metal). Teach Ignition to look under /some_path/$oem_id/{base,default}.ign or have systemd service that copys one to the location Ignition expects. Again, they aren't big and text compresses well, so it shouldn't be too much extra space used.

[dustymabe]

are there examples of varying cloud configs that will be needed?

[ajeddeloh]

https://github.com/coreos/coreos-overlay/tree/master/coreos-base look at the oem-*/files. Packet is a good example where we "phone home" using a systemd service calling curl (granted that could be baked into coreos-metadata probably, but we'd still want to only enable that bit of coreos-metadata on supported clouds).

[bgilbert]

Shipping them in the initramfs means we're paying the unpack overhead (which, yes, is probably minor) on every boot.

[ajeddeloh]

Some notes from the meeting today:

• It's probably not a good idea to put cloud specific things in the rootfs, since if we moved the rootfs and redeployed the ostree they would be lost.
• /boot seems reasonable assuming we're just storing configs. If we end up needing to store larger files we will need to revisit this.
• If configs are small, shipping all of them seems reasonable.
• Trying to detect oem is probably not a good idea.
• Images will need to differ at least in grub configuration to set oemid.

In terms of cloud-specific grub configs, I think baking them in seems like a good idea. Let the bits not in a filesystem differ and have the filesystems be the same for all clouds.

[lucab]

@ajeddeloh some late feedback (sorry, yesterday in the meeting I didn't get a good understanding on this):

• I think you are mixing /boot and the ESP in the comment above (or I misunderstood that point)
• shipping all of them in the initramfs seems a good design tradeoff, as long as size is sane
• in order to support client-side initramfs regeneration by rpm-ostree, we also need to ship the same bits in the rootfs (but there was some discussion about disabling this feature)
• re. oem detection, I'm mainly concerned about runtime auto-detection (e.g. dmidecode and such) but baking-in some external metadata in the different images for introspection seems fine

[lucab]

Following this trend of discussion, there is also another interesting thing we could experiment.

If we ship all OEM bits in all cases, we could try to bake a single raw image and just convert it to multiple formats. In order to identify the OEM once booted, we can use few bytes outside of the filesystem (e.g. I was looking at MBR disk-signature, which is 4 bytes and unused on GPT and not covered by a checksum), inject there a numerical ID and let grub set the kernel cmdline oem-id to a well-known string based on those bytes.