Enable updating of OEM specific tools

[t-lo]

Description
Currently, Flatcar ships OEM specific tools (like VMWare's open-vm-tools or Azure's WAAgent) packaged in the OEM image in a separate partition.

Impact
The tight coupling of OEM tooling and Flatcar releases makes it hard to use more recent OEM tool release on older Flatcar images. More importantly, the OEM tools, once deployed, are never updated.

Implementation options
A more dynamic implementation providing OEM tools would e.g. use container images capable of installing and updating those tools on the OEM partition, or even shipping the tools altogether.
Another option are systemd-sysext images.

Additional information
Related issues: #21

Tasks for systemd-sysext OEM images

• Implement the Nebraska extension to support additional update payloads (it is backwards compatible with old clients by sending the regular payload as first package entry, check that the Nebraska syncing mode also stores all the additional files): Add multi package support nebraska#579
• Add OEM sysext support in update-engine and a helper service: Support systemd-sysext OEM update payloads in update-engine #768
• The flatcar-update tool should get OEM-payload-aware: Tracked in Make flatcar-update provide OEM payloads #767
• Port the OEMs over: A benefit of sysext images is that they can have a strict coupling to the Flatcar VERSION_ID which enables to use dynamical linking to save space and make the OEM tool packages simpler. One can also remove the GCE "container" hack or the use a standard python location for Azure. The base Ignition configuration is not needed anymore because the sysext's OEM files already become part of the /usr hierarchy and systemd services there may be enabled through shipped symlinks and invoke any modifications of the system. The default Ignition configuration is gone in Ignition v3 anyway and we are moving away from it.
• Azure: Implement the Azure OEM tools as systemd-sysext image #769
• GCE Port GCE OEM setup to systemd-sysext image #1146
• VMware Port VMware OEM setup to systemd-sysext image #1144
• AWS: sysext: port AWS OEM to systemd sysext image scripts#1083 Port AWS and OpenStack OEM setup to systemd-sysext image #1145
• Conditionalize systemd units of OpenStack, Equinix Metal, Digital Ocean in base image (see issue above): For other OEMs like Equinix Metal and Digital Ocean we don't really need a sysext and can rather put the config files directly in the generic image: Port Equinix Metal (Packet) OEM away from the base Ignition #1143 and Port DigitalOcean OEM away from the base Ignition #1142
• Move the OEM partition mount out of /usr: Move OEM mount point out of /usr #766

[Lplazier]

Hi t-lo, any indication as to when we can expect an update on this?
Especially regarding the vmtools issues.

Thanks in advance.

[doets001]

I also would be very happy if a standard solution comes up for this vmware-tools issue. We have 300+ flatcare VMs running and normally we upgrade right after a new release. We stopped with this at the last kernel-4 release. I know there are work arounds, but I prefer an out of the box solution :-).
I see the issues in the queued lane, is there any sight on when it will move to 'in progress" ?

[t-lo]

Proof-of-concept PRs on the Flatcar side:

• image_to_vm: export vendor partition contents with vm image build scripts#105
• coreos-base/[oem-]update-vendortools: vendor tools update service flatcar-archive/coreos-overlay#724

Next steps / open issues

• distribution of OEM updates via update server
• clean implementation of PoC

[Winnie81]

I'd like to confirm whether there is any update for this issue. Thanks a lot.

[t-lo]

We're investigating this issue to be the first use case of #548 - and support OEM tools via sysexts. This will solve the update problem and also allow for significant more flexibility (see e.g. #781).