Enable updating of OEM specific tools

[t-lo]

Description
Currently, Flatcar ships OEM specific tools (like VMWare's open-vm-tools or Azure's WAAgent) packaged in the OEM image in a separate partition.

Impact
The tight coupling of OEM tooling and Flatcar releases makes it hard to use more recent OEM tool release on older Flatcar images. More importantly, the OEM tools, once deployed, are never updated.

Implementation options
A more dynamic implementation providing OEM tools would e.g. use container images capable of installing and updating those tools on the OEM partition, or even shipping the tools altogether.
Another option are systemd-sysext images.

Additional information
Related issues: #21

Tasks for systemd-sysext OEM images

• Implement the Nebraska extension to support additional update payloads (it is backwards compatible with old clients by sending the regular payload as first package entry, check that the Nebraska syncing mode also stores all the additional files): Add multi package support nebraska#579
• Add OEM sysext support in update-engine and a helper service: Support systemd-sysext OEM update payloads in update-engine #768
• The flatcar-update tool should get OEM-payload-aware: Tracked in Make flatcar-update provide OEM payloads #767
• Port the OEMs over: A benefit of sysext images is that they can have a strict coupling to the Flatcar VERSION_ID which enables to use dynamical linking to save space and make the OEM tool packages simpler. One can also remove the GCE "container" hack or the use a standard python location for Azure. The base Ignition configuration is not needed anymore because the sysext's OEM files already become part of the /usr hierarchy and systemd services there may be enabled through shipped symlinks and invoke any modifications of the system. The default Ignition configuration is gone in Ignition v3 anyway and we are moving away from it.
• Azure: Implement the Azure OEM tools as systemd-sysext image #769
• GCE Port GCE OEM setup to systemd-sysext image #1146
• VMware Port VMware OEM setup to systemd-sysext image #1144
• AWS: sysext: port AWS OEM to systemd sysext image scripts#1083 Port AWS and OpenStack OEM setup to systemd-sysext image #1145
• Conditionalize systemd units of OpenStack, Equinix Metal, Digital Ocean in base image (see issue above): For other OEMs like Equinix Metal and Digital Ocean we don't really need a sysext and can rather put the config files directly in the generic image: Port Equinix Metal (Packet) OEM away from the base Ignition #1143 and Port DigitalOcean OEM away from the base Ignition #1142
• Move the OEM partition mount out of /usr: Move OEM mount point out of /usr #766

[Lplazier]

Hi t-lo, any indication as to when we can expect an update on this?
Especially regarding the vmtools issues.

Thanks in advance.

[doets001]

I also would be very happy if a standard solution comes up for this vmware-tools issue. We have 300+ flatcare VMs running and normally we upgrade right after a new release. We stopped with this at the last kernel-4 release. I know there are work arounds, but I prefer an out of the box solution :-).
I see the issues in the queued lane, is there any sight on when it will move to 'in progress" ?

[t-lo]

Proof-of-concept PRs on the Flatcar side:

• image_to_vm: export vendor partition contents with vm image build scripts#105
• coreos-base/[oem-]update-vendortools: vendor tools update service flatcar-archive/coreos-overlay#724

Next steps / open issues

• distribution of OEM updates via update server
• clean implementation of PoC

[Winnie81]

I'd like to confirm whether there is any update for this issue. Thanks a lot.

[t-lo]

We're investigating this issue to be the first use case of #548 - and support OEM tools via sysexts. This will solve the update problem and also allow for significant more flexibility (see e.g. #781).

[bignay2000]

What file(s) would tell us the version of OEM stuff?

On my VMWare VM's I see I have this oem-release file, but maybe nice to have a Flatcar version and date?

cat /usr/share/oem/oem-release
ID=vmware VERSION_ID=11.3.5 NAME="VMware" HOME_URL="https://www.vmware.com/" BUG_REPORT_URL="https://issues.flatcar-linux.org"

[bignay2000]

Running on ESXi 8 host. vmWare tools still not updated despite having ran updates to latest stable.
Looks like no way to update to not be vulnerable to CVE-2022-31676.

[pothos]

The update path for OEM software is still in progress. The workaround up to now is this here:
#21 (comment)

[Winnie81]

Hi @pothos ,

I saw this issue was closed. I'd like to confirm whether OVT could be updated when performing in-place upgrade of Flatcar? Thanks a lot.

[pothos]

Yes, it's in Flatcar Beta/Alpha but not yet in Stable. Once it lands in Stable the migration will happen after two updates, only then the old contents get removed and the new ones activated.

[Winnie81]

Yes, it's in Flatcar Beta/Alpha but not yet in Stable. Once it lands in Stable the migration will happen after two updates, only then the old contents get removed and the new ones activated.

Thanks for your reply.

I installed flatcar 3732.0.0(alpha channel), and after power on, it was upgraded by default to 3815.0.0 and OVT was upgraded to 12.3.5.
Then I installed another VM with 3732.0.0, this time I executed "update_engine_client -check_for_update" manually, from /etc/os-release, I could see OS was upgraded to 3815.0.0, but OVT was still 12.3.0, and it was not upgraded. Could you please tell me what I missed? Thanks a lot.

[pothos]

Two upgrade cycles are needed so that both of the A/B partitions have a version where the OEM software is provided in a systemd-sysext image, only then the old contents get removed and the new ones activated.

[Winnie81]

Two upgrade cycles are needed so that both of the A/B partitions have a version where the OEM software is provided in a systemd-sysext image, only then the old contents get removed and the new ones activated.

After one cycle, it was upgraded from 3732.0.0 to 3815.0.0, do you mean the customer need to wait until next new build available?
core@localhost /usr/share/oem/bin $ update_engine_client -update
I0104 02:20:55.050163 1352 update_engine_client.cc:251] Initiating update check and install.
I0104 02:20:55.052597 1352 update_engine_client.cc:256] Waiting for update to complete.
LAST_CHECKED_TIME=1704334856
PROGRESS=0.000000
CURRENT_OP=UPDATE_STATUS_IDLE
NEW_VERSION=0.0.0
NEW_SIZE=0
I0104 02:21:00.487994 1352 update_engine_client.cc:194] No update available

And I'm wondering why auto update can upgrade OVT directly to the latest one.

I installed flatcar 3732.0.0(alpha channel), and after power on, it was upgraded by default to 3815.0.0 and OVT was upgraded to 12.3.5.

[pothos]

Yes, the old partition still relies on the old version living on the OEM partition as extracted binaries. If you want to force a migration you can run flatcar-update to install the latest version again in the second partition.

[Winnie81]

Yes, the old partition still relies on the old version living on the OEM partition as extracted binaries. If you want to force a migration you can run flatcar-update to install the latest version again in the second partition.

Could you please provide the whole command for flatcar-update so that I could upgrade OVT to 12.3.5 after OS was upgraded to latest alpha version 3815.0.0 from 3732.0.0. Thanks a lot for your great help.

[pothos]

Running sudo flatcar-update -V 3815.0.0 should give you the newest version on both partitions then.

[Winnie81]

Running sudo flatcar-update -V 3815.0.0 should give you the newest version on both partitions then.

Thanks for your update. It works.