Ship multiple VM agent binaries

[stgraber]

Currently incus looks for the incus-agent binary in the host PATH and copies that into the VM config drive. That works well enough but has quite a few drawbacks:

• As it's a copy, the guest cannot refresh its agent without a reboot
• Only a single OS and architecture can be made available (that of the host)
• Space and time is wasted as the binary is copied for every VM

Instead I believe we should:

• Ship a directory containing a variety of agent builds, likely named after OS.ARCH (e.g. linux.x86_64).
• That directory should be bind-mounted into the config drive
• The guest should be made to compare its agent with the one in the drive any time it gets reconnected with the host, if the two differ, the agent should exit so it can be restarted and get the new version loaded.

[stgraber]

This will fix handling for the few folks running 32bit Linux in their VMs as well as unblock porting the agent to FreeBSD and other platforms.

[dontlaugh]

How much metadata do we want to pack into the agent binary's filename? Do we need a rust-style "host triple"?

List of qemu architectures is here: https://wiki.qemu.org/Documentation/Platforms ; I doubt you'd need to package for the exotic stuff, but the naming scheme should be future-proofed for extensibility.

[stgraber]

OS.ARCH should be fine, we don't need a full compiler triplet as the binary is static, so just knowing the OS name and kernel architecture (per "uname -m") should do fine.

[pkramme]

The guest should be made to compare its agent with the one in the drive any time it gets reconnected with the host, if the two differ, the agent should exit so it can be restarted and get the new version loaded.

I'd like to point out that there are a few scenarios where this might have unintended consequences for VMs. For example, if there's a problem with the guest kernel like a crashed root filesystem, this might fail and prevent further access to the VM to investigate, depending on how this feature is build. Currently, the agent continues to work in that case, because it is already running.

[stgraber]

I'd like to point out that there are a few scenarios where this might have unintended consequences for VMs. For example, if there's a problem with the guest kernel like a crashed root filesystem, this might fail and prevent further access to the VM to investigate, depending on how this feature is build. Currently, the agent continues to work in that case, because it is already running.

Yeah, that'd be a problem if you had a VM in that state AND the host was upgraded to a new version that comes with a new agent.

Though at the same time, you also have the issue that the newer Incus version on the host requires a newer agent to keep the interaction going...

We'll have to see how we can handle that. Currently the agent isn't aware of where it's stored, retrieving the agent and spawning it is left to the init system, which makes it harder to handle those kind of scenarios.

If we made that logic part of the agent somehow, then the agent could itself mount the config drive again, sync the binaries and configs across to the tmpfs and then re-exec itself.

This would all work even with a kernel stuck on I/O as it would all be on tmpfs or new networked mounts.