GPU passthrough for micro-VM actors

[eliranw]

GPU passthrough for gVisor actors is proposed in #502. Nothing equivalent exists for micro-VM actors, and that PR gates the combination: a microvm WorkerPool requesting nvidia.com/gpu is rejected at apply time, because the pod would otherwise schedule onto a GPU node and hold a device no actor could use.

The gVisor approach does not carry over. It injects the host driver into the sandbox via CDI, which relies on the actor sharing the host kernel. A micro-VM guest has its own kernel, so the device has to be passed through — VFIO PCI passthrough for the GPU, plus getting a matching driver and user-mode libraries into the guest.

Open questions:

• VFIO device binding and IOMMU requirements on the node, and whether that composes with the unprivileged worker posture
• How the guest gets a driver matching the host GPU, and how it stays version-matched
• Whether cloud-hypervisor's VFIO support covers what is needed here
• What this means for snapshot/restore, which for micro-VM captures guest memory

Filing so the design is tracked outside the gVisor PR.

Related: #627, #502

[BenTheElder]

VFIO device binding and IOMMU requirements on the node, and whether that composes with the unprivileged worker posture

We haven't actually moved microVM to unprivileged yet. I have a branch locally though and I was planning to ~today.

How the guest gets a driver matching the host GPU, and how it stays version-matched

Yeah ..... this is a really interesting problem. Maybe picking a sandboxconfig that matches the host .... would need some API changes. But then we could do things like attempting to match the guest environment statically ...

AFAIK it's possible to get GPU passthrough with chv VFIO, but that was without the complication of the ateom pod, and I haven't personally done this yet. Not actually sure how easily I can get GPU instances these days ...

[AiWatson343]

Thanks for tracking this separately. The distinction between gVisor GPU support and micro-VM GPU support is important — the CDI-based injection model clearly does not translate when the actor runs with its own guest kernel.

For micro-VM actors, VFIO passthrough seems like the correct architectural direction, but the design will need to address the operational constraints:

Node-level GPU allocation should remain explicit and validated at scheduling time so unsupported configurations cannot reserve unusable devices.

The worker posture and security model need careful review since VFIO generally requires elevated host capabilities and IOMMU configuration.

Driver lifecycle management will likely be the biggest challenge: the guest needs a compatible NVIDIA stack, and we need a reliable mechanism to keep guest drivers aligned with host hardware/driver versions.

Snapshot/restore semantics need investigation because GPU state is not naturally captured with guest memory snapshots.

It may be worth defining this as a separate GPU-enabled micro-VM worker capability with clear requirements (IOMMU, VFIO, supported GPU models, driver strategy, snapshot limitations) rather than trying to make it behave identically to gVisor GPU passthrough. This will keep the security boundaries and runtime expectations clear.