Reconcile NVIDIA driver root and CDI state on every nspawn machine start

[bcho]

Summary

NVIDIA setup is run during initial node bootstrap and the managed NodeReboot flow, but it is not reliably run when an nspawn machine starts after a physical host reboot or a direct systemd-nspawn@<machine> restart.

The persisted daemon state can report goal state is applied while ephemeral NVIDIA runtime state is stale or incomplete. Containerd, kubelet, and the NVIDIA device plugin can then start before /run/nvidia/driver and the CDI specs are ready.

Observed behavior

Reproduced on an arm64 NVIDIA GB300 host using an nspawn-backed worker:

1. Initial bootstrap completed successfully, including setup-nvidia.

2. The physical host was rebooted.

3. The nspawn machine restarted automatically.

4. The host daemon repeatedly reported:

   daemon reconcile decision decision=ReportSucceeded reason="goal state is applied"
   

5. There was no post-reboot setup-nvidia execution before containerd and kubelet started.

6. /run/nvidia/driver/lib/aarch64-linux-gnu contained absolute symlinks into /run/host-nvidia/0 instead of regular copied driver files.

7. The NVIDIA device plugin generated an incomplete cdi-cri spec. It included devices and helper binaries, but omitted libcuda and libnvidia-ml.

8. A GPU-requesting pod received /dev/nvidia* and /usr/bin/nvidia-smi, but nvidia-smi failed with:

   NVIDIA-SMI couldn't find libnvidia-ml.so library in your system
   

Container startup also intermittently failed against stale CDI paths:

failed to fulfil mount request: open /run/nvidia/driver/lib/aarch64-linux-gnu/libEGL_nvidia.so.<version>: no such file or directory

and:

failed to fulfil mount request: open /run/nvidia/driver/lib/aarch64-linux-gnu/libnvidia-api.so.<version>: no such file or directory

Recovery validation

A MachineOperation with operationKind: NodeReboot repaired the node because it ran the managed sequence:

stop-node
start-nspawn-machine
setup-nvidia
start-containerd
start-kubelet

Afterward:

• Exact-version libcuda and libnvidia-ml files under /run/nvidia/driver/lib/<multiarch> were regular files.
• /etc/cdi/nvidia.yaml was regenerated.
• /var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json was regenerated with the full driver library set.
• Every CDI hostPath existed.
• nvidia-smi worked through the prepared driver root.

Expected behavior

Every nspawn machine lifecycle start should guarantee this ordering on NVIDIA nodes:

nspawn staging mounts and GPU devices available
-> rebuild /run/nvidia/driver
-> regenerate /etc/cdi/nvidia.yaml
-> start containerd
-> start kubelet
-> NVIDIA device plugin generates its CDI spec

The node should not become Ready with a partial NVIDIA driver root or stale CDI specification.

Suggested approaches

1. Install an in-machine unbounded-nvidia-setup.service oneshot and make containerd.service require and order after it.
2. Alternatively, have the host daemon watch nspawn unit lifecycle and run SetupNVIDIA, while gating in-machine containerd on an explicit NVIDIA-ready target.
3. On daemon startup/reconcile, validate ephemeral NVIDIA state before returning goal state is applied:
   • exact-version libcuda and libnvidia-ml files exist and are regular files;
   • /etc/cdi/nvidia.yaml exists;
   • every CDI hostPath exists;
   • the device-plugin CDI spec contains core driver libraries.
4. Atomically replace the driver root and CDI spec so containerd cannot observe partially prepared state.

Acceptance criteria

• A physical host reboot self-repairs NVIDIA state without a manual NodeReboot MachineOperation.
• A direct nspawn machine restart either runs NVIDIA setup automatically or blocks containerd/kubelet until setup completes.
• Newly created GPU pods receive devices, libcuda, NVML, and NVIDIA helpers.
• Non-GPU nodes remain unaffected.
• Initial bootstrap, in-place NodeReboot, and blue/green repave continue to use the same idempotent NVIDIA setup implementation.