NVIDIA GPU Operator: nvidia Runtime Not Re-Added to containerd config.toml Automatically

[tatodorov]

We are running NVIDIA Operator version 25.3.1, which deploys NVIDIA Container Toolkit version 1.17.8.
Our Kubernetes cluster is at version 1.32.5, with containerd version 2.0.5.
The host OS is Ubuntu 22.04 with kernel 6.8.0.
The problem, which we observe is that when we reboot a host with GPU or restart containerd on such host, the configuration for the nvidia-container-runtime is removed.
Afterwards, the GPU is not visible to any application scheduled on such node.
The NVIDIA Container Toolkit, which is running in Pod nvidia-container-toolkit-daemonset-, doesn't detect that change.
The only way to make the GPU available to the applications is to delete the nvidia-container-toolkit-daemonset- Pod.
Afterwards, the NVIDIA Container Toolkit adds the configuration for the nvidia-container-runtime to /etc/containerd/config.toml, and the GPU becomes available to the applications.

I am happy to contribute a fix if the maintainers consider this to be a valid issue.
Please let me know if you would like me to work on a solution.

[elezar]

The problem, which we observe is that when we reboot a host with GPU or restart containerd on such host, the configuration for the nvidia-container-runtime is removed.

Could you describe what happens in each of these situations? If the node is rebooted, the NVIDIA Container Toolkit Container should come up at some point and (re-)apply the modifcations to the containerd config to add the nvidia runtime. There is not way for the pod associated with the nvidia-container-toolkit-daemonset to keep running across a node reboot.

Even when restarting containerd, this should not cause the configuration to be re-written, unless there is some other tooling that is reseting the config (/etc/containerd/config.toml) to its original state.

[tatodorov]

Could you describe what happens in each of these situations? If the node is rebooted, the NVIDIA Container Toolkit Container should come up at some point and (re-)apply the modifcations to the containerd config to add the nvidia runtime. There is not way for the pod associated with the nvidia-container-toolkit-daemonset to keep running across a node reboot.

When we reboot the node, the nvidia-container-toolkit-daemonset was restarted, but not deleted and recreated by the DaemonSet.
When we delete it, it is recreated and then it reapplies the configuration of containerd.
The last such case was from two weeks ago.
However, when I tried to reproduce it today, I couldn't.
I restarted containerd, then I rebooted the node, and in both cases, the nvidia runtime was still in the /etc/containerd/config.toml.
So, I'll close the issue now, and try to collect more information if we face it again.

[sunmetis]

Hi ... going crazy with the same error . Eternal cycle:
Activating the nvidia runtime on the gpu node with the runtime containerd command (from the website)
The runtime is ok the gpu working BUT calico after a while give me errors about this calico node .
Fixing it .. gpu operator is down and no more containerd runtime ..
how to solve this s..t ?
Thx