Failed to get sandbox runtime: no runtime for "nvidia" is configured

[k8s-gpubuilder-markus]

kubelet[1311]: E0505 18:42:19.812688 1311 pod_workers.go:949] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "nvidia-device-plugin-daemonset-h9kxw_gpu-operator(8e491f32-4d92-4657-bbef-abf9d3dcb6ab)" with CreatePodSandboxError: "Failed to create sandbox for pod \"nvidia-device-plugin-daemonset-h9kxw_gpu-operator(8e491f32-4d92-4657-bbef-abf9d3dcb6ab)\": rpc error: code = Unknown desc = failed to get sandbox runtime: no runtime for \"nvidia\" is configured"" pod="gpu-operator/nvidia-device-plugin-daemonset-h9kxw" podUID=8e491f32-4d92-4657-bbef-abf9d3dcb6ab
May 05 18:42:19 hawkeye kubelet[1311]: E0505 18:42:19.812950 1311 remote_runtime.go:209] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox

This error repeats for all the pods / deamonsets of the gpu operator.

I'm setting up the gpu-operator via helm:

helm install --wait gpu-operator -n gpu-operator --create-namespace nvidia/gpu-operator --set driver.enabled=true --set toolkit.enabled=true

The version is v1.10.1, all the OSes are Ubuntu 20.04

There is no driver or container toolkit installed on the worker node with the GPUs, the gpu operator is set to handling it all.

[shivamerla]

@k8s-gpubuilder These logs are expected until the driver is loaded and toolkit is setup(~3-5min). toolkit will setup nvidia runtimeClass with containerd and after that pods(device-plugin, gpu-feature-discovery etc) can run with that runtimeClass.

[k8s-gpubuilder-markus]

Except that I posted it 1 hour ago and no change: the logs still show the same repeating error message. This is looking at journalctl -f on the gpu worker node. Is there a log what the gpu operator is doing to see where that is crashing?

Looks like the driver is not getting activated and everything else is not working because of it:

NAME                                                              READY   STATUS                  RESTARTS         AGE
pod/gpu-feature-discovery-xdk6b                                   0/1     Init:0/1                0                2m
pod/gpu-feature-discovery-zcdkl                                   0/1     Terminating             0                5m37s
pod/gpu-operator-7bfc5f55-hq28b                                   1/1     Running                 1 (51m ago)      112m
pod/gpu-operator-node-feature-discovery-master-6598566c8c-q562s   1/1     Running                 0                112m
pod/gpu-operator-node-feature-discovery-worker-4cbtv              1/1     Running                 0                112m
pod/gpu-operator-node-feature-discovery-worker-7jmnd              1/1     Running                 0                112m
pod/gpu-operator-node-feature-discovery-worker-kc2vc              1/1     Running                 0                112m
pod/gpu-operator-node-feature-discovery-worker-m4mrp              1/1     Running                 0                112m
pod/nvidia-container-toolkit-daemonset-d7h7j                      0/1     Init:0/1                0                2m
pod/nvidia-dcgm-exporter-qk7fk                                    0/1     Init:0/1                0                2m
pod/nvidia-dcgm-exporter-s6lrp                                    0/1     Terminating             0                5m37s
pod/nvidia-device-plugin-daemonset-qqksd                          0/1     Init:0/1                0                2m
pod/nvidia-device-plugin-daemonset-vw947                          0/1     Terminating             0                5m37s
pod/nvidia-driver-daemonset-45bdt                                 0/1     Init:CrashLoopBackOff   26 (2m ago)      112m
pod/nvidia-driver-daemonset-w2v7m                                 0/1     Init:0/1                24 (5m37s ago)   112m
pod/nvidia-operator-validator-f2f8d                               0/1     Init:0/4                0                2m

k logs pod/nvidia-driver-daemonset-45bdt
Error from server (BadRequest): container "nvidia-driver-ctr" in pod "nvidia-driver-daemonset-45bdt" is waiting to start: PodInitializing

[k8s-gpubuilder-markus]

Might this be a regression bug in the newest release v1.10.1? Should I try to revert back to an older release?

[shivamerla]

Can you check logs of initContainer? kubect logs nvidia-driver-daemonset-45bdt -n gpu-operator -c k8s-driver-manager Most likely this will happen if driver is already pre-installed or for some reason already loaded driver modules cannot be unloaded lsmod | grep nvidia

[k8s-gpubuilder-markus]

Oh thank you, that was helpful. I could see that the driver was still loaded. I don't know how this happened, but the nvidia driver was installed on the OS, which it should not have been. After this, now the gpu operator works again on that node and the vectoradd example runs as it should.