nvidia-device-plugin-daemonset stuck in crash loop

[flaker]

The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

1. Quick Debug Checklist

• Are you running on an Ubuntu 18.04 node? -> 18.04.5
• Are you running Kubernetes v1.13+? -> v1.16.9
• Are you running Docker (>= 18.06) or CRIO (>= 1.13+)? -> 18.09.9
• Do you have i2c_core and ipmi_msghandler loaded on the nodes? -> Yes, both
• Did you apply the CRD (kubectl describe clusterpolicies --all-namespaces) -> Yes

1. Issue or feature description

The pod nvidia-device-plugin-daemonset is stuck in Init:CrashLoopBackOff

2. Steps to reproduce the issue

Apply the helm chart

 helm install loadgen-gpu --wait nvidia/gpu-operator -f values.yaml --version 1.6.2 --set operator.defaultRuntime=containerd --debug

wait ...

all the pods will come into place except for that one.

Something that might be of value is that I replaced the containerd.io package that was included in the Ubuntu image with the containerd package from the standard repos since it seemed to provide a newer version.

3. Information to attach (optional if deemed irrelevant)

Describe of the problematic pod

(base) ➜  helm kc describe pod nvidia-device-plugin-daemonset-vlkm2 -n  gpu-operator-resources
Name:         nvidia-device-plugin-daemonset-vlkm2
Namespace:    gpu-operator-resources
Priority:     0
Node:         ip-172-32-62-232.us-east-2.compute.internal/172.32.62.232
Start Time:   Fri, 27 Aug 2021 17:11:04 -0400
Labels:       app=nvidia-device-plugin-daemonset
              controller-revision-hash=7ff4fb4c4b
              pod-template-generation=1
Annotations:  scheduler.alpha.kubernetes.io/critical-pod:
Status:       Pending
IP:           172.32.33.219
IPs:
  IP:           172.32.33.219
Controlled By:  DaemonSet/nvidia-device-plugin-daemonset
Init Containers:
  toolkit-validation:
    Container ID:  docker://6893451d063f1e493ba0f20a94d53455f9b82d7deee91808d9bb515634760cbc
    Image:         nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
    Image ID:      docker-pullable://nvcr.io/nvidia/k8s/cuda-sample@sha256:4593078cdb8e786d35566faa2b84da1123acea42f0d4099e84e2af0448724af1
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      /tmp/vectorAdd
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: connection refused": unavailable
      Exit Code:    128
      Started:      Fri, 27 Aug 2021 17:17:22 -0400
      Finished:     Fri, 27 Aug 2021 17:17:22 -0400
    Ready:          False
    Restart Count:  6
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from nvidia-device-plugin-token-j5wdj (ro)
Containers:
  nvidia-device-plugin-ctr:
    Container ID:
    Image:         nvcr.io/nvidia/k8s-device-plugin:v0.8.2-ubi8
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      --mig-strategy=single
      --pass-device-specs=true
      --fail-on-init-error=true
      --device-list-strategy=envvar
      --nvidia-driver-root=/run/nvidia/driver
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      NVIDIA_VISIBLE_DEVICES:      all
      NVIDIA_DRIVER_CAPABILITIES:  all
    Mounts:
      /var/lib/kubelet/device-plugins from device-plugin (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from nvidia-device-plugin-token-j5wdj (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  device-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/device-plugins
    HostPathType:
  nvidia-device-plugin-token-j5wdj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nvidia-device-plugin-token-j5wdj
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kops.k8s.io/instancegroup=gpu-nodes
                 nvidia.com/gpu.present=true
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
                 nvidia.com/gpu
Events:
  Type     Reason                  Age                     From                                                  Message
  ----     ------                  ----                    ----                                                  -------
  Normal   Scheduled               9m43s                   default-scheduler                                     Successfully assigned gpu-operator-resources/nvidia-device-plugin-daemonset-vlkm2 to ip-172-32-62-232.us-east-2.compute.internal
  Warning  FailedCreatePodSandBox  9m42s                   kubelet, ip-172-32-62-232.us-east-2.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "3658bb95ae86ade79a31794b6f7e5a20f319549170ccca2b0d33dacf0918a55b" network for pod "nvidia-device-plugin-daemonset-vlkm2": networkPlugin cni failed to set up pod "nvidia-device-plugin-daemonset-vlkm2_gpu-operator-resources" network: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused", failed to clean up sandbox container "3658bb95ae86ade79a31794b6f7e5a20f319549170ccca2b0d33dacf0918a55b" network for pod "nvidia-device-plugin-daemonset-vlkm2": networkPlugin cni failed to teardown pod "nvidia-device-plugin-daemonset-vlkm2_gpu-operator-resources" network: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"]
  Normal   SandboxChanged          9m25s (x2 over 9m41s)   kubelet, ip-172-32-62-232.us-east-2.compute.internal  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling                 9m24s                   kubelet, ip-172-32-62-232.us-east-2.compute.internal  Pulling image "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2"
  Normal   Pulled                  9m20s                   kubelet, ip-172-32-62-232.us-east-2.compute.internal  Successfully pulled image "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2"
  Normal   Started                 8m33s (x4 over 9m20s)   kubelet, ip-172-32-62-232.us-east-2.compute.internal  Started container toolkit-validation
  Normal   Created                 7m50s (x5 over 9m20s)   kubelet, ip-172-32-62-232.us-east-2.compute.internal  Created container toolkit-validation
  Normal   Pulled                  7m50s (x4 over 9m19s)   kubelet, ip-172-32-62-232.us-east-2.compute.internal  Container image "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2" already present on machine
  Warning  BackOff                 4m36s (x22 over 9m18s)  kubelet, ip-172-32-62-232.us-east-2.compute.internal  Back-off restarting failed container

Logs for the problematic pod

helm kc logs -f nvidia-device-plugin-daemonset-vlkm2 -n gpu-operator-resources
Error from server (BadRequest): container "nvidia-device-plugin-ctr" in pod "nvidia-device-plugin-daemonset-vlkm2" is waiting to start: PodInitializing

[flaker]

I would believe that containerd is down or something like that. But I see the other pods in the same node alive.

(base) ➜  helm kc get pod -n gpu-operator-resources
NAME                                       READY   STATUS              RESTARTS   AGE
nvidia-container-toolkit-daemonset-zph4w   1/1     Running             1          84m
nvidia-device-plugin-daemonset-kkmgq       0/1     ContainerCreating   0          7m53s
nvidia-driver-daemonset-nrltf              1/1     Running             1          84m

and their logs show that they are indeed running.

[shivamerla]

@flaker existing containers will continue to run, can you check the status of containerd service. Also, can you paste the custom values.yaml used. Any reason you are not using latest v1.7.x or v1.8.x charts?

[flaker]

Hello @shivamerla

I could not use 1.7.x or 1.8.x because of the priorityClass being used. My cluster refused to use those on the default or a different namespace.

The values.yaml file: https://kops.sigs.k8s.io/gpu/

As for containerd you were right. It died somehow in the middle of all of this (seems to die quite often). Now I am stuck in a different error:

2021/08/27 23:03:50 Loading NVML
2021/08/27 23:03:50 Failed to initialize NVML: could not load NVML library.
2021/08/27 23:03:50 If this is a GPU node, did you set the docker default runtime to `nvidia`?
2021/08/27 23:03:50 You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites
2021/08/27 23:03:50 You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start
2021/08/27 23:03:50 If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes

Still the pod is in crashloop.

That node is indeed a gpu enabled instance.

You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start

I didn't see those steps in the helm instalation page (https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#install-gpu-operator). I just executed:

$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

$ sudo apt-get update && sudo apt-get install -y nvidia-docker2
$ sudo systemctl restart docker

My /etc/docker/daemon.json already looked like:

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

I thought that the instalation of the nvidia/gpu-operator chart took care of that part. Am I missing some steps?

[flaker]

Just realized that the nvidia-driver-daemonset-nrltf pod now reads

Caught signal
Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs..

that I guess makes that node non-GPU

[flaker]

I restarted everything. The other pods are working. But the nvidia-device-plugin-daemonset continues to restart. But now I don't see an error:

  Type     Reason                  Age                    From                                                  Message
  ----     ------                  ----                   ----                                                  -------
  Normal   Scheduled               7m9s                   default-scheduler                                     Successfully assigned gpu-operator-resources/nvidia-device-plugin-daemonset-jhqkw to ip-172-32-62-232.us-east-2.compute.internal
  Warning  FailedCreatePodSandBox  6m57s (x13 over 7m9s)  kubelet, ip-172-32-62-232.us-east-2.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "nvidia-device-plugin-daemonset-jhqkw": Error response from daemon: all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: connection refused": unavailable
  Normal   SandboxChanged          6m57s (x12 over 7m8s)  kubelet, ip-172-32-62-232.us-east-2.compute.internal  Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff                 2m2s (x19 over 5m41s)  kubelet, ip-172-32-62-232.us-east-2.compute.internal  Back-off restarting failed container
(base) ➜  helm

as soon as I saw that error I restarted containerd. The error didn't show up anymore, but the pod won't start either.

[flaker]

The other two pods in the namespace seem to be fine.

kc logs nvidia-driver-daemonset-x6mch -n gpu-operator-resources

Detected 4 CPUs online; setting concurrency level to 4.
Installing NVIDIA driver version 460.32.03.
A precompiled kernel interface for kernel '5.4.0-1055-aws' has been found here: ./kernel/precompiled/nvidia-modules-5.4.0.
Kernel module linked successfully.
Kernel module linked successfully.
Kernel module unpacked successfully.
Unable to determine if Secure Boot is enabled: No such file or directory
Kernel messages:
[  371.183497] pci 0000:00:06.0: reg 0x10: [mem 0x00000000-0x00003fff]
[  371.183561] pci 0000:00:06.0: reg 0x18: [mem 0x00000000-0x000fffff pref]
[  371.183718] pci 0000:00:06.0: enabling Extended Tags
[  371.184588] pci 0000:00:06.0: BAR 2: assigned [mem 0xc0000000-0xc00fffff pref]
[  371.184612] pci 0000:00:06.0: BAR 0: assigned [mem 0xc0100000-0xc0103fff]
[  371.184685] ena 0000:00:06.0: enabling device (0000 -> 0002)
[  371.186157] ena: ENA device version: 0.10
[  371.186159] ena: ENA controller version: 0.0.1 implementation version 1
[  371.189974] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0100000, mac addr 02:7b:60:3d:71:e4
[  371.191998] ena 0000:00:06.0 ens6: renamed from eth0
[  469.846362] VFIO - User Level meta-driver version: 0.3
[  469.854046] IPMI message handler: version 39.2
[  469.855053] ipmi device interface
[  469.873819] nvidia: loading out-of-tree module taints kernel.
[  469.873825] nvidia: module license 'NVIDIA' taints kernel.
[  469.873825] Disabling lock debugging due to kernel taint
[  469.890957] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[  469.899376] nvidia-nvlink: Nvlink Core is being initialized, major device number 238
[  469.901046] PCI Interrupt Link [LNKB] enabled at IRQ 10
[  469.944731] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  460.32.03  Sun Dec 27 19:00:34 UTC 2020
[  469.950157] nvidia-uvm: Loaded the UVM driver, major device number 236.
[  469.954979] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  460.32.03  Sun Dec 27 18:51:11 UTC 2020
[  469.956560] nvidia-modeset: Unloading
[  470.036058] nvidia-uvm: Unloaded the UVM driver.
[  470.061677] nvidia-nvlink: Unregistered the Nvlink Core, major device number 238
Installing 'NVIDIA Accelerated Graphics Driver for Linux-x86_64' (460.32.03):
  Installing: [##############################] 100%
Driver file installation is complete.
Running post-install sanity check:
  Checking: [##############################] 100%
Post-install sanity check passed.
Running runtime sanity check:
  Checking: [##############################] 100%
Runtime sanity check passed.

Installation of the kernel module for the NVIDIA Accelerated Graphics Driver for Linux-x86_64 (version 460.32.03) is now complete.

Loading ipmi and i2c_core kernel modules...
Loading NVIDIA driver kernel modules...
Starting NVIDIA persistence daemon...
Mounting NVIDIA driver rootfs...
Done, now waiting for signal

kc logs nvidia-container-toolkit-daemonset-k6t57 -n gpu-operator-resources

time="2021-08-27T23:47:06Z" level=info msg="Setting up runtime"
time="2021-08-27T23:47:06Z" level=info msg="Starting 'setup' for containerd"
time="2021-08-27T23:47:06Z" level=info msg="Parsing arguments: [/usr/local/nvidia/toolkit]"
time="2021-08-27T23:47:06Z" level=info msg="Successfully parsed arguments"
time="2021-08-27T23:47:06Z" level=info msg="Loading config: /etc/containerd/config.toml"
time="2021-08-27T23:47:06Z" level=info msg="Successfully loaded config"
time="2021-08-27T23:47:06Z" level=info msg="Containerd version is 1.2.10"
time="2021-08-27T23:47:06Z" level=info msg="Config version: 1"
time="2021-08-27T23:47:06Z" level=info msg="Updating config"
time="2021-08-27T23:47:06Z" level=info msg="Successfully updated config"
time="2021-08-27T23:47:06Z" level=info msg="Flushing config"
time="2021-08-27T23:47:06Z" level=info msg="Successfully flushed config"
time="2021-08-27T23:47:06Z" level=info msg="Sending SIGHUP signal to containerd"
time="2021-08-27T23:47:06Z" level=info msg="Successfully signaled containerd"
time="2021-08-27T23:47:06Z" level=info msg="Completed 'setup' for containerd"
time="2021-08-27T23:47:06Z" level=info msg="Waiting for signal"

should I be changing the docker json file to explicitly say: "default-runtime": "nvidia", ?

I am a bit confused on wether I should be changing anything on the node since I am using the operator chart.

[elezar]

@flaker you shouldn't have to update the docker json file to include the default runtime setting. For one, you are correct that the GPU Operator (through the toolkit container) should manage this configuration, and two it seems as if you are using containerd and not docker as your container engine.

Could you include the contents of the /etc/containerd/config.toml file on one of the nodes where the device plugin is not starting? I also see that you're using quite an old version of containerd. Is there a specific reason for this?

[shivamerla]

I could not use 1.7.x or 1.8.x because of the priorityClass being used. My cluster refused to use those on the default or a different namespace.

@flaker you can set priorityClass as nil if your cluster doesn't support it (--set daemonsets.priorityClassName=""). Only caveat is GPU Operator pods can get evicted without any priority when there is a node pressure. Also, as @elezar mentioned i don't see operator.defaultRuntime=containerd is being specified in values.yaml you are using.

[flaker]

Hi @elezar,

Correct, it's containerd and the command line I used should cover that:

helm install loadgen-gpu --wait nvidia/gpu-operator -f values.yaml --version 1.6.2 --set operator.defaultRuntime=containerd

since I specified the defaultRuntime. Is that correct?

Containerd config

ubuntu@ip-172-32-83-83:~$ cat /etc/containerd/config.toml
disabled_plugins = ["cri"]
version = 1

Regarding the containerd version

I am using an image provided by Ubuntu (ami-0a817b0856bd5d87a). It's Ubuntu 18.05 that I think it's supported by this configuration.

Out of the box, the included package containerd.io is at version 1.2.10-3.

ubuntu@ip-172-32-83-83:~$ containerd --version
containerd containerd.io 1.2.10 b34a5c8af56e510852c35414db4c1f4fa6172339

since it seems to be quite old (things didn't work) I installed the containerd package (1.5.2-0)

ubuntu@ip-172-32-83-83:~$ containerd --version
containerd github.com/containerd/containerd 1.5.2-0ubuntu1~18.04.2

I think 1.5.5 is the latest release. Is 1.5.2 considered old? In any case it's still not working so I am happy to try with other versions.

Thank you.

[flaker]

Hi @shivamerla,

Thanks.

That allowed me to install the current version. Unfortunately it seems that even more stuff is failing (at least more pods)

(base) ➜  helm kubectl get pod -n gpu-operator-resources
NAME                                       READY   STATUS     RESTARTS   AGE
gpu-feature-discovery-f5cp5                0/1     Init:0/1   0          29m
nvidia-container-toolkit-daemonset-kklq4   1/1     Running    1          29m
nvidia-dcgm-exporter-v8zfj                 0/1     Init:0/1   0          29m
nvidia-dcgm-n6njk                          0/1     Init:0/1   0          29m
nvidia-device-plugin-daemonset-624t8       0/1     Init:0/1   0          29m
nvidia-driver-daemonset-2v78q              1/1     Running    1          29m
nvidia-operator-validator-skkg7            0/1     Init:0/4   0          29m

(base) ➜  helm kubectl describe pod nvidia-operator-validator-skkg7 -n gpu-operator-resources
.
.
.
Warning  FailedCreatePodSandBox  22s (x47 over 10m)  kubelet            Failed create pod sandbox: rpc error: code = Unknown desc = RuntimeHandler "nvidia" not supported 

Maybe some container failed to load the runtime?

[shivamerla]

@flaker the contents you pasted for /etc/containerd/config.toml were entire contents? If yes, than the toolkit doesn't seem to set the config correctly with v1. We recommend to use the updated version and 1.5.2 is better. However we recently found an issue with version string matching for 1.5.2-0ubuntu1~18.04.2 as this is not semver format. @elezar noted this here. Please use the workaround as suggested in the link until official release is out for the container-toolkit.

[flaker]

@shivamerla

Applied the workaround specifying to actual toolkit and using containerd 1.5.2.

Current (while the toolkit pod is up) /etc/containerd/config.toml

disabled_plugins = ["cri"]
version = 1

[plugins]

  [plugins.cri]

    [plugins.cri.containerd]
      default_runtime_name = "nvidia"

      [plugins.cri.containerd.runtimes]

        [plugins.cri.containerd.runtimes.nvidia]
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins.cri.containerd.runtimes.nvidia.options]
            Runtime = "/usr/local/nvidia/toolkit/nvidia-container-runtime"

        [plugins.cri.containerd.runtimes.nvidia-experimental]
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins.cri.containerd.runtimes.nvidia-experimental.options]
            Runtime = "/usr/local/nvidia/toolkit/nvidia-container-runtime-experimental"

I still have pending pods:

(base) ➜  helm kc get pod -n gpu-operator-resources
NAME                                       READY   STATUS     RESTARTS   AGE
gpu-feature-discovery-m2zhx                0/1     Init:0/1   0          7m41s
nvidia-container-toolkit-daemonset-fhzh9   1/1     Running    0          7m41s
nvidia-dcgm-6xctd                          0/1     Init:0/1   0          7m41s
nvidia-dcgm-exporter-mh97f                 0/1     Init:0/1   0          7m41s
nvidia-device-plugin-daemonset-6s6td       0/1     Init:0/1   0          7m41s
nvidia-driver-daemonset-kthwg              1/1     Running    0          7m41s
nvidia-operator-validator-zdrzn            0/1     Init:0/4   0          7m41s
(base) ➜  helm kc describe pod  nvidia-device-plugin-daemonset-6s6td -n gpu-operator-resources
.
.
.
  Warning  FailedCreatePodSandBox  3s (x25 over 5m7s)  kubelet, ip-172-32-83-83.us-east-2.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = RuntimeHandler "nvidia" not supported

I tried the next LTS version and it didn't join the cluster (unrelated to this issue). Is there any other distro you feel might have a better chance?

Thank you.

[shivamerla]

@flaker Is the driver installation complete? Can you run lsmod | grep nvidia on the worker node and confirm driver modules are loaded? Other pods will stay in init phase until driver is loaded. If they are stuck there after driver install, please get logs of kubectl logs nvidia-operator-validator-zdrzn -n gpu-operator-resources -c driver-validation and kubectl logs nvidia-operator-validator-zdrzn -n gpu-operator-resources -c toolkit-validation

[flaker]

@shivamerla

I think the modules are loaded. From the drivers loading pod is not a super smooth process (some error so some pre-cooked binaries seem to load)

ubuntu@ip-172-32-83-83:~$ lsmod | grep nvidia
nvidia_modeset       1196032  0
nvidia_uvm            995328  0
nvidia              35270656  17 nvidia_uvm,nvidia_modeset
drm                   491520  1 nvidia
i2c_core               77824  2 nvidia,drm

The logs you requests

(base) ➜  helm kubectl logs nvidia-operator-validator-zdrzn -n gpu-operator-resources -c driver-validation
Error from server (BadRequest): container "driver-validation" in pod "nvidia-operator-validator-zdrzn" is waiting to start: PodInitializing
(base) ➜  helm kubectl logs nvidia-operator-validator-zdrzn -n gpu-operator-resources -c toolkit-validation
Error from server (BadRequest): container "toolkit-validation" in pod "nvidia-operator-validator-zdrzn" is waiting to start: PodInitializing

These are several pods, containers so I am trying to see where "stuck point" is. So far all I found out was:

(base) ➜  helm kc describe pod  nvidia-device-plugin-daemonset-6s6td -n gpu-operator-resources
.
.
.
Warning  FailedCreatePodSandBox  3s (x25 over 5m7s)  kubelet, ip-172-32-83-83.us-east-2.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = RuntimeHandler "nvidia" not supported

[shivamerla]

disabled_plugins = ["cri"] why is this added to config.toml? Can you delete the config.toml file and restart container-toolkit. It will create the file again with version=2.