Unable to setup gpu-operator | nvidia-cuda-validator init "Failed to create pod sandbox"

[kuonat]

1. Quick Debug Checklist

• Are you running on an Ubuntu 18.04 node?
• Are you running Kubernetes v1.13+?
• Are you running Docker (>= 18.06) or CRIO (>= 1.13+)?
• Do you have i2c_core and ipmi_msghandler loaded on the nodes?
• Did you apply the CRD (kubectl describe clusterpolicies --all-namespaces)

ubuntu 20.04
docker 20.10.7
kubernetes 1.22.3

1. Issue or feature description

I have a kubernetes cluster created with kubeadm that has several worker nodes without a GPU and a fresh worker with a single GeForce RTX 3090 and without any drivers, NVIDIA toolkits etc.

When I install the gpu operator I get to a state, where all nodes run a feature-discovery-worker pod. The GPU worker additionally runs the following pods successfully:

• gpu-feature-discovery-XXXXX
• nvidia-container-toolkit-daemonset-XXXXX
• nvidia-dcgm-XXXXX
• nvidia-device-plugin-daemonset-XXXXX
• nvidia-driver-daemonset-XXXXX

However, the nvidia-cuda-validator-XXXX pod is stuck at status Init:0/1 and the nvidia-operator-validator is stuck at status Init:2/4.

From kubectl describe pod -n gpu-operator-resources nvidia-cuda-validator-ffrw8

Warning FailedCreatePodSandBox 3m35s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4320b99990499dea8f6b68d77b5ffa15edfd0d2ced6d9afc320d14260522a517" network for pod "nvidia-cuda-validator-ffrw8": networkPlugin cni failed to set up pod "nvidia-cuda-validator-ffrw8_gpu-operator-resources" network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "4320b99990499dea8f6b68d77b5ffa15edfd0d2ced6d9afc320d14260522a517"

The nvidia-operator-validator fails with "Back-off restarting failed container".

Additionally, the nvidia-dcgm-exporter pod constantly fails with the following logs:

time="2021-12-02T14:55:53Z" level=info msg="Starting dcgm-exporter"
time="2021-12-02T14:55:53Z" level=info msg="Attemping to connect to remote hostengine at XXX.XXX.XXX.XXX:5555"
time="2021-12-02T14:55:58Z" level=fatal msg="Error connecting to nv-hostengine: Host engine connection invalid/disconnected"

There is currently no firewall enabled.

When I preinstall a driver and the container toolkit the installation works. So this workaround is still possible. However, it would be great if I could bring the whole GPU operator to work correctly.

2. Steps to reproduce the issue

helm install --wait --generate-name nvidia/gpu-operator

3. Information to attach (optional if deemed irrelevant)

• kubernetes pods status: kubectl get pods - n gpu-operator-resources
• kubernetes daemonset status: kubectl get ds --all-namespaces
• If a pod/ds is in an error state or pending state kubectl describe pod -n gpu-operator-resources nvidia-cuda-validator-249tt
• If a pod/ds is in an error state or pending state kubectl describe pod -n gpu-operator-resources nvidia-coperator-validator-
• If a pod/ds is in an error state or pending state kubectl logs -n gpu-operator-resources nvidia-dcgm-exporter-pf6n6

daemonset_status.txt
pod_status.txt
cuda_validator_pod_describe.txt
operator_validator_pod_describe.txt
dcgm_exporter_logs.txt

• Output of running a container on the GPU machine: docker run -it alpine echo foo
• Docker configuration file: cat /etc/docker/daemon.json
• original Docker configuration file

daemon.txt
orig_daemon.txt

• NVIDIA shared directory: ls -la /run/nvidia
• NVIDIA packages directory: ls -la /usr/local/nvidia/toolkit
• NVIDIA driver directory: ls -la /run/nvidia/driver
• kubelet logs journalctl -u kubelet > kubelet.logs

directory_outputs.txt
kubelet.logs.txt

[shivamerla]

@kuonat This looks to be related to NVIDIA/nvidia-container-runtime#157 which was fixed with more recent version of toolkit container 1.7.2-ubuntu18.04. Please try to use v1.9.0-beta helm chart which uses this toolkit version.(delete existing chart , helm repo update kubectl delete crd clusterpolicies.nvidia.com and helm install --wait --generate-name nvidia/gpu-operator --version=v1.9.0-beta

[kuonat]

@shivamerla Thank you very much! When using v1.9.0-beta all the pods indeed started up correctly!

However, there is one thing that is not entirely solved, yet. The DCGM and the DCGM exporter pods are running, but I cannot see any metrics published on my grafana board (which I set up using the kube-prometheus-stack helm chart).

The pods provide the following logs:

kubectl logs nvidia-dcgm-XXXX:
Started host engine version 2.3.1 using port number: 5555

kubectl logs nvidia-dcgm-exporter-XXXX:
time="2021-12-06T08:32:14Z" level=info msg="Starting dcgm-exporter"
time="2021-12-06T08:32:14Z" level=info msg="Attemping to connect to remote hostengine at XXX.XXX.XXX.XXX:5555"
time="2021-12-06T08:32:14Z" level=info msg="DCGM successfully initialized!"
time="2021-12-06T08:32:14Z" level=info msg="Not collecting DCP metrics: Error getting supported metrics: This request is serviced by a module of DCGM that is not currently loaded"
time="2021-12-06T08:32:14Z" level=info msg="No configmap data specified, falling back to metric file /etc/dcgm-exporter/dcp-metrics-included.csv"
time="2021-12-06T08:32:14Z" level=warning msg="Skipping line 55 ('DCGM_FI_PROF_GR_ENGINE_ACTIVE'): DCP metrics not enabled"
time="2021-12-06T08:32:14Z" level=warning msg="Skipping line 58 ('DCGM_FI_PROF_PIPE_TENSOR_ACTIVE'): DCP metrics not enabled"
time="2021-12-06T08:32:14Z" level=warning msg="Skipping line 59 ('DCGM_FI_PROF_DRAM_ACTIVE'): DCP metrics not enabled"
time="2021-12-06T08:32:14Z" level=warning msg="Skipping line 63 ('DCGM_FI_PROF_PCIE_TX_BYTES'): DCP metrics not enabled"
time="2021-12-06T08:32:14Z" level=warning msg="Skipping line 64 ('DCGM_FI_PROF_PCIE_RX_BYTES'): DCP metrics not enabled"
time="2021-12-06T08:32:14Z" level=info msg="Kubernetes metrics collection enabled!"
time="2021-12-06T08:32:14Z" level=info msg="Pipeline starting"
time="2021-12-06T08:32:14Z" level=info msg="Starting webserver"

Does the second log point to any issues that are still remaining? And does the RTX 3090 even support exporting metrics?

[shivamerla]

@kuonat This might be due to that DCGM exporter is now run with hostNetwork: true. This was required as DCGM hostengine and DCGM are separate pods now.

[kuonat]

@shivamerla thank you! I was now able to set up everything correctly. My issue was, however, not the hostNetwork: true, but I had to adjust the namespace in the additionalScrapeConfigs of the kube-prometheus-stack helm chart values (as described in this section: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#setting-up-prometheus) to default instead of gpu-operator, since the pods for v1.9.0-beta are by default installed to the default namespace instead of the gpu-operator namespace. I somehow missed this initially.