A100: The GPU operator will not install the mig-manager

[lsyLearn]

1. Quick Debug Information

• OS/Version(e.g. RHEL8.6, Ubuntu22.04):Ubuntu22.04
• Kernel Version:5.15.0-60-generic
• Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker):Containerd
• K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS):v1.24.15
• GPU Operator Version:v23.6.0
• GPU: A100 PCIe 40GB

2. Issue or feature description

When the driver and nfd were installed in advance, I am trying to install gpu-operator in the A100 environment, but the installation of gpu-operator failed and the mig-manager was missing.
GPU:
root@master1:~# lspci | grep NVIDIA 2f:00.0 3D controller: NVIDIA Corporation GA100 [A100 PCIe 40GB] (rev a1)
The gpu operaotr pod info:

root@master1:~# kubectl get pod -n gpu-operator
NAME                                       READY   STATUS                  RESTARTS         AGE
gpu-feature-discovery-kht9c                1/1     Running                 1 (36m ago)      57m
gpu-operator-8597b78788-4ncg7              1/1     Running                 1 (36m ago)      57m
nvidia-container-toolkit-daemonset-pldv5   1/1     Running                 1 (36m ago)      57m
nvidia-cuda-validator-tgqqk                0/1     Init:CrashLoopBackOff   1 (17s ago)      19s
nvidia-dcgm-exporter-m7hg7                 1/1     Running                 1 (36m ago)      57m
nvidia-device-plugin-daemonset-gjlp7       0/1     CrashLoopBackOff        17 (4m47s ago)   57m
nvidia-operator-validator-7969z            0/1     Init:2/4                6 (2m59s ago)    57m

There is no nvidia-mig-manager pod.
And the error pod logs as follows:

root@master1:~# kubectl logs -n gpu-operator nvidia-device-plugin-daemonset-gjlp7
...
I0109 12:33:24.644789       1 main.go:256] Retreiving plugins.
I0109 12:33:24.645553       1 factory.go:107] Detected NVML platform: found NVML library
I0109 12:33:24.645594       1 factory.go:107] Detected non-Tegra platform: /sys/devices/soc0/family file not found
E0109 12:33:24.699911       1 main.go:123] error starting plugins: error getting plugins: failed to construct NVML resource managers: error building device map: error building device map from config.resources: invalid MIG configuration: At least one device with migEnabled=true was not configured correctly: error visiting device: device 0 has an invalid MIG configuration

3. Steps to reproduce the issue

• Install k8s cluster;
• Install nfd:

root@master1:~# kubectl get pod -n node-feature-discovery
NAME                                                         READY   STATUS    RESTARTS       AGE
nfd-release-node-feature-discovery-master-5564946bcf-x6qzs   1/1     Running   13 (49m ago)   43d
nfd-release-node-feature-discovery-worker-x7nff              1/1     Running   11 (49m ago)   43d

• Install gpu driver: Driver Version: 535.129.03
  
• Install the operator: helm install gpu-operator -n gpu-operator --create-namespace ./gpu-operator --set driver.enabled=false --set nfd.enabled=false
• Check the gpu-operator pod.

[cdesiniotis]

Hi @lsyLearn can you manually disable MIG on the node and then re-install the operator? Some of our components, e.g. k8s-device-plugin, will fail if MIG is enabled without any MIG devices existing. Once the operator is re-installed and in a healthy state, the mig-manager should come up and then you can label the node to apply a MIG configuration.

[larcane97]

Hi @cdesiniotis
Thanks for your comment.
I had the same issue, and after following your comment, I was able to run the GPU Operator successfully.

After that, I enabled MIG again on the node using nvidia-smi -mig 1.
However, it seems that MIG is not being applied in the cluster.
the GPU capacity and allocatable numbers remain the same, even though nvidia-smi shows that MIG is enabled on the GPU.

Do you know how to properly enable MIG in a cluster where the GPU Operator is already installed?

[cdesiniotis]

@larcane97 you can leverage MIG Manager to configure MIG for you. Here is the documentation for it: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html

[larcane97]

@cdesiniotis
thank you for saving my life.
i confused i should set the mig in person.

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[shivamerla]

Closing this as the user query is answered.