GPU's are showing as 0 while we have GPU's in cluster/NVIDIA GPU administration dashboard

[arpitsharma-vw]

Hello Team,
We have installed NVIDIA GPU administration dashboard as following link

https://docs.openshift.com/container-platform/4.11/monitoring/nvidia-gpu-admin-dashboard.html

But we dont see all data in Dashboard. GPU's are showing as 0 while we have GPU's in cluster. For other things, we see as "No datapoint founds". Please check the attached picture. We see this behaviour in our both clusters. Although Graphics Cards in cluster inventory are increasing as expected.

we have installed version 0.2.3 in openshift 4.11.21
Can you please check here?
Thanks

[arpitsharma-vw]

count(count by (UUID,GPU_I_ID) (DCGM_FI_PROF_GR_ENGINE_ACTIVE{exported_pod=~".+"})) or vector(0)
It shows 0

count(count by (UUID,GPU_I_ID) (DCGM_FI_DEV_MEM_COPY_UTIL))
No Datapoint available

sum(max by (UUID) (DCGM_FI_DEV_POWER_USAGE))
No Datapoint available

[shivamerla]

@simonkrenger can you help with this issue? is it similar to #422

[simonkrenger]

@arpitsharma-hexad It seems that all your metrics are 0 or not set, so it might be that they are not gathered correctly.

• Can you see values for any of the DCGM_FI_* metrics in Prometheus in general? You can find the metrics names here: https://docs.openshift.com/container-platform/4.11/monitoring/nvidia-gpu-admin-dashboard.html#viewing-the-gpus-dashboard
• For OpenShift Prometheus to collect these metrics, the namespace hosting the GPU Operator must have the label openshift.io/cluster-monitoring=true. Can you verify using oc get namespace <namespace-name> -o yaml that this label is set?

[arpitsharma-vw]

Hi @simonkrenger
I don't see any values for any of the DCGM_FI_* metrics in Prometheus in general.
openshift.io/cluster-monitoring=true is already set on the namespace.
Thanks

[simonkrenger]

So it seems that Prometheus does not have these metrics. This can either mean the nvidia-dcgm-exporter does not properly export them or they are not scraped correctly by the OpenShift Monitoring stack.

• With oc get servicemonitor, can you verify that you are seeing nvidia-dcgm-exporter listed?

• In the OpenShift Container Platform Web Console, navigate to "Observe", "Targets". When searching for "nvidia", can you see the "nvidia-dcgm-exporter" monitors and are they "Up"? What is the scrape duration?

• In the nvidia-gpu-operator namespace, can you run the following command for all the nvidia-dcgm-exporter Pods?

  oc rsh nvidia-dcgm-exporter-<ID> curl http://localhost:9400/metrics
  

  Then, review the output and check if you can see the DCGM_FI_ metrics being exported. Check also if they have any values (you may also post some output here). Here is a sample output:

  $ oc rsh nvidia-dcgm-exporter-x9lhm curl http://localhost:9400/metrics | grep DCGM_FI_DEV_POWER_USAGE
  Defaulted container "nvidia-dcgm-exporter" out of: nvidia-dcgm-exporter, toolkit-validation (init), init-pod-nvidia-node-status-exporter (init)
  # HELP DCGM_FI_DEV_POWER_USAGE Power draw (in W).
  # TYPE DCGM_FI_DEV_POWER_USAGE gauge
  DCGM_FI_DEV_POWER_USAGE{gpu="0",UUID="GPU-6bf0585f-07d2-eb7d-3bfa-e871f02b81d4",device="nvidia0",modelName="Tesla V100-SXM2-16GB",Hostname="ip-10-0-170-85",DCGM_FI_DRIVER_VERSION="525.60.13",container="",namespace="",pod=""} 24.412000
  

Once we have the above information, we can check if this is an OpenShift Container Platform issue or an NVIDIA issue.

[arpitsharma-vw]

Hi @simonkrenger

can you verify that you are seeing nvidia-dcgm-exporter listed?
No, it's not there. We have only

gpu-operator                  125d
nvidia-node-status-exporter   125d

We have servicemonitor dcgm-exporter in openshift-monitoring namespace. But I am not sure if it's for the same purpose.

In the OpenShift Container Platform Web Console, navigate to "Observe", "Targets". When searching for "nvidia", can you see the "nvidia-dcgm-exporter" monitors and are they "Up"? What is the scrape duration?
We can't find "nvidia-dcgm-exporter" monitors in the target.

In the nvidia-gpu-operator namespace, can you run the following command for all the nvidia-dcgm-exporter Pods?

arpit@arpit-GL62-7RDX:~/Desktop/datalab$ oc rsh nvidia-dcgm-exporter-wdxj6 curl http://localhost:9400/metrics | grep DCGM_FI_DEV_POWER_USAGE
Defaulted container "nvidia-dcgm-exporter" out of: nvidia-dcgm-exporter, toolkit-validation (init), init-pod-nvidia-node-status-exporter (init)
# HELP DCGM_FI_DEV_POWER_USAGE power usage.
# TYPE DCGM_FI_DEV_POWER_USAGE gauge
DCGM_FI_DEV_POWER_USAGE{gpu="0",UUID="GPU-988bd83b-bf78-6ce1-0b5d-5aee3c8e6aac",device="nvidia0",modelName="Tesla T4",Hostname="ip-11-241-108-144",container="",namespace="",pod=""} 15.217000
arpit@arpit-GL62-7RDX:~/Desktop/datalab$ oc rsh nvidia-dcgm-exporter-kkwdn curl http://localhost:9400/metrics | grep DCGM_FI_DEV_POWER_USAGE
Defaulted container "nvidia-dcgm-exporter" out of: nvidia-dcgm-exporter, toolkit-validation (init), init-pod-nvidia-node-status-exporter (init)
# HELP DCGM_FI_DEV_POWER_USAGE power usage.
# TYPE DCGM_FI_DEV_POWER_USAGE gauge
DCGM_FI_DEV_POWER_USAGE{gpu="0",UUID="GPU-694fee07-209d-1264-a28d-12100957f80e",device="nvidia0",modelName="Tesla T4",Hostname="ip-11-241-108-220",container="",namespace="",pod=""} 15.599000

[simonkrenger]

Hello @arpitsharma-hexad

can you verify that you are seeing nvidia-dcgm-exporter listed?
No, it's not there. We have only

Sorry for not being more clear, I meant if there are nvidia-dcgm-exporter Pods running. In the output you shared we can see that these Pods are indeed running.

We have servicemonitor dcgm-exporter in openshift-monitoring namespace. But I am not sure if it's for the same purpose.

The ServiceMonitor object is used to configure Prometheus in the OpenShift Container Platform Monitoring stack. To me it is not clear why in your case the SerivceMonitor is in the openshift-monitoring namespace. Typically this ServiceMonitor is in the nvidia-gpu-operator namespace where the nvidia-dcgm-exporter Pods are located:

$ oc get servicemonitor -n nvidia-gpu-operator 
NAME                          AGE
gpu-operator                  122m
nvidia-dcgm-exporter          122m
nvidia-node-status-exporter   122m
$ oc get po -n nvidia-gpu-operator 
NAME                                                  READY   STATUS      RESTARTS   AGE
[..]
nvidia-dcgm-exporter-x9lhm                            1/1     Running     0          99m
nvidia-dcgm-exporter-z59bg                            1/1     Running     0          98m
nvidia-dcgm-kc8wp                                     1/1     Running     0          99m
nvidia-dcgm-mrszl                                     1/1     Running     0          98m
[..]

We can't find "nvidia-dcgm-exporter" monitors in the target.

This also indicates that the ServiceMonitor is not being picked up by Prometheus. Likely because the openshift-monitoring namespace does not have the openshift.io/cluster-monitoring=true label set, but that is just a guess.

@shivamerla Any idea how to this could happen that the ServiceMonitor is created in the openshift-monitoring namespace instead of the nvidia-gpu-operator namespace? Seems to be filled by the Operator: https://github.com/NVIDIA/gpu-operator/blob/master/assets/state-dcgm-exporter/0700_service_monitor.yaml

[arpitsharma-vw]

Hi @simonkrenger
We don't have nvidia-dcgm-exporter. We have servicemonitor dcgm-exporter in openshift-monitoring namespace.

[arpitsharma-vw]

We also have openshift.io/cluster-monitoring: 'true' label set on openshift-monitoring namespace.
Can you please share the yaml for creating the nvidia-dcgm-exporter servicemonitor?

[simonkrenger]

I would not recommend manually creating the ServiceMonitor object, as this is something that seems to be created by the ClusterPolicy / the Operator. Can you check if your ClusterPolicy object has the "dcgmExporter" enabled and also the "serviceMonitor" enabled (see example below)?

$ oc get clusterpolicy gpu-cluster-policy -o yaml
apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  creationTimestamp: "2023-01-13T10:10:39Z"
  generation: 1
  name: gpu-cluster-policy
  resourceVersion: "57922"
  uid: 757877ef-7f7a-4a83-8b64-a68b22504d34
spec:
[..]
  dcgmExporter:
    config:
      name: ""
    enabled: true
    serviceMonitor:
      enabled: true
[..]

Again, I would not recommend creating the ServiceMonitor yourself as it should be created by the Operator, but you can find the definition here (may need to fill in the nvidia-gpu-operator namespace): https://github.com/NVIDIA/gpu-operator/blob/master/assets/state-dcgm-exporter/0700_service_monitor.yaml

[arpitsharma-vw]

Hi @simonkrenger
No, it was not enabled. We just enable it. I can see the metrics also now. :)
It works now.
Thanks

[simonkrenger]

Glad to hear that it works as expected when you edit the ClusterPolicy to include the serviceMonitor.

@shivamerla I guess this issue can be closed.

[shivamerla]

Thanks a lot @simonkrenger for helping with this.