`namespace` and `pod` labels are sometimes missing from metrics

[Altair-Bueno]

What is the version?

3.4.2

What happened?

Labels like namespace and pod are sometimes missing from metrics that should contain them, like DCGM_FI_DEV_FB_USED

What did you expect to happen?

Quoting from #277

When a GPU workload is running on the cluster, the dcgm-exporter produces labels with pod, container, and namespace.

In reality, sometimes metrics don't include these labels

What is the GPU model?

4 nodes with 2 GPUs each. The nvidia-smi output corresponds to one of those nodes

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla V100-PCIE-32GB           Off | 00000000:5B:00.0 Off |                    0 |
| N/A   26C    P0              24W / 250W |      4MiB / 32768MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla V100-PCIE-32GB           Off | 00000000:D7:00.0 Off |                    0 |
| N/A   28C    P0              22W / 250W |      4MiB / 32768MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

What is the environment?

Pod

How did you deploy the dcgm-exporter and what is the configuration?

Helm chart with these values

runtimeClassName: nvidia
podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9400"

How to reproduce the issue?

I honestly don't know. May be related to CUDA out of memory errors, as we've mostly noticed this issue when TensorFlow crashes. But it is not consistent.

Anything else we need to know?

Logs:

$ kubectl logs -l app.kubernetes.io/name=dcgm-exporter
time="2024-06-13T07:48:03Z" level=info msg="Initializing system entities of type: GPU"
time="2024-06-13T07:48:04Z" level=info msg="Not collecting NvSwitch metrics; no fields to watch for device type: 3"
time="2024-06-13T07:48:04Z" level=info msg="Not collecting NvLink metrics; no fields to watch for device type: 6"
time="2024-06-13T07:48:04Z" level=info msg="Not collecting CPU metrics; no fields to watch for device type: 7"
time="2024-06-13T07:48:04Z" level=info msg="Not collecting CPU Core metrics; no fields to watch for device type: 8"
time="2024-06-13T07:48:04Z" level=info msg="Kubernetes metrics collection enabled!"
time="2024-06-13T07:48:04Z" level=info msg="Pipeline starting"
time="2024-06-13T07:48:04Z" level=info msg="Starting webserver"
time="2024-06-13T07:48:04Z" level=info msg="Listening on" address="[::]:9400"
time="2024-06-13T07:48:04Z" level=info msg="TLS is disabled." address="[::]:9400" http2=false
time="2024-06-13T07:48:54Z" level=info msg="Initializing system entities of type: GPU"
time="2024-06-13T07:48:54Z" level=info msg="Not collecting NvSwitch metrics; no fields to watch for device type: 3"
time="2024-06-13T07:48:54Z" level=info msg="Not collecting NvLink metrics; no fields to watch for device type: 6"
time="2024-06-13T07:48:54Z" level=info msg="Not collecting CPU metrics; no fields to watch for device type: 7"
time="2024-06-13T07:48:54Z" level=info msg="Not collecting CPU Core metrics; no fields to watch for device type: 8"
time="2024-06-13T07:48:54Z" level=info msg="Kubernetes metrics collection enabled!"
time="2024-06-13T07:48:54Z" level=info msg="Pipeline starting"
time="2024-06-13T07:48:54Z" level=info msg="Starting webserver"
time="2024-06-13T07:48:54Z" level=info msg="Listening on" address="[::]:9400"
time="2024-06-13T07:48:54Z" level=info msg="TLS is disabled." address="[::]:9400" http2=false
time="2024-06-13T07:49:46Z" level=info msg="Initializing system entities of type: GPU"
time="2024-06-13T07:49:46Z" level=info msg="Not collecting NvSwitch metrics; no fields to watch for device type: 3"
time="2024-06-13T07:49:46Z" level=info msg="Not collecting NvLink metrics; no fields to watch for device type: 6"
time="2024-06-13T07:49:46Z" level=info msg="Not collecting CPU metrics; no fields to watch for device type: 7"
time="2024-06-13T07:49:46Z" level=info msg="Not collecting CPU Core metrics; no fields to watch for device type: 8"
time="2024-06-13T07:49:46Z" level=info msg="Kubernetes metrics collection enabled!"
time="2024-06-13T07:49:46Z" level=info msg="Pipeline starting"
time="2024-06-13T07:49:46Z" level=info msg="Starting webserver"
time="2024-06-13T07:49:46Z" level=info msg="Listening on" address="[::]:9400"
time="2024-06-13T07:49:46Z" level=info msg="TLS is disabled." address="[::]:9400" http2=false
time="2024-06-13T07:47:12Z" level=info msg="Initializing system entities of type: GPU"
time="2024-06-13T07:47:12Z" level=info msg="Not collecting NvSwitch metrics; no fields to watch for device type: 3"
time="2024-06-13T07:47:12Z" level=info msg="Not collecting NvLink metrics; no fields to watch for device type: 6"
time="2024-06-13T07:47:12Z" level=info msg="Not collecting CPU metrics; no fields to watch for device type: 7"
time="2024-06-13T07:47:12Z" level=info msg="Not collecting CPU Core metrics; no fields to watch for device type: 8"
time="2024-06-13T07:47:12Z" level=info msg="Kubernetes metrics collection enabled!"
time="2024-06-13T07:47:12Z" level=info msg="Starting webserver"
time="2024-06-13T07:47:12Z" level=info msg="Pipeline starting"
time="2024-06-13T07:47:12Z" level=info msg="Listening on" address="[::]:9400"
time="2024-06-13T07:47:12Z" level=info msg="TLS is disabled." address="[::]:9400" http2=false

[Altair-Bueno]

Hopefully this screenshot better illustrates the issue:

As you can see, we sometimes experience random spikes the dcgm-exporter namespace. These spikes correspond to GPU usage from another pod within the cluster, but because pod and namespace labels are missing from these metrics, prometheus adds them server side (see honor_labels https://prometheus.io/docs/prometheus/latest/configuration/configuration/)

These spikes last for 5m, if that rings any bells

[nvvfedorov]

The issue looks like a similar one for #272. @Altair-Bueno, I have a few questions:

1. Do you run workload on MIG-enabled environment?
2. What is the configuration and version of the k8s-device-plugin?
3. Do you see in the DCGM-exporter logs messages like: "No Kubelet socket, ignoring", "failed to transform metrics for transform".
4. Do you see any errors or signs of restart in k8s-device-plugin logs?
5. When the issue happens, do you see that the kubelet was restarted?

[zclyne]

We also notice the same issue on our clusters. Our cluster uses a customized k8s-device-plugin where we enabled its support for specific GPU types, so that GPU resources are advertised in such a format nvidia.com/nvidia-v100-sxm2-32gb rather than nvidia.com/gpu. Does this customization affect the behavior of dcgm-exporter as well?

dcgm-exporter: 3.3.5-3.4.0
k8s-device-plugin: customized based on 0.15.0

@nvvfedorov Looking forward to your answer. Thank you very much.

[Altair-Bueno]

1. Do you run workload on MIG-enabled environment?

I'm afraid i don't know how to verify if my environment has MIG enabled or not. Allegedly we are only using time slicing (which i believe is a different technology).

2. What is the configuration and version of the k8s-device-plugin?

# Values for Helm chart nvidia-device-plugin v0.14.2
config:
  map:
    default: |-
      version: v1
      sharing:
        timeSlicing: 
          renameByDefault: false
          failRequestsGreaterThanOne: false
          resources:
            - name: nvidia.com/gpu
              replicas: 32
runtimeClassName: nvidia

# Values for helm chart gpu-feature-discovery v0.8.2
runtimeClassName: nvidia

3. Do you see in the DCGM-exporter logs messages like: "No Kubelet socket, ignoring", "failed to transform metrics for transform".

No. Logs are clean. No restarts. See original post with the log output of all pods within the daemonset.

4. Do you see any errors or signs of restart in k8s-device-plugin logs?

Nothing weird on logs, neither current pod or previous. However, the pod restarted 23h ago (manual node reboot). The spikes do not correspond to said reboot.

$ kubectl logs -l app.kubernetes.io/name=nvidia-device-plugin
Defaulted container "nvidia-device-plugin-sidecar" out of: nvidia-device-plugin-sidecar, nvidia-device-plugin-ctr, nvidia-device-plugin-init (init)
Defaulted container "nvidia-device-plugin-sidecar" out of: nvidia-device-plugin-sidecar, nvidia-device-plugin-ctr, nvidia-device-plugin-init (init)
Defaulted container "nvidia-device-plugin-sidecar" out of: nvidia-device-plugin-sidecar, nvidia-device-plugin-ctr, nvidia-device-plugin-init (init)
Defaulted container "nvidia-device-plugin-sidecar" out of: nvidia-device-plugin-sidecar, nvidia-device-plugin-ctr, nvidia-device-plugin-init (init)
W0424 07:53:27.055384      26 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0424 07:53:27.055918      26 main.go:246] Waiting for change to 'nvidia.com/device-plugin.config' label
I0424 07:53:27.066508      26 main.go:248] Label change detected: nvidia.com/device-plugin.config=
I0424 07:53:27.066726      26 main.go:364] No value set and no default set. Attempting fallback strategies: [named single]
I0424 07:53:27.066748      26 main.go:368] Attempting to find config named: default
I0424 07:53:27.066759      26 main.go:300] Updating to config: default
I0424 07:53:27.066885      26 main.go:308] Already configured. Skipping update...
I0424 07:53:27.066897      26 main.go:246] Waiting for change to 'nvidia.com/device-plugin.config' label
W0612 15:35:52.942663      25 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0612 15:35:52.943224      25 main.go:246] Waiting for change to 'nvidia.com/device-plugin.config' label
I0612 15:35:52.953462      25 main.go:248] Label change detected: nvidia.com/device-plugin.config=
I0612 15:35:52.953606      25 main.go:364] No value set and no default set. Attempting fallback strategies: [named single]
I0612 15:35:52.953621      25 main.go:368] Attempting to find config named: default
I0612 15:35:52.953628      25 main.go:300] Updating to config: default
I0612 15:35:52.953739      25 main.go:308] Already configured. Skipping update...
I0612 15:35:52.953747      25 main.go:246] Waiting for change to 'nvidia.com/device-plugin.config' label
W0424 07:54:00.142806      24 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0424 07:54:00.143622      24 main.go:246] Waiting for change to 'nvidia.com/device-plugin.config' label
I0424 07:54:00.154254      24 main.go:248] Label change detected: nvidia.com/device-plugin.config=
I0424 07:54:00.154435      24 main.go:364] No value set and no default set. Attempting fallback strategies: [named single]
I0424 07:54:00.154453      24 main.go:368] Attempting to find config named: default
I0424 07:54:00.154462      24 main.go:300] Updating to config: default
I0424 07:54:00.154623      24 main.go:308] Already configured. Skipping update...
I0424 07:54:00.154633      24 main.go:246] Waiting for change to 'nvidia.com/device-plugin.config' label
W0424 07:53:27.416598      26 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0424 07:53:27.417122      26 main.go:246] Waiting for change to 'nvidia.com/device-plugin.config' label
I0424 07:53:27.430240      26 main.go:248] Label change detected: nvidia.com/device-plugin.config=
I0424 07:53:27.430386      26 main.go:364] No value set and no default set. Attempting fallback strategies: [named single]
I0424 07:53:27.430401      26 main.go:368] Attempting to find config named: default
I0424 07:53:27.430408      26 main.go:300] Updating to config: default
I0424 07:53:27.430492      26 main.go:308] Already configured. Skipping update...
I0424 07:53:27.430505      26 main.go:246] Waiting for change to 'nvidia.com/device-plugin.config' label

5. When the issue happens, do you see that the kubelet was restarted?

The cluster is running k3s, not kubelet. K3s logs show nothing weird

jun 13 14:52:04 fog4 k3s[2163]: W0613 14:52:04.180703    2163 policy.go:689] Ignoring IPv6 dest IP blocks [[2000::/3 timeout 0] [fd00::/8 timeout 0]] from policy firewall because we are not IPv6 Enabled!
jun 13 14:52:04 fog4 k3s[2163]: W0613 14:52:04.181164    2163 policy.go:689] Ignoring IPv6 dest IP blocks [[2000::/3 timeout 0] [fd00::/8 timeout 0]] from policy firewall because we are not IPv6 Enabled!
jun 13 14:52:05 fog4 k3s[2163]: I0613 14:52:05.980995    2163 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef7ea26-f695-4e2f-9948-c2a377e72988" path="/var/lib/kubelet/pods/9ef7ea26-f695-4e2f-9948-c2a377e72988/volumes"
jun 13 14:52:07 fog4 k3s[2163]: E0613 14:52:07.980625    2163 remote_runtime.go:222] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"7708ac18434b07bbc1c47d50fdcd0c67084cde7dc8672a4179e8321aedfc033d\": failed to get network \"cbr0\" cached result: decoding version from network config: unexpected end of JSON input" podSandboxID="7708ac18434b07bbc1c47d50fdcd0c67084cde7dc8672a4179e8321aedfc033d"
jun 13 14:52:07 fog4 k3s[2163]: E0613 14:52:07.980694    2163 kuberuntime_manager.go:1375] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"7708ac18434b07bbc1c47d50fdcd0c67084cde7dc8672a4179e8321aedfc033d"}
jun 13 14:52:07 fog4 k3s[2163]: E0613 14:52:07.980765    2163 kubelet.go:2001] failed to "KillPodSandbox" for "e1972373-4832-40b4-ad4a-3c064911a3b1" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"7708ac18434b07bbc1c47d50fdcd0c67084cde7dc8672a4179e8321aedfc033d\": failed to get network \"cbr0\" cached result: decoding version from network config: unexpected end of JSON input"
jun 13 14:52:07 fog4 k3s[2163]: E0613 14:52:07.980821    2163 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"e1972373-4832-40b4-ad4a-3c064911a3b1\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"7708ac18434b07bbc1c47d50fdcd0c67084cde7dc8672a4179e8321aedfc033d\\\": failed to get network \\\"cbr0\\\" cached result: decoding version from network config: unexpected end of JSON input\"" pod="istio-system/istiod-d865cfbcd-ph8m8" podUID="e1972373-4832-40b4-ad4a-3c064911a3b1"
jun 13 14:52:10 fog4 k3s[2163]: I0613 14:52:10.435600    2163 scope.go:117] "RemoveContainer" containerID="1410059e83df0b84cfafc043257f48aa2521a6008ca8b7b51cb8aa5f71031017"
jun 13 14:52:10 fog4 k3s[2163]: I0613 14:52:10.442217    2163 scope.go:117] "RemoveContainer" containerID="1dc40278061cd47d4d82aed11336a2b919d72433fcaef7b6ceb03d35faaa104e"
jun 13 14:52:13 fog4 k3s[2163]: I0613 14:52:13.746556    2163 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager

[nvvfedorov]

The k8s-device-plugin is the source of information about namespaces, pods etc. If kubelet or k8s-device-plugin experience issues with high load, the DCGM-exporter may not get information about namespace, pods etc mapping on GPU from /var/lib/kubelet/pod-resources/kubelet.sock aka pod-resources API.

At the moment, we don't have a reliable steps to reproduce the issue on our environments. Any observations about k8s cluster health, CPU, memory usage, type of workload, may help us to identity the root of the issue.

[nvvfedorov]

@Altair-Bueno , However, the pod restarted 23h ago (manual node reboot). The spikes do not correspond to said reboot. - What is the time lag between the reboot and the spike?

[zclyne]

Hi @nvvfedorov , aftering inspecting the code I found this line:

dcgm-exporter/pkg/dcgmexporter/kubernetes.go

Line 150 in 961ee35

if resourceName != nvidiaResourceName {

Looks like it checks whether a pod is requesting for nvidia.com/gpu and if not, it skips the pod. In our situation since we dustomized the pod to request a GPU in nvidia.com/tesla-v100-sxm2-32gb format, the dcgm-exporter regards the pod as not requesting GPUs so it doesn't add namespace and pod tags to those metrics. I'm wondering if we can make it customizable? I am happy to contribute if the community is ok with it.

Thanks!

[Altair-Bueno]

issues with high load

Indeed it could be related to GPU load. As I noted earlier, we sometimes get CUDA Out of Memory errors due to how big our compute tasks are. But we have no CPU nor memory nor disk issues.

Any observations about k8s cluster health, CPU, memory usage, type of workload, may help us to identity the root of the issue.

Ubuntu 22.04.4 LTS system, composed of 4 nodes (2 Tesla V100-PCIE-32GB each). Multimaster K3s v1.28.3-rc2+k3s2. Running TensorFlow and Pytorch CUDA workloads (primarily training ML but also inference).

Can't really share more details about the workload, as it differs between teams. But CUDA is almost always involved.

What is the time lag between the reboot and the spike?

5 hours difference. Seems unrelated to me.

---

What bugs me most is the fact this software works flawlessly BUT randomly does this.

---

PD: i just recalled that we had a somewhat similar issue with GPUs not being available randomly. The device plugin performs healthchecks on GPUs and ours didn't support that feature. So we disabled healthchecks NVIDIA/k8s-device-plugin#33 (comment) (DP_DISABLE_HEALTHCHECKS=xids). Any idea if might be related?

[nvvfedorov]

It sound like we need to try scenario, when CUDA fails with out of memory errors and device plugin runs with disabled health checks.

[Altair-Bueno]

@nvvfedorov This might be helpful:

GPU Framebuffer Mem Used-data-as-joinbyfield-2024-06-14 12_01_56.csv

The steps were:

1. Training of a Tensorflow Keras model on Jupyter Hub notebook
2. Jupyterhub killed the pod for inactivity (11:53)
3. User launched jupyter notebook again (11:56)
4. User started training for model again

[krishh85]

Hi @nvvfedorov / @Altair-Bueno while the issue is very similar to #272 , it is clearly not due to load. The debugging in #272 point to nvml library errors as MIG device not found.

Als0, for the repro were you able to run a pytorch/tf load overnight or a prolonged period of time? I don't think its matter what the job type is tbh.

[Altair-Bueno]

Als0, for the repro were you able to run a pytorch/tf load overnight or a prolonged period of time? I don't think its matter what the job type is tbh.

Yep. It's only some random spikes during the workload. It doesn't affect the GPU usage in any way

[nvvfedorov]

@Altair-Bueno, @krishh85 , Do you have an example of the workload, that we can run to produce GPU load, similar to yours?

[Altair-Bueno]

Something like this

def create_dense_model():
    model = Sequential()
    model.add(Flatten(input_shape=(num_timesteps_input, num_features_input)))
    model.add(Dense(512, activation='relu'))
    model.add(Dense(256, activation='relu'))
    model.add(Dropout(0.1))
    model.add(Dense(128, activation='relu'))
    model.add(Dense(num_timesteps_output))
    model.compile(optimizer='adam', loss='huber', metrics=['mae', 'mse', 'mape'])
    return model
 
# dataset len 2500
# shape 18x1