dcgm-exporter running on "g4dn.metal" in AWS EKS fails with "fatal: morestack on gsignal" #208

[sidewinder12s]

We are running the "dcgm-exporter" Kubernetes DaemonsetSet on AWS EKS, and whenever we use a "g4dn.metal" EC2 instance, the "dcgm-exporter" gets stuck in a crashloop with the following log message:

time="2021-08-13T20:07:08Z" level=info msg="Starting dcgm-exporter"
time="2021-08-13T20:07:09Z" level=info msg="DCGM successfully initialized!"
time="2021-08-13T20:07:27Z" level=info msg="Collecting DCP Metrics"
fatal: morestack on gsignal

This does not happen on any other G4DN class of machine, only with the "metal" variant. The NVIDIA drivers are installed and user code utilizing the GPUs is running fine. Using "nvidia-smi" results shows all 8 GPUs as expected. I have done searching and I cannot find any information on this.

Copied from here: NVIDIA/gpu-monitoring-tools#208

[nikkon-dev]

Hello,

Could you reproduce the issue with the GOTRACEBACK=system environment variable set and provide the crash logs?

WBR,
Nik

[sidewinder12s]

Sure, also going to upgrade to see if it's fixed, we're a few releases behind.

[sidewinder12s]

Well, after upgrading and increasing some resources for the exporter, this doesn't appear to be happening anymore. I'll re-open if we see it again with that env var set.