Conflicts with other profiling tools

[FindHao]

Hi, I'm using DCGM on a server with multiple A100 GPUs. When other profiling tools are running such as CUPTI based profilers, the dcgmi will raise an error the third-party profiling module returned an unrecoverable error. It makes sense. But if other profilers only profile on one GPU, and I specify the other GPUs such as dcgmi -i 1, the error doesn't make sense.

Is it possible to implement such a feature that dcgm still works if the specified GPUs are not used and will not be used in the future by other processes?

[nikkon-dev]

@FindHao,

If you are using DCP metrics (1001,1002...) then it's impossible to collect metrics at the same time other profiling tools are enabled. DCGM is able to temporarily pause and resume the DCP metrics collection via dcgmi profile pause/resume command to help is such situations.
Profiling and DCGM both lock the same primitive in the driver, so that does not depend on which GPU you run profiling or the DCGM - it's a system-wide lock.

[FindHao]

@FindHao,

If you are using DCP metrics (1001,1002...) then it's impossible to collect metrics at the same time other profiling tools are enabled. DCGM is able to temporarily pause and resume the DCP metrics collection via dcgmi profile pause/resume command to help is such situations. Profiling and DCGM both lock the same primitive in the driver, so that does not depend on which GPU you run profiling or the DCGM - it's a system-wide lock.

I see. Thanks for your explanation.

But if someone uses only one GPUs on the 8 GPUs cluster, other users can't profile other GPUs at the same time. It's not a smart policy I think. Do you have any suggestions?

[nikkon-dev]

@FindHao,

Unfortunately, for pre-Hopper GPUs, there are few options to mitigate this. You could use virtualization with GPU-passthrough to avoid the lock (each VM would use its driver). Starting from H100, we introduced new hardware counters that allow monitoring and profiling simultaneously without competing for resources.

[FindHao]

@FindHao,

Unfortunately, for pre-Hopper GPUs, there are few options to mitigate this. You could use virtualization with GPU-passthrough to avoid the lock (each VM would use its driver). Starting from H100, we introduced new hardware counters that allow monitoring and profiling simultaneously without competing for resources.

Interesting! Glad to know H100 will solve this issue!
Thanks for your reply.

[FindHao]

I'd like to reopen this issue since I tested some new cases.

I tried to run a dcgmi command on a 8xA100 server. Someone else is using GPU0 with dcgm client. But when I run dcgmi dmon -e 1007 -i 5, the error is the same with the previous discussion. the third-party profiling module returned an unrecoverable error.

But the field id 155 works well. 155 is the field id for power usage and 1007 is the field id for FP32 active.

Does that means if one client use profile field ids others will can't run dcgm clients with profile field ids?

[nikkon-dev]

@FindHao,

Which DCGM version are you using? Multiple dcgm dmon clients for DCP fields (1001...1014) were implemented sometime before version 2.1.0. An exception - there should be only one nv-hostengine process in the system, including embedded engines.

For such an error, we need to see the debug logs from the nv-hostengine to understand the reason. There are several subsystems involved. The fields other than 1001...1014 do not use profiling facilities to get values and thus are unaffected.

[FindHao]

I'm using 2.4.6.
I'll try to get the debug log from the server admin. Thanks for your explanations!