How to show processes in container with cmd nvidia-smi?

[fredy12]

Hi, in container I execute nvidia-smi shows: ( it shown nothing)
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+

But in physical-host when I execute nvidia-smi show: (it shown Process with pid 28290)
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 28290 C python 1012MiB |
+-----------------------------------------------------------------------------+

How can I show the running processes in container?
I figure out one way is to run a container with --pid=host, any other graceful ways?

[flx42]

It's a current limitation of our driver, sorry about that!
As you realized, it is related to PID namespaces, the driver is not aware of the PID namespace and thus nvidia-smi in the container doesn't see any process running.

[fredy12]

Thanks for you reply. @flx42
So we can consider that add PID namespace perception into the driver.

[qiaohaijun]

mark it

[qiaohaijun]

@fredy12 your way works

[loretoparisi]

This should be implemented. 👍

[dharmeshkakadia]

How can we help make this happen?
Checking the usage of GPU is part of workflow when running large training. Without the support for nvidia-smi inside container, there is no good way to keep an eye on the usage. 😿

[therc]

Are there plans to fix this soon? BTW, this is a problem with not only the nvidia-smi tool, but also when calling NVML directly.

[bhack]

Any news about this? Seems to me quite fundamental.
nvmlDeviceGetAccountingStats cannot work with getpid() argument for the process itself inside nvidia-docker.

[bhack]

@flx42 Can we track this issue somewhere? Cause seems that this is closed.

[Breakend]

+1

[bhack]

Please reopen this

[nvjmayo]

re-opening. It's a driver limitation and still exists. There is a mediocre work-around using node-manager but it is unsatisfactory for some work flows.

Requirement for closure is for the nvidia-smi to at least show processes running in the same container. The underlying libraries and drivers can't support this under the current architecture, don't expect this to be fixed any time soon. No ETA, sorry!