Process not displayed

[ryanxingql]

Describe the bug

My process is displaced by NVIDIA-SMI but not by gpustat.

Screenshots or Program Output

40901                       Fri Jun 30 11:52:07 2023  535.54.03
[0] NVIDIA GeForce RTX 4090 | 50°C,  73 % | 12894 / 24564 MB | user1(12874M) gdm(4M)
[1] NVIDIA GeForce RTX 4090 | 52°C,  49 % | 12894 / 24564 MB | user1(12874M) gdm(4M)
[2] NVIDIA GeForce RTX 4090 | 45°C, 100 % | 15202 / 24564 MB | user1(15182M) gdm(4M)
[3] NVIDIA GeForce RTX 4090 | 57°C,  99 % | 21305 / 24564 MB | user1(15180M) gdm(4M)
[4] NVIDIA GeForce RTX 4090 | 55°C,  71 % |  7444 / 24564 MB | user2(7418M) gdm(4M)
[5] NVIDIA GeForce RTX 4090 | 63°C,  99 % |  5330 / 24564 MB | user2(5304M) gdm(4M)
[6] NVIDIA GeForce RTX 4090 | 25°C,   0 % |    13 / 24564 MB | gdm(4M)
[7] NVIDIA GeForce RTX 4090 | 52°C,  52 % |  4652 / 24564 MB | user3(4632M) gdm(4M)

It can be observed that the PID 3617115 in GPU 3 is not displayed by gpustat, which occupies 6100 MiB memory.

Environment information:

• OS: Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-45-generic x86_64)
• NVIDIA Driver version: NVIDIA-SMI 535.54.03; Driver Version: 535.54.03; CUDA Version: 12.2
• The name(s) of GPU card: NVIDIA GeForce RTX 4090
• gpustat version: gpustat 1.1
• pynvml version: nvidia-ml-py 11.525.112

[wookayin]

Do you see any useful message from gpustat --debug?

[ryanxingql]

Do you see any useful message from gpustat --debug?

Hi,
The output messages of gpustat and gpustat --debug are the same.

[wookayin]

NVIDIA Drivers 535.xx are broken. 535.54 and 535.86 are affected versions, where nvmlProcessInfo_st has a wrong extra field (which now has been removed and reverted).

A workaround is to use nvidia-ml-py == 12.535.77 (must be a very specific version) for these buggy NVIDIA Drivers 535.54 and 535.86. Other versions (like the latest, nvidia-ml-py 12.535.108 as of now) will not report the correct process information.

Let me track this issue in #161.