Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use too much CPU resource #54

Closed
matthew-z opened this issue Nov 8, 2018 · 14 comments · Fixed by #166
Closed

use too much CPU resource #54

matthew-z opened this issue Nov 8, 2018 · 14 comments · Fixed by #166
Labels
enhancement
Milestone
1.2

Comments

@matthew-z
Copy link

matthew-z commented Nov 8, 2018 •
edited

Hi, gpustat -i uses about 80% CPU on my machine, is it expected or a bug?
In contrast, nvidia-smi -l 1 uses less than 10%.

OS: Ubuntu 18.04
Nv Driver: 410.48
CUDA: 10.0.130
CPU: AMD Threadripper 1900x
GPU: 2080Ti + 1080

image
@Stonesjtu
Copy link
Collaborator

whats your interval value

@matthew-z
Copy link
Author

I didn't set, but gpustat -i 1 will reproduce the same result.

@Stonesjtu
Copy link
Collaborator

Can you test if watch -n 1 nvidia-smi and watch -n gpustat use the same amount of CPU time.

@matthew-z
Copy link
Author

I tested, and with watch -n 1 they use the same amount of CPU time (about 0-20%).

@matthew-z
Copy link
Author

Hi, I just found that the CPU time problem of gpustat -i can be solved by running nvidia-smi daemon first.

Stonesjtu added a commit that referenced this issue Nov 9, 2018
@Stonesjtu
Add reference to issue #54
51d63ee
@wookayin
Copy link
Owner

wookayin commented Nov 9, 2018 •
edited

A difference is that in the watch mode (i.e. gpustat -i) handle resources are fetched at every time step, which is somewhat expensive. Therefore we could optimize in a way that GPU handles are fetched only once in the beginning, and use the (cached) resources. This would be possible in the watch mode as the gpustat process won't terminate until interrupted.

@wookayin
Copy link
Owner

wookayin commented Nov 9, 2018

Top four most expensive operations:

 36.50%  36.50%    4.59s     4.59s   nvmlDeviceGetHandleByIndex (pynvml.py:945)
 16.50%  16.50%    1.85s     1.85s   nvmlDeviceGetPowerUsage (pynvml.py:1289)
  9.50%   9.50%    1.18s     1.18s   nvmlDeviceGetUtilizationRates (pynvml.py:1379)
  7.50%   7.50%   0.805s    0.805s   nvmlDeviceGetComputeRunningProcesses (pynvml.py:1435)

@wookayin wookayin added this to the 0.6 milestone Nov 10, 2018
@Stonesjtu
Copy link
Collaborator

@wookayin Good point, I'm working on that.

@Stonesjtu Stonesjtu mentioned this issue Feb 8, 2019
Closed
wookayin added a commit that referenced this issue Feb 24, 2019
@wookayin
Do not include power-related results unless requested (#54)
d334475 
Querying power status (current draw and limit -- may be cached?) is
quite expensive. Thus we can skip querying power status unless
it is explicitly requested. This leads to 50% less CPU usage in my case.

The default behavior of `new_query()` still includes power information.

TODO: Add unit test case for the behavior
@wookayin wookayin mentioned this issue Feb 24, 2019
Closed
@wookayin
Copy link
Owner

wookayin commented Feb 24, 2019 •
edited

Working on this as #61.

In my case querying power usage is most expensive, so I made it optional whenever possible. Could anybody check whether it leads to less CPU usage?

@wookayin wookayin modified the milestones: 0.6, 1.0 Jul 22, 2019
@JalinWang

This comment was marked as resolved.

Sign in to view
@Stonesjtu

This comment was marked as resolved.

Sign in to view
@wookayin wookayin modified the milestones: 1.0, 1.1 Aug 13, 2021
@wookayin wookayin modified the milestones: 1.1, 1.2 Mar 2, 2023
@rkooo567
Copy link

Has this issue been resolved? I am observing this behavior from https://github.com/ray-project/ray/ when we run gpustat.new_query() repetitively at GCE.

profile

@rkooo567
Copy link

Lots of time is spent on NvmlInit & shutdown & nvmlDeviceGetHandleByIndex

wookayin added a commit that referenced this issue Nov 22, 2023
@wookayin
perf: Call nvmlInit() and nvmlShutdown() only once (#54)
9c51ddf 
Lots of time is spent on nvmlInit() and nvmlShutdown() for each
new_query call. When running in a loop mode (-i), we do not need to
initialize and shutdown the nvml library because nvml APIs will be used
throughout the lifespan of the gpustat process.

Upon importing `gpustat.pynvml`, nvmlInit() will always be called.
@wookayin wookayin mentioned this issue Nov 22, 2023
Merged
wookayin added a commit that referenced this issue Nov 24, 2023
@wookayin
perf: Call nvmlInit() and nvmlShutdown() only once (#54)
313b58d 
Lots of time is spent on nvmlInit() and nvmlShutdown() for each
new_query call. When running in a loop mode (-i), we do not need to
initialize and shutdown the nvml library because nvml APIs will be used
throughout the lifespan of the gpustat process.

Upon importing `gpustat.pynvml`, nvmlInit() will always be called.
@wookayin
Copy link
Owner

In the recent versions of pynvml, nvmlDeviceGetHandleByIndex doesn't seem to be a bottleneck according to profiling result (If this is still slow, please let me know) so I did not optimize on redundant calls of nvmlDeviceGetHandleByIndex. #166 makes nvmlInit() called only once, so it should have some performance benefit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
1.2
5 participants
@wookayin @Stonesjtu @matthew-z @rkooo567 @JalinWang