Maximum Sampling Rate of GPU Power

[afzalxo]

Hey! This might be slightly unrelated but I was wondering what the maximum sampling rate of call to nvmlDeviceGetPowerUsage would be. It must be limited by the sampling rate of the power measurement hardware on the GPU. Do you know about this based on your usage of the NVML API?

Thanks!

[Stonesjtu]

We are now restricting the sampling rate to be lower than 10/sec (interval >= 0.1s), however you may try even higher rate to see what happens.

gpustat/gpustat/cli.py

Line 127 in fb0f54e

args.interval = max(0.1, args.interval)

[afzalxo]

Thanks! I tried a very high sampling rate (calling nvmlDeviceGetPowerUsage without any delay) and the number of samples achieved in 1 second is around 2743, however, I don't think GPU power measurement hardware can have 2.7kHz sampling rate (I am using a Tesla K40m). I think the function returns the last measured power if we try to call it when the power measurement hardware is not ready or has recently returned a measured power from the sensor.
I will try the NVIDIA developer forums for exact sampling rate figures.
Closing this issue.

[piotr-dobrogost]

@europa1610
Could you please share a link to the forum post you refer to?

[afzalxo]

Hey @piotr-dobrogost I posted my query on the NVIDIA Devtalk forums (NVML Subtopic) here but haven't received a response to my query yet. Here's my post for reference.