|
Between DCGM_FI_DEV_GPU_UTIL and DCGM_FI_PROF_SM_ACTIVE, which should drive autoscaling decisions? |
Answered by
iacker
Jul 10, 2026
Replies: 1 comment
|
Prefer DCGM_FI_PROF_SM_ACTIVE. GPU_UTIL only reports whether any kernel was running in the sample window, so it saturates at 100% even when the SMs are mostly idle between small kernels. SM_ACTIVE (profiling metric) reflects the fraction of SMs actually doing work, which correlates far better with real headroom. Drive HPA/Kueue signals off SM_ACTIVE and keep GPU_UTIL only as a coarse liveness indicator. |
0 replies
Answer selected by
iacker
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Prefer DCGM_FI_PROF_SM_ACTIVE. GPU_UTIL only reports whether any kernel was running in the sample window, so it saturates at 100% even when the SMs are mostly idle between small kernels. SM_ACTIVE (profiling metric) reflects the fraction of SMs actually doing work, which correlates far better with real headroom. Drive HPA/Kueue signals off SM_ACTIVE and keep GPU_UTIL only as a coarse liveness indicator.