Design and Implement a good GPU utilization metrics

[Fizzbb]

Not a perfect measure from default Nvidia function.
"Percent of time over the past sample period during which one or more kernels was executing on the GPU."
https://docs.nvidia.com/deploy/nvml-api/structnvmlUtilization__t.html#structnvmlUtilization__t

[Fizzbb]

Some research papers use an indirect way to measure, e.g., how many jobs can be served, what batch size can be served within SLO before and after. Some metrics in the throughput category seem to make sense.