Metric collector should collect more GPU metrics

[msaroufim]

🚀 The feature

I'm seeing customers run commands like the below

nvidia-smi --format=csv,noheader,nounits --query-gpu=utilization.gpu,utilization.memory,memory.total,memory.used,temperature.gpu,power.draw,clocks.current.sm,clocks.current.memory -l 10 > logs/gpu-stats.log &

However in our existing metric_collector.py we have only have metrics for utilization and memory used but customers may also be interested in clock speed and power draw. We should make those available as well

serve/ts/metrics/system_metrics.py

Lines 72 to 88 in e181fee

	for value in info:
	dimension_gpu = [
	Dimension("Level", "Host"),
	Dimension("device_id", value["index"]),
	]
	system_metrics.append(
	Metric(
	"GPUMemoryUtilization",
	value["mem_used_percent"],
	"percent",
	dimension_gpu,
	)
	)
	system_metrics.append(
	Metric("GPUMemoryUsed", value["mem_used"], "MB", dimension_gpu)
	)

Motivation, pitch

This is easy enough to do by using nvml which already instruments this data, we just need to create new metrics objects

Alternatives

No response

Additional context

No response

[msaroufim]

Something I'm a bit worried about, nvgpu doesn't seem to be actively maintained rossumai/nvgpu#26 - I made a PR to fix some stuff but I'm not optimistic it'll get merged - we use it for our metric_collector.py and we might need to pull out some code we need and get rid of this dependency

Also related we have some badly instrumented metrics #1886

[msaroufim]

I think the easiest thing to do would be to extend available metrics in https://github.com/pytorch/pytorch/blob/master/torch/cuda/__init__.py

I will do this