logger: track system metrics automatically

[aguschin]

Right now dvclive doesn't track system metrics, to name a few, CPU, GPU, RAM utilisation. This is useful when comparing experiment results (e.g., I can get +1% accuracy, but what is the price in terms of time/resources?) and analysing how long it takes to run training on different GPUs (e.g., I can rent another GPU model and get 2x speedup?).

The usual practice is to log this somehow manually with https://github.com/giampaolo/psutil and analyse the results later, but because heavy experiments in ML require this quite often, IMO it makes sense to have this functionality out-of-the-box.

Also it would be great to have summary on these metrics in .json file produced in experiment to make quick decisions instead of diving too deep (e.g., the average CPU utilisation was 4 cores, my script doesn't utilise all 32 cores I have; the peak RAM utilisation was 8GB, that means I can rent a smaller server on aws to run this training; etc).

To name one example among ml tools, this is already tracked with W&B, see the bottom of dashboard: https://wandb.ai/stacey/estuary?workspace=user-lavanyashukla

This page states main metrics logged in W&B https://docs.wandb.ai/ref/app/features/system-metrics

If this will be useful, I could gather a list of metrics with notes describing the cases these metrics are helpful to a user in ml tasks.

[pared]

Well, currently summary is just a gathering of lastly logged metrics. It seems that those system metrics would provide a lot of information that some users might find useful.
Some questions that needs answering before proceeding implementation:

1. How psutil gathers the data? If its from current point of time, we might get untrustworthy results (since logging usually happens in the end of epoch.
2. How to store the summary? Make summary.json just log all values? Or update the structure, to contain, for example "metrics", "system_performace_metrics" subdicts?

[dberenbaum]

1. How psutil gathers the data? If its from current point of time, we might get untrustworthy results (since logging usually happens in the end of epoch.

This is in the docs:

wandb automatically logs system metrics every 2 seconds, averaged over a 30 second period.

[pared]

We could probably achieve that by creating dvclive context that would start "system observer" thread.

[daavoo]

This open source repo has a "hardware monitoring" functionality implemented with psutil (and py3nvml for GPUs):

https://github.com/labmlai/labml/blob/master/guides/hardware_monitoring.md

https://github.com/labmlai/labml/search?q=psutil

[daavoo]

I think this feature makes more sense to be implemented with a wider scope as part of DVC pipelines

[dberenbaum]

@daavoo Reopening to discuss. Does at least some of this info (CPU, GPU, RAM utilization) make sense in dvclive? For one thing, I think it's important to monitor those in realtime rather than only have a summary at the end.

[daavoo]

Reopening to discuss. Does at least some of this info (CPU, GPU, RAM utilization) make sense in dvclive? For one thing, I think it's important to monitor those in realtime rather than only have a summary at the end.

Yes, I think a simple approach that collects the numbers and use log_metric makes sense to be implemented in DVCLive