nvidia monitoring

[lars-t-hansen]

This adds GPU monitoring for nvidia cards, as well as zombie hunting functionality for the same. It uses the nvidia-smi tool for (relative) simplicity, but needs to invoke it twice and the output is documented as not being stable, so it may be better to consider going directly to the monitoring libraries. Also, the nvidia code has been folded into ps.rs for now but we could discuss whether it's worth disentangling.

[bast]

Thanks! I read through the change and will also test it locally. One thing I am thinking now is that we should perhaps somehow check whether the node is GPU-able/aware before issuing the NVIDIA_PMON_COMMAND. But I will test how this behaves on computers and nodes that do not have such capability.

[lars-t-hansen]

Thanks! I read through the change and will also test it locally. One thing I am thinking now is that we should perhaps somehow check whether the node is GPU-able/aware before issuing the NVIDIA_PMON_COMMAND. But I will test how this behaves on computers and nodes that do not have such capability.

It works locally on my laptop (no nvidia) and on a local HPC system with only AMD cards, but the reason in both cases is that nvidia-smi is not found. That feels a little brittle and it might be good to have a better test than that. Also, I probably want to expand the functionality to encompass AMD cards.

[bast]

I confirm that it works fine on an nvidia-unaware laptop. Thanks for the work! But I agree that it feels a bit unsafe. I am mostly worried that the process starts hanging somewhere and piling up so we can think about a robust pre-test.

[lars-t-hansen]

Will the timeout in safe_command not prevent us from hanging?

[bast]

Will the timeout in safe_command not prevent us from hanging?

Ideally/hopefully yes. And I hope I tested it well but I am still really worried. But ideally this should prevent it.

[bast]

Thanks!