Enhance Nvidia Device plugin with more health checking features

[jiayingz]

Quoting what @RenaudWasTaken mentioned in another thread:
"The Nvidia Device plugin has a lot of such features coming up a few of these are:

memory scrubbing
healthCheck and reset in case of bad state
GPU Allocated memory checks
"Zombie processes" checks
...
"

Creating this issue to track the progress on these improvements.

@RenaudWasTaken could you also provide more details on some of these features, like what GPU Allocated memory checks and "Zombie processes" checks do?

[jiayingz]

cc @jiayingz @vishh @mindprince

[flx42]

what GPU Allocated memory checks and "Zombie processes" checks do?

Those are related, and it's related to reset too. If it's not possible to reset your card, you at least need to detect when things are broken.

When things go awfully wrong, you can have the following:

• There is no process using the GPU, but nvidia-smi shows a non-trivial amount of memory being used. e.g. something like 352MiB / 12181MiB
• There is a process already using the GPU you are supposed to give to a new container (excluding voluntary sharing). This can happen when there is a GPU fault and the process that had an open CUDA context can't teardown properly.

These kinds of checks are useful safety checks in addition to event-based healthchecks like XIDs and ECCs. Some of these errors could go unnoticed otherwise.

[ScorpioCPH]

@flx42 Thanks for your detailed explanation, it's very useful for us!

[wsxiaozhang]

@flx42 @jiayingz any plan to enhance current device health check?

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed.