reboot after a nvidia-smi error

[georgettica]

while installing on a machine, I got an error stated in https://stackoverflow.com/questions/43022843/nvidia-nvml-driver-library-version-mismatch

(the step it failed for me is lost forever now)
but I found a reboot did solve this issue.

hopefully I can create a PR to mitigate this for the next runner up

[ajdecon]

@georgettica : Can you clarify the steps you used here? It's not clear to me that this is a DeepOps issue, but rather part of the normal behavior of the NVIDIA driver.

In general, we do expect that after a driver install or upgrade, the system will need to be rebooted. (Or all processes using the GPU will need to be stopped, and the kernel modules unloaded; but rebooting is the most reliable way to do that!)

[georgettica]

I used the slurm method of course installation and for sure this is a driver issue.
But we can mitigate it via rebooting once this exact error is raised

[ajdecon]

Closing because this isn't a DeepOps issue. That said, if you want to open a PR at some point and include logic to work around the driver issue with a reboot, I'd be happy to review it. 😄