Reliably repeating pytorch system crash/reboot when using imagenet examples

[castleguarders]

So I have a 100% repeatable system crash (reboot) when trying to run the imagenet example (2012 dataset). resnet18 defaults. The crash seems to happen at Variable.py at torch.autograd.backward(..) (line 158).

I am able to run the basic mnist example successfully.

Setup: Ubuntu 16.04, 4.10.0-35-generic #39~16.04.1-Ubuntu SMP Wed Sep 13 09:02:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

python --version Python 3.6.2 :: Anaconda, Inc.

/usr/local/cuda/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

nvidia-smi output.
Sat Oct 7 23:51:53 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.81 Driver Version: 384.81 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:03:00.0 On | N/A |
| 14% 51C P8 18W / 250W | 650MiB / 11170MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1335 G /usr/lib/xorg/Xorg 499MiB |
| 0 2231 G cinnamon 55MiB |
| 0 3390 G ...-token=C6DE372B6D9D4FCD6453869AF4C6B4E5 93MiB |
+-----------------------------------------------------------------------------+

torch/vision was built locally on the machine from master. No issues at compile or install time, other than the normal compile time warnings...

Happy to help get further information..

[vadimkantorov]

I have experienced random system reboots once due to motherboard - GPU incompatibility. This has manifested during long training. Do other frameworks (e.g. caffe) succeed in training on ImageNet?

[castleguarders]

Haven't tried that yet. However ran some long running graphics bench ;) with no problems. I could probably look at giving other frameworks a shot, what's your recommendation. Caffe?

Do keep in mind, the crash I reported happens practically immediately (mnist-cuda example runs to completion many times without an issue). So I doubt that it's a h/w incompatibility issue.

[apaszke]

Can you try triggering the crash once again and see if anything relevant is printed in /var/log/dmesg.0 or /var/log/kern.log?

[castleguarders]

Zero entries related to this in either dmesg or kern.log. The machine does an audible click and resets, so I think it's the h/w registers or memory being twiddled in a way it doesn't like. No real notice to kernel to log anything. Reboots at the same line of code each time, at least the few times I stepped through it.

[apaszke]

That's weird. To be honest I don't have any good ideas for debugging such issues. My guess would be that it's some kind of a hardware problem, but I don't really know.

[soumith]

it's definitely a hardware issue as well. Whether it's at an nvidia driver level, or a bios / hardware failure.
I'm closing the issue, as there's no action to be taken on the pytorch project side.

[castleguarders]

For future reference, the issue was due to steep power ramp of 1080ti's triggering server power supply over voltage protection. Only some pytorch examples caused it to show up.

[yurymalkov]

@castleguarders Have you figured out how to solve this issue? It seems that even 1200W "platinum" power supply is not enough for just 2X 1080Ti, it reboots from time to time.

[pmcrodrigues]

@castleguarders I am having similar issues, how did you found that that was the problem?

[castleguarders]

@pmcrodrigues There was an audible click whenever the issue happened. I used nvidia-smi to soft control the power draw, this allowed the tests a bit longer, but trip anyways. I switched to a 825W Delta powersupply and it took care of the issue fully. Furmark makes easy work of testing this if you run windows. I ran it fully pegged for a couple of days, while driving the CPUs 100% with a different script. It's zero issues since then.

@yurymalkov I only have 1x 1080ti, didn't dare to put a second one.

[yurymalkov]

@pmcrodrigues @castleguarders
I've also "solved" the problem by feeding the second GPU from a separate PSU (1000W+1200W for 2X 1080Ti). Reducing the power draw by 0.5X via nvidia-smi -pl also helped, but it killed the performance. Also tried different motherboards/GPUs but it didn't help.

[pmcrodrigues]

@castleguarders @yurymalkov Thank you both. I have also tried to reduce power draw via nvidia-smi and it stopped crashing the system. But with stress tests at full power draw simultaneously on my 2 xeons (with http://people.seas.harvard.edu/~apw/stress/) and the 4 1080ti (with https://github.com/wilicc/gpu-burn) didn't make it crash. So for now I have only seen this problem on pytorch. Maybe I need other stress tests?

[yurymalkov]

@pmcrodrigues gpuburn seems to be a bad test for this, as it does not create steep power ramps.
I.e. a machine could pass gpuburn with 4 gpus, but failed at 2 gpus with a pytorch script.

The problem reproduces on some other frameworks (e.g. tensorflow), but it seems that pytorch scripts are the best test, probably because of the highly synchronous nature.

[gurkirt]

I am having the same issue. Has anybody found any soft solution to this?
I have 4 GPU system with one CPU and 1500W power supply. Using 3 out of 4 or 4/4 causes the reboot.
@castleguarders @yurymalkov @pmcrodrigues How to reduce power draw via nvidia-smi?