What to do if CUDA doesn't work

[natetyoung]

I've been having considerable difficulty getting cuda to work, and I've found very sparse resources on what to do when it doesn't.

I'm on Ubuntu 16.04; cuda is installed; cuDNN is in place; all the cuda samples work properly. There were no problems installing anything; the output of nvidia-smi looks fine, etc. etc.

However, torch.cuda.is_available() returns False, and torch.Tensor().cuda() gives:
THCudaCheck FAIL file=/py/conda-bld/pytorch_1493681908901/work/torch/lib/THC/THCGeneral.c line=66 error=30 : unknown error Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/nate/miniconda3/lib/python3.6/site-packages/torch/_utils.py", line 65, in _cuda return new_type(self.size()).copy_(self, async) File "/home/nate/miniconda3/lib/python3.6/site-packages/torch/cuda/__init__.py", line 272, in __new__ _lazy_init() File "/home/nate/miniconda3/lib/python3.6/site-packages/torch/cuda/__init__.py", line 85, in _lazy_init torch._C._cuda_init() RuntimeError: cuda runtime error (30) : unknown error at /py/conda-bld/pytorch_1493681908901/work/torch/lib/THC/THCGeneral.c:66

Is there some troubleshooting document I should be looking at? Can anyine help me here? I've been searching in circles for quite some time.

The oddest thing is, it worked yesterday, but I woke up today and it doesn't. That suggests to me it was an environment variable issue or something similar, but I haven't found any way of telling what the environment variables are actually supposed to be so I'm at a loss.

[ngimel]

How did you install pytorch? Conda binaries come with their own cuda libraries, and they might be conflicting with those on your system.

[natetyoung]

I did install pytorch using conda, so this is likely the issue. How do I fix it?

[ngimel]

It is easiest if you install pytorch to its own conda environment and use it from that environment, that way the paths are automatically set correctly. Otherwise, make sure that whatever directory conda installed cuda libraries to is searched before you system cuda directory (e.g. export LD_LIBRARY_PATH to point to conda lib directory first).

[GuruMulay]

I had a similar issue today on Ubuntu 16.04 (with similar error trace), where it used to work earlier and it didn't all of a sudden. Before trying any of the solutions I restarted my computer, and it worked fine.

[pavanky]

where it used to work earlier and it didn't all of a sudden. Before trying any of the solutions I restarted my computer, and it worked fine.

This usually happens when the nvidia driver gets updated.

[ngimel]

Heh, another reason not to use .deb driver packages.

[natetyoung]

Came home today and it's working again. Outside of conda environments too. Absolutely no idea why it's now working but my current theory is magic.

[soumith]

i'm going to close this, because i cant reproduce this and your stuff is working magically again :)

[kindlychung]

Probably related: https://discuss.pytorch.org/t/resolved-cuda-runtime-error-30/1116/5

[kirk86]

We had this issue recently on a ubuntu 18 system. Simply

sudo rmmod nvidia_uvm
sudo rmmod nvidia
sudo modprobe nvidia
sudo modprobe nvidia_uvm

Didn't do the trick for us.
The cuda driver was updated to 10 but nvcc tools where still pointing to 9.1, and 9.2 drivers
The weird thing is that this only happened with pytorch while tensorflow was working fine