Check failed: error == cudaSuccess (30 vs. 0) unknown error

[BrandyJer]

Hello,

I have a problem with using GeForce GTX 1080Ti for machine learning (CAFFA framework)
My platform:
ubuntu 16.04, Cuda V8.0.61, CuDNN8.0

I suggest my version is too new and I have to downgrade.
Could you advise the best way for solve my problem?

Find in following more details:

nvcc is warning about deprecation. But it not error and as I know it is about future.

nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release

CAFFE & py-faster-rcnn install with no error, but on training with py-faster-rcnn I recieve next massage:

I0601 15:30:44.833746 28338 layer_factory.hpp:77] Creating layer input-data
I0601 15:30:44.834151 28338 net.cpp:106] Creating Layer input-data
I0601 15:30:44.834161 28338 net.cpp:411] input-data -> data
I0601 15:30:44.834169 28338 net.cpp:411] input-data -> im_info
I0601 15:30:44.834178 28338 net.cpp:411] input-data -> gt_boxes
F0601 15:30:44.852488 28338 syncedmem.hpp:18] Check failed: error == cudaSuccess (30 vs. 0)  unknown error
*** Check failure stack trace: ***

More outputs:

~/caffe#nvcc -V gives

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61

~/caffe# nvidia-smi 
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 378.13                 Driver Version: 378.13                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Graphics Device     Off  | 0000:01:00.0      On |                  N/A |
| 23%   39C    P8    17W / 250W |    578MiB / 11171MiB |     15%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1155    G   /usr/lib/xorg/Xorg                              20MiB |
|    0      1341    G   /usr/lib/xorg/Xorg                             262MiB |
|    0      1794    G   compiz                                          82MiB |
|    0      1939    G   fcitx-qimpanel                                   9MiB |
|    0      2015    G   ...el-token=3645A468136299F390B7B0886FE96671   173MiB |
+-----------------------------------------------------------------------------+

[lukeyeager]

Sounds like a Caffe problem. What version of Caffe are you using? You might want to ask over at NVIDIA/caffe instead of here.

You can ignore the warnings about sm20, etc.

[BrandyJer]

I have solved this problem with
sudo ldconfig /usr/local/cuda/lib64
And just in case someone encounters the same problem this link may be helpful: https://askubuntu.com/questions/889015/cant-install-cuda-8-but-have-correct-nvidia-driver-ubuntu-16
Thank you for your reply anyway!

[barnoncb]

CUDA runtime error (30) might show if your program is unable to create or open the /dev/nvidia-uvm device file. This is usually fixed by installing package nvidia-modprobe:

sudo apt-get install nvidia-modprobe

thanks to @gheinrich

[LaviniaChen]

Hi, I met the same problem yesterday, Google search gave me a lot of related posts but none of them solved my case. Actually, this CudaSuccess (30 vs 0) error may not related to a specific problem, but usually indicates that the GPU driver or CUDA tool kits is not functioning well.

In my case (Ubuntu14.04), the GPU driver version (384) is somehow incompatible with my CUDA toolkit 8.0. At that time, I installed GPU dirver and CUDA toolkits seperately without knowing they can not work with each other. I followed the CUDA installation guide provided by Nvidia to remove my previous GPU driver and CUDA toolkit. After that, I downloaded CUDA toolkit from Nvidia again. This time, I install GPU driver along with CUDA tookits and the problem was gone.

Hope my experience will be useful.

[laibruce]

you can check whether the CUDA version is suitable for GPU driver version , by running the CUDA example deviceQuery in sample directory. For the result, check the item " CUDA Driver Version / Runtime Version ",if they are different, please change(reinstall) another GPU version, or CUDA version, make sure the number in the item is same.

[arya-coding]

Indeed, the problem was the version of Nvidia driver and CUDA.
As @laibruce said, to easily check this :

cd /usr/local/cuda/samples/1_Utilities/deviceQuery
sudo make
./deviceQuery

With Nvidia 384 drivers and CUDA 8.0. I had : CUDA Driver Version / Runtime Version 9.0 / 8.0

[Light--]

sudo ldconfig /usr/local/cuda/lib64
@BrandyJer
use$ sudo ldconfig /usr/local/cuda/lib64
but outputs:

/sbin/ldconfig.real: /usr/local/cuda/lib64/libcudnn.so.7 is not a symbolic link