kernel version 367.44.0 does not match DSO version 352.99.0 #4349
Description
I can't seem to load GPUs. After a first install of CUDA, I didn't have this error, but tensorflow couldn't find any GPUs (despite nvidia-smi finding one). After a second install based on these instructions: http://tech.marksblogg.com/tensorflow-nvidia-gtx-1080.html I had the following error.
Thanks for your help!
Environment info
Operating System: Ubuntu 14.04
Installed version of CUDA and cuDNN:
-rw-r--r-- 1 root root 189170 Sep 12 23:50 /usr/local/cuda/lib/libcudadevrt.a
lrwxrwxrwx 1 root root 16 Sep 12 23:50 /usr/local/cuda/lib/libcudart.so -> libcudart.so.7.5
lrwxrwxrwx 1 root root 19 Sep 12 23:50 /usr/local/cuda/lib/libcudart.so.7.5 -> libcudart.so.7.5.18
-rwxr-xr-x 1 root root 311596 Sep 12 23:50 /usr/local/cuda/lib/libcudart.so.7.5.18
-rw-r--r-- 1 root root 558020 Sep 12 23:50 /usr/local/cuda/lib/libcudart_static.a
Compiled from:
http://developer.download.nvidia.com/compute/cuda/7.5/Prod/local_installers/cuda_7.5.18_linux.run
- The output from
python -c "import tensorflow; print(tensorflow.__version__)".
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:102] Couldn't open CUDA library libcudnn.so. LD_LIBRARY_PATH: /usr/local/cuda/lib64:
I tensorflow/stream_executor/cuda/cuda_dnn.cc:2259] Unable to load cuDNN DSO
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
0.10.0
If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)
tf.Session()
E tensorflow/stream_executor/cuda/cuda_driver.cc:491] failed call to cuInit: CUDA_ERROR_NO_DEVICE
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:153] retrieving CUDA diagnostic information for host: XXX
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:160] hostname: XXX
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:185] libcuda reported version is: 352.99.0
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:356] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module 367.44 Wed Aug 17 22:24:07 PDT 2016
GCC version: gcc version 4.9.4 (Ubuntu 4.9.4-2ubuntu1~14.04.1)
"""
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] kernel reported version is: 367.44.0
E tensorflow/stream_executor/cuda/cuda_diagnostics.cc:296] kernel version 367.44.0 does not match DSO version 352.99.0 -- cannot find working devices in this configuration
I tensorflow/core/common_runtime/gpu/gpu_init.cc:81] No GPU devices available on machine.