Cudnn version of the problem

[zhangshengsheng]

Loaded runtime CuDNN library: 7.1.2 but source was compiled with: 7.3.1. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.

cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR 7
#define CUDNN_MINOR 4
#define CUDNN_PATCHLEVEL 1

#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#include "driver_types.h"

No 7.1.2 installed.

tf-nightly-gpu 1.13.0.dev20181110

tensorflow-gpu 1.12.0rc2

[PK15946]

I have exactly the same error, do you have any progress yet? @wt-huang

[jjyunlp]

amazing .... check carefully whether there is a 7.1.12 cudnn. Maybe you have only updated the cudnn.h but left the old libcudnn.so?

[abdel]

I have a similar issue but with tensorflow-gpu 1.12 stable, but instead with this error:
Loaded runtime CuDNN library: 7.0.5 but source was compiled with: 7.1.4. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.

Versions:
Ubuntu 16.04: Ubuntu 16.04.2 LTS
TensorFlow 1.12: tensorflow-gpu 1.12.0
CUDA 9.0: Cuda compilation tools, release 9.0, V9.0.176
cuDNN 7.1.4:

#define CUDNN_MAJOR 7
#define CUDNN_MINOR 1
#define CUDNN_PATCHLEVEL 4

There's definitely no 7.0.5, and I even tried to re-install everything from scratch.. so there wouldn't be any leftover 7.0.5 libcudnn.so. Any ideas?

[zubairahmed-ai]

@abdel In my case I had two installations of CUDA on different drives and I was copying cuDNN files to the wrong one initially, so it kepts loading old binaries and gave me the same error until I copied to the right one, make sure you do the same

[abdel]

@zubairahmed-ai Thanks for the tip. There's definitely only one CUDA in our case on a single drive, and all PATHs seem to indicate it should be using that one, even CUDA/cuDNN test samples confirm that.. yet TensorFlow seems to be getting 7.0.5 from somewhere...?

[ZacMilano]

@abdel I am having the exact same problem.

[ZacMilano]

Installing cuDNN 7.1.4 seemed to fix my problem.

[acotterson]

I found a fix on my system! On my Windows computer using Anaconda, I found that using conda for some package (tensorflow? although I'm successfully using a custom tensorflow package) automatically grabbed another version of cudnn as a dependency--the version that was giving me troubles. Updating through conda and pip wouldn't give me a version more recent, so I fixed it through manually updating the cudnn package. I went to my install of Anaconda3 and went here:

C:\Anaconda3\pkgs\cudnn-7.1.4-cuda9.0_0\Library

In here, you find a bin, include, and lib folder. Using the exact same process that was used installing cudnn with cuda, i replaced the cudnn files in these folders with the ones from the version I wanted.
I also went to the library path of my environment

C:\Anaconda3\envs\ai1\Library\lib\x64

and replaced this cudnn.lib file as well.

Hopefully this makes sense and it helps somebody!

[jvishnuvardhan]

I think it was resolved. I am closing the issue. Please open new ticket if you see similar issue again. Thanks!

[XYudong]

add export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}} to the ~/.bashrc file, and then source ~/.bashrc. This solves my problem!

tf:1.12, CUDA 9.0, 384, cuDNN.7.1.4

[Pidem]

I had a similar issue working with TF2. Turns out this was due to the fact that I was using an old conda environment that didn't have access to the same cuDNN.

conda install -c anaconda cudnn saved my day

[wm2012011492]

I have exactly the same error, do you have any solution now?

[mpodlasin]

I also have this error - @jvishnuvardhan can we reopen? :(

[jvishnuvardhan]

@mpodlasin Please open a new issue with your issue details and error trace. Thanks!

[Corly]

I also had this error on Windows. On my case I had installed cupy-coda which also added de cudnn dll and the program always looked there for it.

In order to fix this I just removed the cupy-coda dll from the cupy-cuda install folder (..Python36\Lib\site-packages\cupy\cuda) and then the program loaded the correct cudnn dll from CUDA_PATH.

[berylsheep-up]

I solve the same problem by follow method.

I add the follow values into the System environment variable.
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\libnvvp
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\lib
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\include

[ameicler]

Downgrading to a compatible version (from 1.15 to 1.14) of TF for my CUDA version solved it for me.

Compatility table : https://www.tensorflow.org/install/source#linux

[oerton42]

I also had this error on Windows. On my case I had installed cupy-coda which also added de cudnn dll and the program always looked there for it.

In order to fix this I just removed the cupy-coda dll from the cupy-cuda install folder (..Python36\Lib\site-packages\cupy\cuda) and then the program loaded the correct cudnn dll from CUDA_PATH.

You made my day !!!
Tensorflow did not find cudnn after installiong cupy.

1. install tensorflow (using conda install keras-gpu is easy(
2. install cupy-conda101 or another wheel as explains the cupy website
3. uninstall cudnn.dll from your anaconda env
4. make sure you have the correct version of cudnn installed and your windows path correctly entered

[chandreshiit]

I was having similar issue. I had an old version of cudnn (8.0.4) somewhere in /usr/lib and LD_LIBRARY-PATH was set in .bashrc and somehow still referring the old version of cudnn. If you re getting this error, check if you have an old version of cudnn somewhere. So what I did was added a conf file in /etc/ld.so.conf.d/ and did sudo ldconfig` so that tf can load cudnn8.2.x . Surpringly, my code is running super fast.