CUDA 7.5 fails with pip install and docker (Ubuntu 14.04)

[soumith]

Installing via:

# For GPU-enabled version (only install this version if you have the CUDA sdk installed)
$ pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl

Tried to run the alexnet_benchmark.py and it's looking for CUDA 7.0 specifically.

I have CUDA 7.5 on my machine.

Full stack:

Traceback (most recent call last):
  File "alexnet_benchmark.py", line 21, in <module>
    import tensorflow.python.platform
  File "/home/awesomebox/anaconda/lib/python2.7/site-packages/tensorflow/__init__.py", line 4, in <module>
    from tensorflow.python import *
  File "/home/awesomebox/anaconda/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 22, in <module>
    from tensorflow.python.client.client_lib import *
  File "/home/awesomebox/anaconda/lib/python2.7/site-packages/tensorflow/python/client/client_lib.py", line 35, in <module>
    from tensorflow.python.client.session import InteractiveSession
  File "/home/awesomebox/anaconda/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 11, in <module>
    from tensorflow.python import pywrap_tensorflow as tf_session
  File "/home/awesomebox/anaconda/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in <module>
    _pywrap_tensorflow = swig_import_helper()
  File "/home/awesomebox/anaconda/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: libcudart.so.7.0: cannot open shared object file: No such file or directory

Tried the docker install, but the docker image is configured for a particular NVIDIA driver version, and doesn't work with others. (this is a known issue: docker driver version and system driver version must exactly match)

[soumith]

@nivwusquorum haha that's a terrible workaround, as it'll start issues with other libraries. Thanks a lot though. I'm installing CUDA 7.0

[lukesimo]

@nivwusquorum no.

I'm running into the same issue. Looks like I'll be downgrading then.

[ebrevdo]

Out of curiosity, have you set your LD_LIBRARY_PATH to your cuda installation's lib64 directory?

[lukesimo]

@ebrevdo yes

printenv LD_LIBRARY_PATH
/usr/local/cuda-7.5/lib64

[mdda]

On the subject of CUDA library versions ... CUDA 7.0 works for me (as expected), but it really insists on cuDNN 6.5 (which Nvidia now has as 'legacy').

Exact same library locations, etc, but downgrading from cuDNN 7.0 to 6.5 worked.

[graphific]

yes its within the tensorflow code, so just some simple python hacking wont solve it :)
(_pywrap_tensorflow.so when you pip install the binary):

_mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: libcudart.so.7.0: cannot open shared object file: No such file or directory

[emergix]

i assume i have same problem:
I have the 7.5 installed with tensorflow and when I try (like in the tutorial about gpu)
with tf.device('/gpu:0'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
print(c)
sess.run(c)

it breaks !
(in torch7, I have no pbs with gpus)

[jimaldon]

Can we assume tensorflow to be forward compatible with cuda 7.5?

[andorremus]

I've got the same problem.

So are you saying that downgrading from cuda 7.5 to 7 should do the trick?

[andorremus]

If it helps, I've installed cuda toolkit 7.0 and changed the bash profile reference to the new one and it works.

[emergix]

yes you do not need to reinstall the cuda 7.0 driver, just provide the path to libcudart.so.7.0 at the end of the LD_LIBRARY_PATH variable. After discussing of it with someone of the team, they told me a strange story. It appears thar the old CUDA drivers are very much in demand by the people using AWS of amazon !
can someone confirm ?

[ebrevdo]

Long story short: tensorflow currently requires cuda 7.0. If you install version 7.0 in a separate directory from 7.5, and point tensorflow at it via the configure script (or LD_LIBRARY_PATH), it will work. Leaving this open to track future upgrades to the 7.5 SDK.

[FabHan]

I'm curious why it is hard to upgrade to CUDA 7.5 and CuDNN v3? Anyone helps me understand?

[pannous]

@emergix so you got it to work just by providing libcudart.so.7.0 without reinstalling / downgrading to old cuda?

[andorremus]

Yes. It doesn't require you to uninstall the previous one. You can just install them separately and reference v7.0 in your bashrc file

[pannous]

Thanks! Workaround confirmed here.

[fivejjs]

symbolic link libcuda_.7.5 to libcuda_.7.0
It works.

[esube]

This issue and co. are open for more than two months now. Is there any progress to support 7.5 cuda and 7.0 cudnn other than the workarounds? This could be a turn off for some people who already have been working with 7.5 cuda for a while.

[kmhofmann]

I completely agree. Ideally (assuming reasonable API/ABI stability on NVIDIA's side), TensorFlow should not be dependent on specific older versions of CUDA and cuDNN. (I'd rather understand it if the latest version was required to make use of certain features.)
This is the one issue that puts me off using TensorFlow with GPU support.

[cmcneil]

+1 Please support CUDA 7.5

[ville-k]

If you want to try out CUDA 7.5 under Linux, you could try building my pull request branch:
#664
It adds support for CUDA on OSX and uses CUDA 7.5 when building under OSX. I haven't tried 7.5 under Linux, but it seems to work ok with OSX.
The only change you'd need to make is to edit the configure file and set CUDA_VERSION='7.5' when Linux is detected. Lines 48-49 would look like:

if [ "$OSNAME" == "Linux" ]; then
  CUDA_VERSION='7.5'