tensorflow-gpu doesn't pack CUDA and cuDNN anymore through conda-forge

[BptGrm]

System information

• OS Platform and Distribution : Windows 10
• TensorFlow installed from (source or binary): Binary
• TensorFlow version: 2.3.0
• Python version: 3.8.12
• Installed using virtualenv? pip? conda?: conda
• CUDA/cuDNN version: 10.1/7.6
• GPU model and memory: NVIDIA Quadro P400, 2Gb

Describe the problem

A few months ago, I installed TensorFlow on a machine through conda-forge. After running into some issues installing it at the system level, I found the simplest solution was to install the tensorflow-gpu package in a conda environment.
That meta-package came in with cudatoolkit and cudnn, and it worked out of the box.

However, as I tried to replicate that install on another machine, I found that the list of packages provided when trying to install tensorflow-gpu through conda-forge did not contain cudatoolkit or cudnn.

I also tried again on the machine where I had a successful install, in another environment, and the issue persisted.

Provide the exact sequence of commands / steps that you executed before running into the problem

conda install tensorflow-gpu==2.3.0 -c conda-forge

conda outputs the following :

The following NEW packages will be INSTALLED:

  _tflow_select      pkgs/main/win-64::_tflow_select-2.3.0-gpu
  absl-py            conda-forge/noarch::absl-py-1.0.0-pyhd8ed1ab_0
  aiohttp            conda-forge/win-64::aiohttp-3.8.1-py38h294d835_0
  aiosignal          conda-forge/noarch::aiosignal-1.2.0-pyhd8ed1ab_0
  astor              conda-forge/noarch::astor-0.8.1-pyh9f0ad1d_0
  astunparse         conda-forge/noarch::astunparse-1.6.3-pyhd8ed1ab_0
  async-timeout      conda-forge/noarch::async-timeout-4.0.2-pyhd8ed1ab_0
  attrs              conda-forge/noarch::attrs-21.4.0-pyhd8ed1ab_0
  blinker            conda-forge/noarch::blinker-1.4-py_1
  brotlipy           conda-forge/win-64::brotlipy-0.7.0-py38h294d835_1003
  cachetools         conda-forge/noarch::cachetools-5.0.0-pyhd8ed1ab_0
  cffi               conda-forge/win-64::cffi-1.15.0-py38hd8c33c5_0
  charset-normalizer conda-forge/noarch::charset-normalizer-2.0.12-pyhd8ed1ab_0
  click              conda-forge/win-64::click-8.0.3-py38haa244fe_1
  colorama           conda-forge/noarch::colorama-0.4.4-pyh9f0ad1d_0
  cryptography       conda-forge/win-64::cryptography-36.0.1-py38hb7941b4_0
  frozenlist         conda-forge/win-64::frozenlist-1.3.0-py38h294d835_0
  gast               conda-forge/noarch::gast-0.4.0-pyh9f0ad1d_0
  google-auth        conda-forge/noarch::google-auth-2.6.0-pyh6c4a22f_1
  google-auth-oauth~ conda-forge/noarch::google-auth-oauthlib-0.4.6-pyhd8ed1ab_0
  google-pasta       conda-forge/noarch::google-pasta-0.2.0-pyh8c360ce_0
  grpcio             conda-forge/win-64::grpcio-1.43.0-py38he5377a8_0
  h5py               conda-forge/win-64::h5py-2.10.0-nompi_py38he6c2248_106
  hdf5               conda-forge/win-64::hdf5-1.10.6-nompi_h5268f04_1114
  idna               conda-forge/noarch::idna-3.3-pyhd8ed1ab_0
  importlib-metadata conda-forge/win-64::importlib-metadata-4.11.1-py38haa244fe_0
  intel-openmp       conda-forge/win-64::intel-openmp-2022.0.0-h57928b3_3663
  keras-applications conda-forge/noarch::keras-applications-1.0.8-py_1
  keras-preprocessi~ conda-forge/noarch::keras-preprocessing-1.1.2-pyhd8ed1ab_0
  krb5               conda-forge/win-64::krb5-1.19.2-h20d022d_3
  libblas            conda-forge/win-64::libblas-3.9.0-13_win64_mkl
  libcblas           conda-forge/win-64::libcblas-3.9.0-13_win64_mkl
  libcurl            conda-forge/win-64::libcurl-7.81.0-h789b8ee_0
  liblapack          conda-forge/win-64::liblapack-3.9.0-13_win64_mkl
  libprotobuf        conda-forge/win-64::libprotobuf-3.19.4-h7755175_0
  libssh2            conda-forge/win-64::libssh2-1.10.0-h680486a_2
  libzlib            conda-forge/win-64::libzlib-1.2.11-h8ffe710_1013
  m2w64-gcc-libgfor~ conda-forge/win-64::m2w64-gcc-libgfortran-5.3.0-6
  m2w64-gcc-libs     conda-forge/win-64::m2w64-gcc-libs-5.3.0-7
  m2w64-gcc-libs-co~ conda-forge/win-64::m2w64-gcc-libs-core-5.3.0-7
  m2w64-gmp          conda-forge/win-64::m2w64-gmp-6.1.0-2
  m2w64-libwinpthre~ conda-forge/win-64::m2w64-libwinpthread-git-5.0.0.4634.697f757-2
  markdown           conda-forge/noarch::markdown-3.3.6-pyhd8ed1ab_0
  mkl                conda-forge/win-64::mkl-2022.0.0-h0e2418a_796
  msys2-conda-epoch  conda-forge/win-64::msys2-conda-epoch-20160418-1
  multidict          conda-forge/win-64::multidict-6.0.2-py38h294d835_0
  numpy              conda-forge/win-64::numpy-1.22.2-py38hcf66579_0
  oauthlib           conda-forge/noarch::oauthlib-3.2.0-pyhd8ed1ab_0
  opt_einsum         conda-forge/noarch::opt_einsum-3.3.0-pyhd8ed1ab_1
  protobuf           conda-forge/win-64::protobuf-3.19.4-py38h885f38d_0
  pyasn1             conda-forge/noarch::pyasn1-0.4.8-py_0
  pyasn1-modules     conda-forge/noarch::pyasn1-modules-0.2.7-py_0
  pycparser          conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0
  pyjwt              conda-forge/noarch::pyjwt-2.3.0-pyhd8ed1ab_1
  pyopenssl          conda-forge/noarch::pyopenssl-22.0.0-pyhd8ed1ab_0
  pyreadline         conda-forge/win-64::pyreadline-2.1-py38haa244fe_1005
  pysocks            conda-forge/win-64::pysocks-1.7.1-py38haa244fe_4
  python_abi         conda-forge/win-64::python_abi-3.8-2_cp38
  pyu2f              conda-forge/noarch::pyu2f-0.1.5-pyhd8ed1ab_0
  requests           conda-forge/noarch::requests-2.27.1-pyhd8ed1ab_0
  requests-oauthlib  conda-forge/noarch::requests-oauthlib-1.3.1-pyhd8ed1ab_0
  rsa                conda-forge/noarch::rsa-4.8-pyhd8ed1ab_0
  scipy              conda-forge/win-64::scipy-1.8.0-py38ha1292f7_1
  six                conda-forge/noarch::six-1.16.0-pyh6c4a22f_0
  tbb                conda-forge/win-64::tbb-2021.5.0-h2d74725_0
  tensorboard        conda-forge/noarch::tensorboard-2.8.0-pyhd8ed1ab_1
  tensorboard-data-~ conda-forge/win-64::tensorboard-data-server-0.6.0-py38haa244fe_1
  tensorboard-plugi~ conda-forge/noarch::tensorboard-plugin-wit-1.8.1-pyhd8ed1ab_0
  tensorflow         pkgs/main/win-64::tensorflow-2.3.0-mkl_py38h8557ec7_0
  tensorflow-base    pkgs/main/win-64::tensorflow-base-2.3.0-eigen_py38h75a453f_0
  tensorflow-estima~ conda-forge/noarch::tensorflow-estimator-2.5.0-pyh81a9013_1
  tensorflow-gpu     pkgs/main/win-64::tensorflow-gpu-2.3.0-he13fc11_0
  termcolor          conda-forge/noarch::termcolor-1.1.0-py_2
  tk                 conda-forge/win-64::tk-8.6.12-h8ffe710_0
  typing-extensions  conda-forge/noarch::typing-extensions-4.1.1-hd8ed1ab_0
  typing_extensions  conda-forge/noarch::typing_extensions-4.1.1-pyha770c72_0
  urllib3            conda-forge/noarch::urllib3-1.26.8-pyhd8ed1ab_1
  werkzeug           conda-forge/noarch::werkzeug-2.0.3-pyhd8ed1ab_1
  win_inet_pton      conda-forge/win-64::win_inet_pton-1.1.0-py38haa244fe_3
  wrapt              conda-forge/win-64::wrapt-1.13.3-py38h294d835_1
  yarl               conda-forge/win-64::yarl-1.7.2-py38h294d835_1
  zipp               conda-forge/noarch::zipp-3.7.0-pyhd8ed1ab_1
  zlib               conda-forge/win-64::zlib-1.2.11-h8ffe710_1013

Then, running those lines :

import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))

Outputs the following :

2022-02-17 18:50:41.041229: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2022-02-17 18:50:41.046722: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-02-17 18:50:46.483787: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2022-02-17 18:50:46.490003: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found      
2022-02-17 18:50:46.496485: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found  
2022-02-17 18:50:46.506046: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusolver64_11.dll'; dlerror: cusolver64_11.dll not found
2022-02-17 18:50:46.512642: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found  
2022-02-17 18:50:46.519233: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2022-02-17 18:50:46.524611: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.       
Skipping registering GPU devices...
Num GPUs Available:  0

[0xffccdd]

Looks like you have to be careful you match the correct Nvidia drivers, could be an issue? https://www.tensorflow.org/install/gpu

[mohantym]

@BptGrm ! Could you please check with above comment ?

[BptGrm]

Looks like you have to be careful you match the correct Nvidia drivers, could be an issue? https://www.tensorflow.org/install/gpu

The drivers I have installed can support CUDA 11.2. The TensorFlow install I'm looking for is compatible with CUDA 10.1.
Is there a problem with retrocompatibility that would require downgrading my GPU drivers ?

Also, as I stated before, even on the machine that already has a functional TF install, and thus presumably correct drivers, conda-forge doesn't provide cudatoolkit or cudnn either.

[mohantym]

Please try with "conda install -c anaconda cudatoolkit " command . If that does not resolve the issue please check with instructions from here with TF 2.8 after activating Conda environment?

[BptGrm]

The GPU still isn't recognized, and cuDNN still isn't installed either through conda with cudatoolkit or pip with tensorflow.

[gadagashwini]

@BptGrm,
Follow these steps to install Tensorflow-gpu version with CUDA and cuDNN using Conda package

#Create virtual environment 
conda create --name tf_gpu
#Activate environment 
conda activate tf_gpu
#Install Python 
conda install python==3.9
#Install CUDA
conda install -c anaconda cudatoolkit=11.2
#Install cuDNN
conda install cudnn=8.1
#Install Tensorflow-gpu =~2.8
pip install tensorflow==2.8

[BptGrm]

I have been able to install TF like this previously.
My issue was that those packages used to be wrapped in a single metapackage that, if I remember correctly, would install the correct verions of everything needed.
It was, in my opinion, an easier and safer way to install TF, and I was wondering why it isn't available anymore.

[gadagashwini]

@BptGrm, It used be earlier. When all the compatible packages release happen at the same time, in that case we no need to explicitly mention any of the versions. Binary installer takes the latest stable version. Since when the CUDA and cuDNN and Tensorflow compatibility is required, we need to mention each package version explicitly. Thanks!

[google-ml-butler]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No