once gain: tf.2.16.1 fails to recognize GPUs

[JuanVargas]

OS = Linux Ubuntu 22.04.04 LTS
python = 3.10.12
CUDA = 12.4

According to the most recent "install instructions" I coud issue the line below in a python venv to get TF installed with GPU support

python -m pip install tensorflow**

I did that and it failed

tf.version
'2.16.1'

tf.config.list_physical_devices('GPU')
[ ]

so I created another python venv wherein I tried

python -m pip install tensorflow[and-cuda]

that failed too, same error messages.

BTW, my system works wonderfully well with the GPUs when I use pytorch and Julia.

[SuryanarayanaY]

Hi @JuanVargas ,

Have you installed Cuda driver ? Please verify and confirm the output of nvidia-smi command.

Verified the same on colab and attached gist here.

Thanks!

[sampathweb]

Please open this issue in TensorFlow GitHub where you might get better responses. This issue is not related to Keras.

[JuanVargas]

yes, cuda driver is NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4

…

On Mon, Mar 11, 2024 at 7:26 AM Surya ***@***.***> wrote: Hi @JuanVargas <https://github.com/JuanVargas> , Have you installed Cuda driver ? Please verify and confirm the output of nvidia-smi command. Verified the same on colab and attached gist <https://colab.sandbox.google.com/gist/SuryanarayanaY/808553205834b3043b619980c9ae3ffc/63362.ipynb> here. Thanks! — Reply to this email directly, view it on GitHub <#19276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGK34OUBRTMLR6YKNEJ6K3YXWIFZAVCNFSM6AAAAABEOFZPESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGIZDAOJWGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[SuryanarayanaY]

Hi @JuanVargas ,

There is duplicate of # 63362 on tensorflow repo for same issue.Could you please close it here and track it there.It seems the issue is related to CUDA driver compatibility which we can discuss at TF repo.

Thanks!

[JuanVargas]

Done.

…

On Tue, Mar 12, 2024 at 12:19 AM Surya ***@***.***> wrote: Hi @JuanVargas <https://github.com/JuanVargas> , There is duplicate of # 63362 <tensorflow/tensorflow#63362> on tensorflow repo for same issue.Could you please close it here and track it there.It seems the issue is related to CUDA driver compatibility which we can discuss at TF repo. Thanks! — Reply to this email directly, view it on GitHub <#19276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGK34LJSNOWOFCYIZAXZUDYXZ66RAVCNFSM6AAAAABEOFZPESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJQGI2TIMRTHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mihaimaruseac]

This was not closed, instead the issue on TF was closed (and reopened)

[chaudharyachint08]

Almost final and automated fix below

• Where I found the resolution

  • TF 2.16.1 Fails to work with GPUs
    • Solution proposed by "sh-shahrokhi", improved by "ChristofKaufmann"
    • See specially Comment by COntributor
  • Related Issues
    • GPU not detected on WSL2, where I have post some comments
    • Tensorflow WSL GPU CUDA recognition issue RTX3090
    • Once gain: tf.2.16.1 fails to recognize GPUs
  • Other mention on social media
    • Reddit issue

• Exact solution

  • Temporary fix (after activating environment in which Tensorflow 2.16.1 is installed)

      export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
      export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
      

  • Automating the variable set/unset process with Anaconda (one-time setup)
    • Activate your environement in which TF 2.16.1 is installed
    • Two files to be created in "anaconda3/envs/<ENV_NAME>/etc/conda"
      • anaconda3/envs/<ENV_NAME>/etc/conda/activate.d/env_vars.sh

        #!/bin/sh
        export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
        export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
        

      • anaconda3/envs/<ENV_NAME>/etc/conda/deactivate.d/env_vars.sh

        #!/bin/sh
        unset NVIDIA_DIR
        unset LD_LIBRARY_PATH
        

    • Official documentation to do this via conda.io
    • Stack-overflow question where I got this Set environment vars when activating conda env

• What else helped me

  • How to list all variables names and their current values?
  • Conda documenation for "env_vars.sh" for activate/deactivate

[SuryanarayanaY]

Hi @JuanVargas ,

Since the issue is related to tensorflow repo which is still open there. To avoid duplicacy could you please track the issue there at TF repo and close it here. Thanks in advance.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[JuanVargas]

done

…

On Mon, Apr 1, 2024 at 3:45 AM Surya ***@***.***> wrote: Hi @JuanVargas <https://github.com/JuanVargas> , Since the issue is related to tensorflow repo which is still open there. To avoid duplicacy could you please track the issue there at TF repo and close it here. Thanks in advance. — Reply to this email directly, view it on GitHub <#19276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGK34JVIBVYSY4TVV32JFTY3EGABAVCNFSM6AAAAABEOFZPESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZGMZDQMZVGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>