Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found #43193

Closed
CalendulaED opened this issue Sep 13, 2020 · 101 comments
Assignees
Labels
comp:gpu GPU related issues stat:awaiting response Status - Awaiting response from author TF 2.3 Issues related to TF 2.3 type:support Support issues

Comments

@CalendulaED
Copy link

I got this issue when I run import tensorflow as tf in command line

import tensorflow as tf
2020-09-13 17:53:34.644899: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-09-13 17:53:34.649089: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

I use win10 and install the CUDA and cuDNN
But I don't understand why this still happened.
image
image

@CalendulaED CalendulaED added the type:build/install Build and install issues label Sep 13, 2020
@bhack
Copy link
Contributor

bhack commented Sep 13, 2020

Please pip install tf-nightly for CUDA 11

@CalendulaED
Copy link
Author

After I tried pip install tf-nightly, it still shows the same error I think
image

@bhack
Copy link
Contributor

bhack commented Sep 13, 2020

Is not the same error cause now it is looking for 110.

Are you compilant with these steps?

Install the Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017, and 2019. Starting with the TensorFlow 2.1.0 version, the msvcp140_1.dll file is required from this package (which may not be provided from older redistributable packages). The redistributable comes with Visual Studio 2019 but can be installed separately:

  • Go to the Microsoft Visual C++ downloads,
  • Scroll down the page to the Visual Studio 2015, 2017 and 2019 section.
  • Download and install the Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019 for your platform.
  • Make sure long paths are enabled on Windows.
  • Install the 64-bit Python 3 release for Windows (select pip as an optional feature).

@bhack
Copy link
Contributor

bhack commented Sep 13, 2020

You will find more on Windows tabs at https://www.tensorflow.org/install/pip#windows

@CalendulaED
Copy link
Author

Is not the same error cause now it is looking for 110.

Are you compilant with these steps?

Install the Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017, and 2019. Starting with the TensorFlow 2.1.0 version, the msvcp140_1.dll file is required from this package (which may not be provided from older redistributable packages). The redistributable comes with Visual Studio 2019 but can be installed separately:

  • Go to the Microsoft Visual C++ downloads,
  • Scroll down the page to the Visual Studio 2015, 2017 and 2019 section.
  • Download and install the Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019 for your platform.
  • Make sure long paths are enabled on Windows.
  • Install the 64-bit Python 3 release for Windows (select pip as an optional feature).

I think the problem is solved after I Download and install the Microsoft Visual C++ Redistributable.
Thank you so much !

@bhack
Copy link
Contributor

bhack commented Sep 13, 2020

You need also to rely on tf-nightly for CUDA 11 untill TF 2.4 is released.

@CalendulaED
Copy link
Author

I tried to use the example given by https://github.com/tensorflow/tensorflow
after I run the tf.add(1, 2).numpy()
it still shows some errors but evenlty output the 3
will this be a problem?
image

@bhack
Copy link
Contributor

bhack commented Sep 13, 2020

Check that your software requirements are ok https://www.tensorflow.org/install/gpu#windows_setup

@CalendulaED
Copy link
Author

Check that your software requirements are ok https://www.tensorflow.org/install/gpu#windows_setup

I think I have everything it required.
And after I update my GPU driver, it shows the same error as the beginning.
image

@CalendulaED CalendulaED changed the title W Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found Sep 13, 2020
@bhack
Copy link
Contributor

bhack commented Sep 13, 2020

Have you double checked the %PATH%?

@CalendulaED
Copy link
Author

Have you double checked the %PATH%?

I run all four Path in https://www.tensorflow.org/install/gpu#windows_setup.
But it still missing for 110

@bhack
Copy link
Contributor

bhack commented Sep 14, 2020

Have you changed the documentation PATH strings for your CUDA 11 (doc has CUDA 10)?
Have you checked that these paths exist on your system?

@CalendulaED
Copy link
Author

Have you changed the documentation PATH strings for your CUDA 11 (doc has CUDA 10)?
Have you checked that these paths exist on your system?

Yes, I manually added it for the first three, but the last one "C:\tools\cuda\bin", I don't think I have this folder

image
image

@CalendulaED
Copy link
Author

At the beginning, I copy the cuDNN's file to bin include and lib respectively, I tried to add a Path directly to cuDNN and it is still having this error

@bhack
Copy link
Contributor

bhack commented Sep 14, 2020

Have you set in PATH
SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11\bin;%PATH%

Do you have this dir in your system?

@CalendulaED
Copy link
Author

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11\bin

Yes, and I already added to the path, but it still shows the error

@CalendulaED
Copy link
Author

This is all the PATH I have, I think I already include all that required
image
image

@bhack
Copy link
Contributor

bhack commented Sep 14, 2020

Open C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11\bin.
What file do you have there?
Can you see the cudart64_100.dll?

@CalendulaED
Copy link
Author

I think I only have cudart64_110.dll
(The one that I highlighted)
image

@bhack
Copy link
Contributor

bhack commented Sep 14, 2020

Yes was a typo It is correct.
On the same terminal where you are trying to launch Tensorflow can you verify that you have the new path with echo %PATH:;=&echo.%?

@CalendulaED
Copy link
Author

Yes, I believe I have this path in PATH
image

@bhack
Copy link
Contributor

bhack commented Sep 14, 2020

I see a typo there:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11\bin\bin

@CalendulaED
Copy link
Author

I see a typo there:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11\bin\bin

Yes, It is the path to cuDNN, After I download the cuDNN, I copy the bin in cuDNN to here.
I have both:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11\bin\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11\bin

Before I add the
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11\bin\bin
it still doesn't work, so I tired to add it

@CalendulaED
Copy link
Author

And I delete it from PATH, it still has the same error

@bhack
Copy link
Contributor

bhack commented Sep 14, 2020

Do you meant v11.0 or v11?

@CalendulaED
Copy link
Author

Do you meant v11.0 or v11?

I mean v11.0, I delete the one ~/bin/bin
but it is still not working

@bhack
Copy link
Contributor

bhack commented Sep 14, 2020

Is a python 64bit installation?

@bhack
Copy link
Contributor

bhack commented Sep 28, 2020

Nothing specific is to maintain a clean separate environment.

@CalendulaED
Copy link
Author

Nothing specific is to maintain a clean separate environment.

Ok, thank you so much!!!! really help me a lots

@caxelrud
Copy link

caxelrud commented Oct 2, 2020

I have to renamed these files in the bin folder to make it work:
cudart64_101.dll -> cudart64_110.dll
cusparse64_10.dll->cusparse64_11.dll
cudnn64_7.dll->cudnn64_8.dll
cublas64_10.dll->cublas64_11.dll

@caxelrud
Copy link

caxelrud commented Oct 2, 2020

The reality is that each version of Tensorflow uses an specific Cuda and Cudnn. In my case, for TF 2.3 it is Cuda 11.0. Installing this one solves the problem.

@magpayang
Copy link

magpayang commented Nov 22, 2020

I have to renamed these files in the bin folder to make it work:
cudart64_101.dll -> cudart64_110.dll
cusparse64_10.dll->cusparse64_11.dll
cudnn64_7.dll->cudnn64_8.dll
cublas64_10.dll->cublas64_11.dll

also encountered cusolver64_10.dll not found. when I checked the bin folder, it is named as cusolver64_11.dll

@aparico
Copy link

aparico commented Dec 22, 2020

I was able to solve this by following the software requirements here: https://www.tensorflow.org/install/gpu#software_requirements

Here's my system:
NVIDIA Driver 460.89
CUDA 11.0.3
CuDNN 8.0.5.39
Python 3.7.2

I also made sure that the CUDA was compatible with the CuDNN version by checking it here: https://developer.nvidia.com/rdp/cudnn-download

image

Finally, don't forget to set the installation directories to the PATH environment variable. https://www.tensorflow.org/install/gpu#windows_setup

@vanc
Copy link

vanc commented Dec 31, 2020

The key is to install Python from python.org. All versions from Microsoft App Store don't work. Tried both 3.7, 3.8. Once installed the official version 3.8.7 downloaded from python.org, everything is fine now.

This is my current system:

NVIDIA Driver 460.89
CUDA 11.0.3
CuDNN 8.0.5.39
Python 3.8.7 (from python.org)

I also tried with CUDA 11.1.1. It also worked after renaming cusolver64_11.dll to cusolver64_10.dll. It's a known issue tracked in issue #44291.

@GPlaying
Copy link

I got a way to solve this problem, you can download cudart64_101.dll to C:\Users\hp and C:\Windows\System32 from
https://www.ijinshan.com/filerepair/cudart64_101.dll.shtml.

@Rob192
Copy link

Rob192 commented Jan 21, 2021

Hello,
Thanks @aparico for the tip ! It did work with this configuration and pip install tensorflow. However, the software versions in the software requirements section of the https://www.tensorflow.org/install/gpu#software_requirements are not correct as they recommand CUDA 10.1 and CuDNN 7.6 and the install works with CUDA 11.0 and CuDNN 8.0.
Maybe this page needs update ?

@mefistaa
Copy link

mefistaa commented Jun 17, 2021

CUDA 11.2
cuDNN 8.1
tensorflow 2.5
python 3.9.5

in virtual env write "where cudart64_110.dll" and get path.
when taping "import tensorflow as tf" get:
"W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found"

Tried to install tf-nightly, tensorflow
I have been installing to main enviroment and to virtual enviroment, in both cases have got the same result.
VC_redist.x64 installed

Solution:

import os
os.add_dll_directory("C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2/bin") 

Before importing tensorflow in virtualenv and still does not work in main env.

@OmarJay1
Copy link

Thank you for posting the os.add_dll_directory tip. I believe the behavior is by design for newer versions of Python on Windows. The Windows Path environment variable no longer works. That's what I'm guessing is going on.

https://docs.python.org/3/whatsnew/3.8.html#bpo-36085-whatsnew

Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution.

@nuramzan
Copy link

I do an easy way to deal with this problem like below :

  1. Install anaconda with python 3.8.8
  2. Install tensor flow : pip install tensorflow -> got an error like above
    "W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found"
  3. Install package -> pip install tf-nightly
  4. Install the Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017, and 2019 -> https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0
  5. Verify the system has a CUDA-capable GPU -> http://developer.nvidia.com/cuda-gpus.
    image
  6. Using Conda to Install the CUDA Software (https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/) -> conda install cuda -c nvidia
  7. Verify the Installation of CUDA toolkit -> nvcc -v
    image
  8. import tensorflow as tf is succeed, and you are ready to implement deep learning using jupyter notebook
    Hope this information is helpful....

@BornSupercharged
Copy link

BornSupercharged commented Sep 27, 2021

I had this error, as well as:
Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found

Here's what worked for me, with Python 3.9.6 running on my Windows 10 laptop with a mobile RTX 3080.

Upgrade to the latest NVidia drivers:
-- https://www.nvidia.com/drivers

Install CUDA 11.4:
-- https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exe_local

Install cuDNN 8.2.4 for CUDA 11.4:
-- https://developer.nvidia.com/rdp/cudnn-download
-- choose cuDNN Library for Windows (x64), extract to C:\tools so you have C:\tools\cuda\bin\cudnn64_8.dll

Update your Windows %PATH%, by going to Windows Advanced System Properties -> Environment Variables -> PATH -> New, adding these:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\extras\CUPTI\lib64
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\include
C:\tools\cuda\bin

Download and install the x64 C++ Redistributable:
-- https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0

Download and install the x86 C++ Redistributable:
-- https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0

pip install tensorflow
pip install tf-nightly

Reboot your computer.

Works perfectly now.

@lbiemans
Copy link

What fixed it for us:

import os
os.add_dll_directory("C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2/bin") # (Where v11.2 is your version)

@twinssbc
Copy link

add_dll_directory also works for me. This is due to since Python 3.8, %PATH% is not longer in the dll search path.
https://docs.python.org/3/library/os.html#os.add_dll_directory

@RajdeepBorgohain
Copy link

this works like a gem :)

@brechtdecock
Copy link

Hi, i have a similar issue. Made a post about it https://stackoverflow.com/questions/73886668/tensorflow-gpu-could-not-load-dynamic-library-cudart64-110-dll, tried some suggentions from here but things are still not working

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:gpu GPU related issues stat:awaiting response Status - Awaiting response from author TF 2.3 Issues related to TF 2.3 type:support Support issues
Projects
None yet
Development

No branches or pull requests