Could not load dynamic library 'cupti64_110.dll'; dlerror: cupti64_110.dll not found

[ghost]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 Education
• TensorFlow version: tf-nightly-2.4.0.dev20200907
• Python version: 3.8
• Installed using virtualenv? pip? conda?: pip
• CUDA/cuDNN version: 11.0/8.0.2
• GPU model and memory: GeForce GTX 1050, 4GB

Describe the problem
cupti64_110.dll cannot be found, and the only CUPTI .dll file I can find in the CUDA 11.0 files is C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\extras\CUPTI\lib64\cupti64_2020.1.1.dll

Am I missing some files/part of the installation?

Thanks

[bhack]

I suppose CUDA 11.0 Update 1 renamed the library but TF code it is not ready.

[bhack]

Check JuliaGPU/CUDA.jl#300

[XingjiaHan]

I also encountered this problem

[XingjiaHan]

add this in your code
import tensorflow as tf
config = tf.compat.v1.ConfigProto(gpu_options=tf.compat.v1.GPUOptions(allow_growth=True))
sess = tf.compat.v1.Session(config=config)

[sanjoy]

This is a known issue in TF that was caught very late in the 2.4 release process. So for TF 2.4 you'll have to use a workaround -- copy cupti64_2020.1.1.dll to cupti64_110.dll to use the profiler on Windows. We'll fix this for TF 2.5.

[brianmanderson]

can confirm solution works, thank you @sanjoy !

[king398]

Can you please give a link to the file

[thunguyenth]

@king398: as @sanjoy said, you can create a copy of the file "cupti64_2020.1.1.dll" (which is already in the folder CUPTI) and then change the file's name to "cupti64_110.dll "

[dickreuter]

still having this issue

[claeyzre]

As a Windows User, I solved this issue by using a Python installed from the official repository. Previously my Python was installed (naively) using the Microsoft Store.

As these were the exact same versions for Windows I can't explain why. My guess would be that the Python installed by the Microsoft store wasn't given enough permission to see the dlls maybe ?

[bhack]

As a Windows User, I solved this issue by using a Python installed from the official repository. Previously my Python was installed (naively) using the Microsoft Store.
As these were the exact same versions for Windows I can't explain why. My guess would be that the Python installed by the Microsoft store wasn't given enough permission to see the dlls maybe ?

Yes you will find this reported in multiple ticket, you need to use python from the official repository. /cc @MarkDaoust Do we have this info somewhere in the official Doc?

[MarkDaoust]

Not that I know of. It seems like the relevant page would be: https://www.tensorflow.org/install/pip

Or under "windows setup" on the install GPU page: https://www.tensorflow.org/install/gpu#windows_setup

I'm not sure I understand the problem well enough to suggest an update. What's this "official repository"?

@lamberta owns the install pages

[bhack]

@MarkDaoust see #36111 (comment)

[rdslater]

I have the same issue:
Windows 10
Python 3.6.1
TensorFlow 2.5.0-gpu
CUDA 11.2
cuDNN 8.1

2021-05-28 09:23:53.958843: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1611] Profiler found 1 GPUs
2021-05-28 09:23:53.960936: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cupti64_112.dll'; dlerror: cupti64_112.dll not found
2021-05-28 09:23:53.961837: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cupti.dll'; dlerror: cupti.dll not found
2021-05-28 09:23:53.962304: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1661] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2021-05-28 09:23:54.039846: I tensorflow/core/profiler/lib/profiler_session.cc:66] Profiler session collecting data.
2021-05-28 09:23:54.040074: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1752] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
2021-05-28 09:23:54.052862: I tensorflow/core/profiler/internal/gpu/cupti_collector.cc:673]  GpuTracer has collected 0 callback api events and 0 activity events. 
2021-05-28 09:23:54.057927: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session tear down.

I had an almost identical setup and the dll is still named cupti64_2020.3.1.dll
However, the renaming "trick" works cupti64_2020.3.1.dll -> cupti64_112.dll
I put all the files from the "Extras" folder in the proper Cuda (lib to lib, include to include etc...) rather than add to my path.
Win 10
TF-gpu 2.5.0
CuDNN 8.1.1
CUDA 11.2.1
python 3.8.7

[zainulabidin302]

2021-07-26 19:00:59.081093: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1611] Profiler found 1 GPUs
2021-07-26 19:00:59.086072: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cupti64_112.dll'; dlerror: cupti64_112.dll not found
2021-07-26 19:00:59.092193: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cupti.dll'; dlerror: cupti.dll not found
2021-07-26 19:00:59.097167: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1661] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2021-07-26 19:01:00.126079: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
20

as suggested by @rdslater, renaming cupti64_2021.1.1.dll -> cupti64_112.dll worked for me as well.

Tf 2.5 GPU
Windows 10
CUDA 11.3

[fiora0]

TF 2.5 GPU
CUDA v11.4
Windows 10
Cuda compilation tools V11.4.48

I renamed cupti64_2021.2.0.dll into cupti64_112.dll
and do not have the error:
2021-08-06 10:30:20.552092: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cupti64_112.dll'; dlerror: cupti64_112.dll not found

Yet, I still have the following:
cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2021-08-06 10:41:41.747407: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session tear down.
2021-08-06 10:41:41.747421: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1752] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.

[thunguyenth]

My settings:

• Wins 10
• TF 2.5
• CUDA v11.1
• GPU GTX 3090

I have tried several methods but it still did not work. Then, I tried using the "cudatoolkit" and "cudnn" from conda and it works.
You can try by opening the terminal of your virtual environment and type:
conda install cudatoolkit
conda install cudnn

I know that it is not the absolute solution, but hope it works for you

[fiora0]

@thunguyenth
thanks for your answer but only problems with CUPTI not loading some libraries for tensorboard.

In any case, I solved it: the path to C:\Program Files\NVIDIA Corporation\Nsight Systems 2021.2.4\target-windows-x64
was not set while there was a path to
C:\Program Files\NVIDIA Corporation\Nsight Compute 2021.2.0
So I added the path to
C:\Program Files\NVIDIA Corporation\Nsight Systems 2021.2.4\target-windows-x64

[rjtp5670]

Hi, I had the exact same issue as @zainulabidin302 suffered from the cupti dll issue.

My configuration

- Tensor 2.6
- CUDA 11.4
- Nvidia Geforce 1050 (not ti)
- Windows 10 
- Python 3.8.8 (Setup with Anaconda)
- Visual Studio Code

Here is what helps me,

1. Add path C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\extras\CUPTI\lib64 to the System path as below

• 

2. Rename cupti65_2022_xx.dll to cupti64_112.dll

3. Make sure to reopen your code editor. and build your code to see if there is any error occured.

[zainulabidin302]

I can confirm @rjtp5670 's advice works. Thanks for the help.

[IchiruTake]

at was caught very late in the 2.4 release process. So for TF 2.4 you'll have to use a workaround -- copy cupti64_2020.1.1.dll to cupti64_110.dll to use the profiler on Windows. We'll

I have used TF 2.5 but it did not fix yet

[gadagashwini]

@IchiruTake,
Renaming cupti64_2020.1.1.dll -> cupti64_112.dll should solve your proble. Thanks!

[google-ml-butler]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No