Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CUDA 10.1 on Windows 10 shows error: pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n' theano #587

Open
anupverma20 opened this issue May 14, 2019 · 11 comments
Open

CUDA 10.1 on Windows 10 shows error: pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n' theano #587

anupverma20 opened this issue May 14, 2019 · 11 comments

Comments

@anupverma20
Copy link

anupverma20 commented May 14, 2019 •
edited
Loading

OS: Windows 10
CUDA: 10.1

 Name                     Version                           Build  Channel
python                     3.5.6                            he025d50_0
libgpuarray                0.7.6                            vc14_0  [vc14]  mila-udem
pygpu                      0.7.6            	            py35h452e1ab_0
theano                     1.0.2            		    py35he980bc4_0
keras                      2.2.2                            0

Content of .theanorc.txt file:
[global]
device = cuda
floatX = float32

In System variables, the following are set:
CUDA_PATH
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1

CUDA_PATH_V10_1
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1

DEVICE
cuda

In the Path Variable of System variables, the following are set:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin

Error:

(gpuml) C:\Users\Anup\Desktop\zdata\wiki_crop_out>python -c "from keras import backend; print(backend._BACKEND")
Using Theano backend.
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 227, in <module>
    use(config.device)
  File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 214, in use
    init_dev(device, preallocate=preallocate)
  File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray\__init__.py", line 99, in init_dev
    **args)
  File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
  File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n'
theano

Similar issue was raised but closed without any solution. If CUDA 10.1 is not supported currently, when will it be supported?

Thanks!
@nouiz
Copy link
Member

nouiz commented May 15, 2019 via email

@anupverma20
Copy link
Author

It was fixed upstream, but no new release was done. So you must compile libgpuarray yourself.
…
On Tue, May 14, 2019 at 3:00 PM Anup Verma @.***> wrote: OS: Windows 10 CUDA: 10.1 Name Version Build Channel python 3.5.6 he025d50_0 libgpuarray 0.7.6 vc14_0 [vc14] mila-udem pygpu 0.7.6 py35h452e1ab_0 theano 1.0.2 py35he980bc4_0 keras 2.2.2 0 Content of .theanorc.txt file: [global] device = cuda floatX = float32 In System variables, the following are set: CUDA_PATH C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 CUDA_PATH_V10_1 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 GPUARRAY_TEST_DEVICE cuda In the Path Variable of System variables, the following are set: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin Error: (gpuml) C:\Users\Anup\Desktop\zdata\wiki_crop_out>python -c "from keras import backend; print(backend.BACKEND") Using Theano backend. ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray_init.py", line 227, in use(config.device) File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray_init_.py", line 214, in use init_dev(device, preallocate=preallocate) File "C:\Users\Anup\Anaconda3\envs\gpuml\lib\site-packages\theano\gpuarray_init_.py", line 99, in init_dev **args) File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": The specified module could not be found.\r\n' theano Similar issue was raised but closed without any solution. If CUDA 10.1 is not supported currently, when will it be supported? Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#587?email_source=notifications&email_token=AABMF65F4OFHAHWYO5QTOEDPVMD5VA5CNFSM4HM4YTW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GTYBGHA>, or mute the thread https://github.com/notifications/unsubscribe-auth/AABMF64MUDG3BCDXM6LPBP3PVMD5VANCNFSM4HM4YTWQ .

Thanks for the reply.
I made libgpuarray.sln using cmake in Anaconda.:
(gpuml) C:\Users\Anup\Desktop\lib\libgpuarray\build>cmake .. -DCMAKE_BUILD_TYPE=Release -G "Visual Studio 16 2019" -A X64"

I opened libgpuarray.sln in Visual Studio.
When I select Build > gpuarray, I get the following error:

1>------ Build started: Project: gpuarray, Configuration: Release X64 ------
1>Generating ../../src/gpuarray_types.c, ../../src/gpuarray/types.h
1>'python' is not recognized as an internal or external command,
1>operable program or batch file.
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe" exited with code 9009.
1>Done building project "gpuarray.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

I could not fix the python-related error
I am new to cmake, Visual Studio and dll. Please help!

@NellTrico
Copy link

Hello! I experience something similar.
I have Windows 10, CUDA 10.1, and tried to install from the main branch and got the following error:

File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_102.dll": The specified module could not be found.\r\n'

I checked which version of nvrtc I have installed and it turns out I have nvrtc64_101_0.dll.

Could I please get some help with that?

@anupverma20
Copy link
Author

I installed python, added its path in Environment Variables > Path. Now, that error is fixed.
I also installed Mako, restarted the computer. But, I get the following error while building the sln file in Visual Studio:

1>------ Build started: Project: gpuarray, Configuration: Release X64 ------
1>Generating ../../src/gpuarray_types.c, ../../src/gpuarray/types.h
1>Traceback (most recent call last):
1>  File "gen_types.py", line 3, in <module>
1>    from mako import exceptions
1>ModuleNotFoundError: No module named 'mako'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe" exited with code 1.
1>Done building project "gpuarray.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

I was able to build gpuarray.dll in my office laptop. But, in the personal laptop, I am getting the above error. Trying to fix it now.

@anupverma20
Copy link
Author

Hello! I experience something similar.
I have Windows 10, CUDA 10.1, and tried to install from the main branch and got the following error:

File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_102.dll": The specified module could not be found.\r\n'

I checked which version of nvrtc I have installed and it turns out I have nvrtc64_101_0.dll.

Could I please get some help with that?

As far as I have learnt from answers to similar issues on the internet, there are 2 options:

  1. The easier option is to install CUDA 9.
  2. The tougher option is to build libgpuarray yourself as pointed out by @nouiz above.
    Refer the answer of cknave at https://www.bountysource.com/issues/68975408-theano-1-0-3-issue-with-pygpu-error-theano-gpuarray-could-not-initialize-pygpu-support-disabled to get a hint of the process.

@NellTrico
Copy link

NellTrico commented May 20, 2019 •
edited
Loading

Hello! I experience something similar.
I have Windows 10, CUDA 10.1, and tried to install from the main branch and got the following error:
File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_102.dll": The specified module could not be found.\r\n'
I checked which version of nvrtc I have installed and it turns out I have nvrtc64_101_0.dll.
Could I please get some help with that?

As far as I have learnt from answers to similar issues on the internet, there are 2 options:

1. The easier option is to install CUDA 9.

2. The tougher option is to build libgpuarray yourself as pointed out by @nouiz above.
   Refer the answer of cknave at https://www.bountysource.com/issues/68975408-theano-1-0-3-issue-with-pygpu-error-theano-gpuarray-could-not-initialize-pygpu-support-disabled to get a hint of the process.

Thanks for your reply @anupverma20 !
I tried both options without success. (Note that I don't use conda so I was already building libgpuarray myself)

I am confused as to why the library insists on pointing to nvrtc64_102.dll, even after I uninstalled CUDA 10 and kept 9. I looked at the code in gpuarray_buffer_cuda.c and I am at loss to understand why and how I can even obtain major=10, minor=2... Nevertheless I tried changing major and minor in the source code to 9 and 1 respectively (having installed CUDA 9.1 with the corresponding nvrtc64_91.dll), but I was then faced with

Could not load "nvrtc64_91.dll": %1 is not a valid Win32 application.\r\n'

Similarly, when I still had CUDA 10.1, I tried renaming the dll to the expected "nvrtc64_102.dll", and had a similar error message.

EDIT: It seems I get this error because I am using Python 36-32. I thought a compatible 32/64 bit version of Python would let me do it though? Am I wrong? If yes, do I have any option, like Running GPU acceleration with OpenCL for instance, which I saw in the tutorials, but have not given time yet, or is it limited for 64 bit architecture only?

@kumarnikhil936
Copy link

Any solutions so far?
Still facing this issue.

@ThoenigAdrian
Copy link

same issue here with cuda 11

@nouiz
Copy link
Member

nouiz commented Oct 20, 2020

This MR is the last one that update the supported CUDA version: https://github.com/Theano/libgpuarray/pull/580/files
So the most recent CUDA version supported is 10.0

This lib isn't maintained. But maybe you can find now to support newer version with that url.

@ThoenigAdrian
Copy link

@nouiz
Apparently there is an up to date fork : https://github.com/pymc-devs/Theano-PyMC . Haven't tried it though.
Afaik it should be possible to install multiple CUDA versions on a machine. So one could install CUDA 9.0 and use Theano with it.

@nouiz
Copy link
Member

nouiz commented Oct 23, 2020

This is a Theano fork. I do not know if they forked libgpuarray too. Maybe you can ask them?
I would recommend to try CUDA 10.0 first and not 9.0 as from the quick look at the code ,10.0 was supported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@nouiz @anupverma20 @ThoenigAdrian @NellTrico @kumarnikhil936