nd4j-cuda-10.1-platform failure due to jnicusparse.dll issue #7781

zollen · 2019-05-24T00:29:50Z

I setup nd4j-cuda-10.1-platform in my pom.xml but when I run my program. It threw an exception with following root cause:

Caused by: java.lang.UnsatisfiedLinkError: C:\Users\zollen.javacpp\cache\cuda-10.1-7.5-1.5-windows-x86_64.jar\org\bytedeco\cuda\windows-x86_64\jnicusparse.dll: The specified procedure could not be found

System Information

<dependency>
	<groupId>org.deeplearning4j</groupId>
	<artifactId>deeplearning4j-core</artifactId>
	<version>1.0.0-beta4</version>
</dependency>
 <dependency>
          <groupId>org.nd4j</groupId>
          <artifactId>nd4j-cuda-10.1-platform</artifactId>
           <version>1.0.0-beta4</version>
</dependency>

C:\Users\zollen>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:11:20_Pacific_Daylight_Time_2019
Cuda compilation tools, release 10.1, V10.1.168

C:\Users\zollen>systeminfo | findstr /B /C:"OS Name" /C:"OS Version"
OS Name:                   Microsoft Windows 10 Home
OS Version:                10.0.17763 N/A Build 17763

Nvidia GeForce GTX 1070
Driver Version: 430.64

The text was updated successfully, but these errors were encountered:

AlexDBlack · 2019-05-24T01:38:35Z

Do you have cuDNN set up also (i.e., cudnn DLL on the system PATH environment variable)? IIRC I've seen "The specified procedure could not be found" before with wrong cuDNN version, though you'd probably need deeplearning4j-cuda-10.1 dependency also for that to be a cause of this...

zollen · 2019-05-24T02:08:29Z

May I ask what are the deeplearning4j-cuda-10.1 dependencies?

AlexDBlack · 2019-05-24T02:21:10Z

May I ask what are the deeplearning4j-cuda-10.1 dependencies?

https://deeplearning4j.org/docs/latest/deeplearning4j-config-cudnn

So literally,

<dependency>
	<groupId>org.deeplearning4j</groupId>
	<artifactId>deeplearning4j-cuda-10.1</artifactId>
	<version>1.0.0-beta4</version>
</dependency>

To confirm - you don't also have that in your pom.xml? (If not, that's fine, we can rule out one possible cause).

zollen · 2019-05-24T02:22:45Z

I don't have the following in my pom.xml.

<dependency>
	<groupId>org.deeplearning4j</groupId>
	<artifactId>deeplearning4j-cuda-10.1</artifactId>
	<version>1.0.0-beta4</version>
</dependency>

zollen · 2019-05-24T02:32:00Z

This is my latest pom.xml and I have also installed Nvidia Cudnn, but I am still getting the same error.

<dependency>
	<groupId>org.deeplearning4j</groupId>
	<artifactId>deeplearning4j-core</artifactId>
	<version>1.0.0-beta4</version>
</dependency>
 <dependency>
          <groupId>org.nd4j</groupId>
          <artifactId>nd4j-cuda-10.1-platform</artifactId>
           <version>1.0.0-beta4</version>
</dependency>
<dependency>
	<groupId>org.deeplearning4j</groupId>
	<artifactId>deeplearning4j-cuda-10.1</artifactId>
	<version>1.0.0-beta4</version>
</dependency>

raver119 · 2019-05-24T02:53:51Z

Show nvidia-smi output please.

Aelentel · 2019-05-24T09:04:01Z

Do you have cuDNN set up also (i.e., cudnn DLL on the system PATH environment variable)? IIRC I've seen "The specified procedure could not be found" before with wrong cuDNN version, though you'd probably need deeplearning4j-cuda-10.1 dependency also for that to be a cause of this...

is cudnn a mandatory library ? (just because the documentation doesn't specify it at all in the set-up phase)

raver119 · 2019-05-24T09:05:19Z

No, cuDNN is not mandatory. Just recommended if you're working with convolutional/recurrent stuff.

Aelentel · 2019-05-24T09:24:52Z

No, cuDNN is not mandatory. Just recommended if you're working with convolutional/recurrent stuff.

thanks, but i've exactly the same problem, even after installing CuDNN (i checked the version against Cuda 10.1 to be sure)
i'm working on Word2Vec and TSNE to reduce dimensionality, the final goal is to train a text categorisation model.

i'll put all the information i can collect, please if something is wrong, doesn't hesitate to bash me :)

here's the actual stacktace on my side when doing Word2Vec with Cuda :

Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5900)
	at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5766)
	at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:202)
	... 4 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5860)
	... 6 more
Caused by: java.lang.UnsatisfiedLinkError: no jnicusparse in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1857)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1119)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1336)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1077)
	at org.bytedeco.javacpp.Loader.load(Loader.java:947)
	at org.bytedeco.cuda.global.cusparse.<clinit>(cusparse.java:15)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:340)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1006)
	at org.bytedeco.javacpp.Loader.load(Loader.java:947)
	at org.bytedeco.cuda.global.cusolver.<clinit>(cusolver.java:19)
	at org.nd4j.jita.allocator.context.impl.BasicContextPool.createNewSolverHandle(BasicContextPool.java:220)
	at org.nd4j.jita.allocator.context.impl.LimitedContextPool.fillPoolWithResources(LimitedContextPool.java:115)
	at org.nd4j.jita.allocator.context.impl.LimitedContextPool.<init>(LimitedContextPool.java:80)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.<init>(CudaZeroHandler.java:145)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.<init>(AtomicAllocator.java:145)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.<clinit>(AtomicAllocator.java:93)
	at org.nd4j.linalg.jcublas.JCublasNDArrayFactory.<init>(JCublasNDArrayFactory.java:85)
	... 11 more
Caused by: java.lang.UnsatisfiedLinkError: C:\Users\Sandro\.javacpp\cache\cuda-10.1-7.5-1.5-windows-x86_64.jar\org\bytedeco\cuda\windows-x86_64\jnicusparse.dll: The specified procedure could not be found
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1929)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1814)
	at java.lang.Runtime.load0(Runtime.java:809)
	at java.lang.System.load(System.java:1083)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1316)
	... 26 more

my POM deps and props :

<dl4j.version>1.0.0-beta4</dl4j.version>

		<dependency>
			<groupId>org.deeplearning4j</groupId>
			<artifactId>deeplearning4j-ui_2.11</artifactId>
			<version>${dl4j.version}</version>
		</dependency>
		<dependency>
			<groupId>org.deeplearning4j</groupId>
			<artifactId>deeplearning4j-modelimport</artifactId>
			<version>${dl4j.version}</version>
		</dependency>
		<dependency>
			<groupId>org.deeplearning4j</groupId>
			<artifactId>deeplearning4j-nn</artifactId>
			<version>${dl4j.version}</version>
		</dependency>
		<dependency>
			<groupId>org.deeplearning4j</groupId>
			<artifactId>deeplearning4j-cuda-10.1</artifactId>
			<version>${dl4j.version}</version>
		</dependency>
		<dependency>
			<groupId>org.nd4j</groupId>
			<artifactId>nd4j-cuda-10.0-platform</artifactId>
			<version>${dl4j.version}</version>
		</dependency>
		<dependency>
			<groupId>org.nd4j</groupId>
			<artifactId>nd4j-native-platform</artifactId>
			<version>${dl4j.version}</version>
		</dependency>

my nvcc and device query .

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:11:20_Pacific_Daylight_Time_2019
Cuda compilation tools, release 10.1, V10.1.168

deviceQuery.exe Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 970"
  CUDA Driver Version / Runtime Version          10.1 / 10.1
  CUDA Capability Major/Minor version number:    5.2
  Total amount of global memory:                 4096 MBytes (4294967296 bytes)
  (13) Multiprocessors, (128) CUDA Cores/MP:     1664 CUDA Cores
  GPU Max Clock rate:                            1329 MHz (1.33 GHz)
  Memory Clock rate:                             3505 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 1835008 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               zu bytes
  Total amount of shared memory per block:       zu bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          zu bytes
  Texture alignment:                             zu bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  CUDA Device Driver Mode (TCC or WDDM):         WDDM (Windows Display Driver Model)
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            No
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
  Device PCI Domain ID / Bus ID / location ID:   0 / 2 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.1, CUDA Runtime Version = 10.1, NumDevs = 1, Device0 = GeForce GTX 970
Result = PASS

zollen · 2019-05-24T09:32:52Z

I did a system search for jnicusparse.dll:

C:\Users\zollen.javacpp\cache\cuda-10.1-7.5-1.5-windows-x86_64.jar\org\bytedeco\cuda\windows-x86_64>dir
Volume in drive C has no label.
Volume Serial Number is 945B-4DD2

Directory of C:\Users\zollen.javacpp\cache\cuda-10.1-7.5-1.5-windows-x86_64.jar\org\bytedeco\cuda\windows-x86_64

2019-05-22 07:50 PM

.
2019-05-22 07:50 PM ..
2019-04-09 09:03 AM 894,464 jnicublas.dll
2019-04-09 09:03 AM 1,948,672 jnicudart.dll
2019-04-09 09:04 AM 1,940,992 jnicusparse.dll
3 File(s) 4,784,128 bytes
2 Dir(s) 160,367,009,792 bytes free

zollen · 2019-05-24T09:36:39Z

Curious.... It displays CUDA Version 10.2...

C:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi
Fri May 24 05:35:58 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.64       Driver Version: 430.64       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070   WDDM  | 00000000:01:00.0  On |                  N/A |
|  0%   35C    P8    10W / 185W |    399MiB /  8192MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1200    C+G   Insufficient Permissions                   N/A      |
|    0      3916    C+G   ...oftEdge_8wekyb3d8bbwe\MicrosoftEdge.exe N/A      |
|    0      6784    C+G   ...)\Microsoft\Skype for Desktop\Skype.exe N/A      |
|    0      6832    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    0      7352    C+G   C:\Windows\explorer.exe                    N/A      |
|    0      8116    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0      8980    C+G   ....481.0_x64__8wekyb3d8bbwe\YourPhone.exe N/A      |
|    0     10452    C+G   ...)\Microsoft\Skype for Desktop\Skype.exe N/A      |
|    0     11044    C+G   ...DIA GeForce Experience\NVIDIA Share.exe N/A      |
|    0     11628    C+G   ...\bin\cef\cef.win7x64\steamwebhelper.exe N/A      |
|    0     12560    C+G   ...6)\Google\Chrome\Application\chrome.exe N/A      |
|    0     14092    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0     15244    C+G   ... Files (x86)\Dropbox\Client\Dropbox.exe N/A      |
|    0     15484    C+G   ...mmersiveControlPanel\SystemSettings.exe N/A      |
|    0     16152    C+G   ...Corsair\Corsair Utility Engine\iCUE.exe N/A      |
|    0     17360    C+G   ...hell.Experiences.TextInput.InputApp.exe N/A      |
+-----------------------------------------------------------------------------+

raver119 · 2019-05-24T09:40:21Z

CUDA Version: 10.2

See this line in your nvidia-smi output? That one looks strange, since latest CUDA is 10.1

Also, you're showing me two different systems here. One is cc 52, other one is cc 61.

zollen · 2019-05-24T09:45:37Z

That makes no senses. If the official Nvidia CUDA latest release 10.1. How did I managed to get a 10.2 CUDA? Is there any difference between CUDA version and CUDA driver version?
What is cc52 and cc61? Where did you spot those?
I have only a single Windows 10. What do you mean two different systems?

raver119 · 2019-05-24T09:47:00Z

2/3:

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 970"
  CUDA Driver Version / Runtime Version          10.1 / 10.1
  CUDA Capability Major/Minor version number:    5.2

and

 GeForce GTX 1070   WDDM

that's two different GPUs, with different compute capabilities.

raver119 · 2019-05-24T09:49:07Z

Well, anyway, the problem looks like binary incompatibility between what we've used to build and what you use. You're probably using CUDA 10.1 Update 1. Please downgrade to regular CUDA 10.1 (there are previous versions available on nvidia website), and everything should work fine.

And dont forget to reboot your box after downgrade.

zollen · 2019-05-24T09:56:47Z

I shall downgrade my CUDA SDK and see how it goes. Thanks for the tips.

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\extras\demo_suite>deviceQuery.exe
deviceQuery.exe Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 1070"
  CUDA Driver Version / Runtime Version          10.2 / 10.1
  CUDA Capability Major/Minor version number:    6.1

saudet · 2019-05-24T11:27:21Z

Yeah, CUDA updates have been getting really strange since 10.1. Just use the bundled binaries as per this pom.xml file here instead of trying to figure out which version of CUDA you need:
https://github.com/bytedeco/javacpp-presets/tree/1.5/cuda#the-pomxml-build-file

zollen · 2019-05-24T22:19:58Z

It works!! Thanks for the quick help!!

Aelentel · 2019-05-25T12:14:31Z

Yeah, CUDA updates have been getting really strange since 10.1. Just use the bundled binaries as per this pom.xml file here instead of trying to figure out which version of CUDA you need:
https://github.com/bytedeco/javacpp-presets/tree/1.5/cuda#the-pomxml-build-file

thanks a bunch ! worked on my side too, this should really go on the official documentation if problem arise !

BourneSuper · 2019-11-14T18:43:30Z

it works. just add few lines to pom.xml

<dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>cuda</artifactId>
    <version>10.1-7.5-1.5</version>
    <classifier>windows-x86_64-redist</classifier>
</dependency>

zollen changed the title ~~CUDA-10.1 failure due to jnicusparse.dll issue~~ nd4j-cuda-10.1-platform failure due to jnicusparse.dll issue May 24, 2019

zollen closed this as completed May 24, 2019

saudet added ND4J ND4J Issues Question labels May 26, 2019

saudet mentioned this issue Jun 18, 2019

windows CUDA org.bytedeco.javacpp.Loader.loadLibrary Error #7909

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nd4j-cuda-10.1-platform failure due to jnicusparse.dll issue #7781

nd4j-cuda-10.1-platform failure due to jnicusparse.dll issue #7781

zollen commented May 24, 2019

AlexDBlack commented May 24, 2019

zollen commented May 24, 2019

AlexDBlack commented May 24, 2019

zollen commented May 24, 2019

zollen commented May 24, 2019 •

edited

raver119 commented May 24, 2019

Aelentel commented May 24, 2019

raver119 commented May 24, 2019

Aelentel commented May 24, 2019

zollen commented May 24, 2019 •

edited

zollen commented May 24, 2019 •

edited

raver119 commented May 24, 2019

zollen commented May 24, 2019 •

edited

raver119 commented May 24, 2019

raver119 commented May 24, 2019

zollen commented May 24, 2019

saudet commented May 24, 2019

zollen commented May 24, 2019

Aelentel commented May 25, 2019

BourneSuper commented Nov 14, 2019

nd4j-cuda-10.1-platform failure due to jnicusparse.dll issue #7781

nd4j-cuda-10.1-platform failure due to jnicusparse.dll issue #7781

Comments

zollen commented May 24, 2019

AlexDBlack commented May 24, 2019

zollen commented May 24, 2019

AlexDBlack commented May 24, 2019

zollen commented May 24, 2019

zollen commented May 24, 2019 • edited

raver119 commented May 24, 2019

Aelentel commented May 24, 2019

raver119 commented May 24, 2019

Aelentel commented May 24, 2019

zollen commented May 24, 2019 • edited

zollen commented May 24, 2019 • edited

raver119 commented May 24, 2019

zollen commented May 24, 2019 • edited

raver119 commented May 24, 2019

raver119 commented May 24, 2019

zollen commented May 24, 2019

saudet commented May 24, 2019

zollen commented May 24, 2019

Aelentel commented May 25, 2019

BourneSuper commented Nov 14, 2019

zollen commented May 24, 2019 •

edited

zollen commented May 24, 2019 •

edited

zollen commented May 24, 2019 •

edited

zollen commented May 24, 2019 •

edited