Image detection fails with CUDNN=1

[hrishikeshm13]

I am trying to run darknet on Jetson TX2.
When I set CUDNN flag to 1 in the Makefile, darknet binary is failing to make detections in the images that I pass. If CUDNN flag is set to 0, the image detection work fine, but the time taken per image detection is ~0.5 - 1.5 secs.
I have tried with a bunch of images, however the observations are same in each case. Also I tried using tiny-yolo, yolo-voc weights files.

[ahsan856jalal]

cuDNN supports cc3.0 and higher. What version of CUDA do you use? cuDNN requires CUDA 6.5 or higher. Check if cuDNN is properly installed.

[hrishikeshm13]

Thanks @ahsan856jalal , I am using CUDA 8.0. I tried re-installing cuDNN as well without much success on the detection front.

[ahsan856jalal]

one more thing , please check the CudNN version requirement for cuda 8.0

[hrishikeshm13]

cudNN version is 7.0.1

[ahsan856jalal]

I have cudNN v6 installed with cuda v8.0.61 and it is working fine with detection

[hrishikeshm13]

@ahsan856jalal Just to confirm, you are using jetson TX2 for the detections ?

[ahsan856jalal]

No I am using Titan X but this problem is cudNN specific

[TheMikeyR]

@hrishikeshm13 remember to pick the right cuDNN version for your CUDA version.
https://developer.nvidia.com/rdp/cudnn-download

For cuDNN 7.0.1 there are 3 version available each for a variant of CUDA (8.0, 9.0 and 9.1), did you make sure to download the right version compatible with your CUDA version ?

[AlexeyAB]

@hrishikeshm13

• Do you use this repo or original repo?

• Try to use Download cuDNN v7.1.1 (Feb 28, 2018), for CUDA 8.0 https://developer.nvidia.com/cudnn

• Try to comment these lines:

  darknet/Makefile

  Lines 8 to 12 in a6c51e3

  	ARCH= -gencode arch=compute_30,code=sm_30 \
  	-gencode arch=compute_35,code=sm_35 \
  	-gencode arch=compute_50,code=[sm_50,compute_50] \
  	-gencode arch=compute_52,code=[sm_52,compute_52] \
  	-gencode arch=compute_61,code=[sm_61,compute_61]

and un-comment this line:

darknet/Makefile

Line 27 in a6c51e3

# ARCH= -gencode arch=compute_62,code=[sm_62,compute_62]

[hrishikeshm13]

Hi @AlexeyAB

Do you use this repo or original repo?
This repo

Try to comment these lines
Tried it. No change in observations

[AlexeyAB]

@hrishikeshm13

This is strange. Some people succeed in using TX2 with cuDNN.

#406 (comment)

As of now, I am using Tiny YOLO as my framework through Darknet, compiled with GPU and CUDNN support. The images are 640x360 in dimensions and I have about 100000 of them with around 10 classes of objects in total. I've trained tiny YOLO for about 80000 iterations and on an average, this has given me IoUs of around 50% on the test dataset with a performance of around 18 fps on the Jetson TX2

[TheMikeyR]

@hrishikeshm13 Did you run jetson-clocks.sh from the home directory, as well as the command sudo nvpmodel -m 0 for maximum performance?

[TheMikeyR]

@hrishikeshm13 what command are you running? Then I can test on a jetson as well, but I have no issues compiling with CUDNN=1 flag.

[hrishikeshm13]

@AlexeyAB Yes. After seeing those comments, I decided to give cudNN a try :)

@TheMikeyR The command I run is -
./darknet detector test ./cfg/coco.data ./cfg/yolo.cfg ./yolo.weights

The command runs OK. However it does not generate any predictions.

[TheMikeyR]

Try to run the image-voc.sh in root directory of alexeyab's fork.

I get these results, if you don't have weights file download it from here

Done!
data/dog.jpg: Predicted in 0.146533 seconds.
car: 76%
bicycle: 79%
dog: 90%
Not compiled with OpenCV, saving to predictions.png instead

See if your results match mine.

[hrishikeshm13]

@TheMikeyR Does not seem to work at my end. I am using cuda 8.0.84 and cudnn 7.0.1

[TheMikeyR]

@hrishikeshm13 I did some investigations and figured out some stuff about jetson, I will share my findings and hope it will help you!

If you have installed cudnn in the "normal" way according to the documentation by doing these commands

$ tar -xzvf cudnn-9.0-linux-x64-v7.tgz
$ sudo cp cuda/include/cudnn.h /usr/local/cuda/include
$ sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
$ sudo chmod a+r /usr/local/cuda/include/cudnn.h

You will have to undo that by doing

$ sudo rm /usr/local/cuda/include/cudnn.h
$ sudo rm /usr/local/cuda/lib64/libcudnn*

Try to do make clean && make in the darknet repo and see if you can image_voc.sh and get detections, it worked for me. If not then you need to either reflash your device with new jetpack or do the instructions as listed in this thread

The jetson expects its cuDNN library to be located at /usr/lib/aarch64-linux-gnu/ and you can make a $ ls /usr/lib/aarch64-linux-gnu/libcudnn* and see if you have your cudnn library already installed. If its not they are missing and you should do the installation from the thread above. The cudnn library should not be located at the default spot in /usr/local/cuda/ I guess it's due to the arm architecture, but honestly I don't know.

[hrishikeshm13]

@TheMikeyR Thanks Mike. That helps ! Getting correct detections with CUDNN=1