Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Image detection fails with CUDNN=1 #436

Closed
hrishikeshm13 opened this issue Mar 9, 2018 · 18 comments
Closed

Image detection fails with CUDNN=1 #436

hrishikeshm13 opened this issue Mar 9, 2018 · 18 comments
Labels

Comments

@hrishikeshm13
Copy link

I am trying to run darknet on Jetson TX2.
When I set CUDNN flag to 1 in the Makefile, darknet binary is failing to make detections in the images that I pass. If CUDNN flag is set to 0, the image detection work fine, but the time taken per image detection is ~0.5 - 1.5 secs.
I have tried with a bunch of images, however the observations are same in each case. Also I tried using tiny-yolo, yolo-voc weights files.

@ahsan856jalal
Copy link

cuDNN supports cc3.0 and higher. What version of CUDA do you use? cuDNN requires CUDA 6.5 or higher. Check if cuDNN is properly installed.

@hrishikeshm13
Copy link
Author

Thanks @ahsan856jalal , I am using CUDA 8.0. I tried re-installing cuDNN as well without much success on the detection front.

@ahsan856jalal
Copy link

one more thing , please check the CudNN version requirement for cuda 8.0

@hrishikeshm13
Copy link
Author

cudNN version is 7.0.1

@ahsan856jalal
Copy link

I have cudNN v6 installed with cuda v8.0.61 and it is working fine with detection

@hrishikeshm13
Copy link
Author

@ahsan856jalal Just to confirm, you are using jetson TX2 for the detections ?

@ahsan856jalal
Copy link

No I am using Titan X but this problem is cudNN specific

@TheMikeyR
Copy link

@hrishikeshm13 remember to pick the right cuDNN version for your CUDA version.
https://developer.nvidia.com/rdp/cudnn-download

For cuDNN 7.0.1 there are 3 version available each for a variant of CUDA (8.0, 9.0 and 9.1), did you make sure to download the right version compatible with your CUDA version ?

@AlexeyAB
Copy link
Owner

AlexeyAB commented Mar 9, 2018

@hrishikeshm13

  • Do you use this repo or original repo?

  • Try to use Download cuDNN v7.1.1 (Feb 28, 2018), for CUDA 8.0 https://developer.nvidia.com/cudnn

  • Try to comment these lines:

    darknet/Makefile

    Lines 8 to 12 in a6c51e3

    ARCH= -gencode arch=compute_30,code=sm_30 \
    -gencode arch=compute_35,code=sm_35 \
    -gencode arch=compute_50,code=[sm_50,compute_50] \
    -gencode arch=compute_52,code=[sm_52,compute_52] \
    -gencode arch=compute_61,code=[sm_61,compute_61]

and un-comment this line:

# ARCH= -gencode arch=compute_62,code=[sm_62,compute_62]

@hrishikeshm13
Copy link
Author

Hi @AlexeyAB

Do you use this repo or original repo?
This repo

Try to comment these lines
Tried it. No change in observations

@AlexeyAB
Copy link
Owner

AlexeyAB commented Mar 9, 2018

@hrishikeshm13

This is strange. Some people succeed in using TX2 with cuDNN.

#406 (comment)

As of now, I am using Tiny YOLO as my framework through Darknet, compiled with GPU and CUDNN support. The images are 640x360 in dimensions and I have about 100000 of them with around 10 classes of objects in total. I've trained tiny YOLO for about 80000 iterations and on an average, this has given me IoUs of around 50% on the test dataset with a performance of around 18 fps on the Jetson TX2

@TheMikeyR
Copy link

@hrishikeshm13 Did you run jetson-clocks.sh from the home directory, as well as the command sudo nvpmodel -m 0 for maximum performance?

@TheMikeyR
Copy link

@hrishikeshm13 what command are you running? Then I can test on a jetson as well, but I have no issues compiling with CUDNN=1 flag.

@hrishikeshm13
Copy link
Author

@AlexeyAB Yes. After seeing those comments, I decided to give cudNN a try :)

@TheMikeyR The command I run is -
./darknet detector test ./cfg/coco.data ./cfg/yolo.cfg ./yolo.weights

The command runs OK. However it does not generate any predictions.

@TheMikeyR
Copy link

Try to run the image-voc.sh in root directory of alexeyab's fork.

I get these results, if you don't have weights file download it from here

Done!
data/dog.jpg: Predicted in 0.146533 seconds.
car: 76%
bicycle: 79%
dog: 90%
Not compiled with OpenCV, saving to predictions.png instead

See if your results match mine.

@hrishikeshm13
Copy link
Author

@TheMikeyR Does not seem to work at my end. I am using cuda 8.0.84 and cudnn 7.0.1

@TheMikeyR
Copy link

@hrishikeshm13 I did some investigations and figured out some stuff about jetson, I will share my findings and hope it will help you!

If you have installed cudnn in the "normal" way according to the documentation by doing these commands

$ tar -xzvf cudnn-9.0-linux-x64-v7.tgz
$ sudo cp cuda/include/cudnn.h /usr/local/cuda/include
$ sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
$ sudo chmod a+r /usr/local/cuda/include/cudnn.h

You will have to undo that by doing

$ sudo rm /usr/local/cuda/include/cudnn.h
$ sudo rm /usr/local/cuda/lib64/libcudnn*

Try to do make clean && make in the darknet repo and see if you can image_voc.sh and get detections, it worked for me. If not then you need to either reflash your device with new jetpack or do the instructions as listed in this thread

The jetson expects its cuDNN library to be located at /usr/lib/aarch64-linux-gnu/ and you can make a $ ls /usr/lib/aarch64-linux-gnu/libcudnn* and see if you have your cudnn library already installed. If its not they are missing and you should do the installation from the thread above. The cudnn library should not be located at the default spot in /usr/local/cuda/ I guess it's due to the arm architecture, but honestly I don't know.

@hrishikeshm13
Copy link
Author

@TheMikeyR Thanks Mike. That helps ! Getting correct detections with CUDNN=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants