Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR

[houwenbo87]

Hi, shicai

I want to use 'caffe time' evaluate network computing time. But encountered the same problem on different GPU.
Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR.

Do you know how to solve this problem, and have you evaluated the network performance on different GPU?

Thank you!

[shicai]

please uncomment engine: CAFFE used in the conv layers with group.
do not use default CUDNN engine.

[Zehaos]

I benchmark the inference time on my laptop in CPU mode, time cost almost double as tf. It seems that caffe's implementation(depth wise conv) is not so efficiency.(while backward is faster than tf)

[shicai]

caffe uses group (it is actually a for-loop) to implement channel-wise conv, while tf uses a specialized implementation.

[siddharthm83]

Do I need to recompile caffe without cudnn for inference?

[shicai]

@siddharthm83 no.

[qingzew]

@shicai why mobilenet can not run with cudnn, do you have any idea? in my case, the used memory is increasing, then out of memory,

[wjxiz1992]

@qingzew this may be a bug for cudnn, not caffe? I'm not sure, but in caffe, if you use engine:CAFFE and gpu mode, mobileNet is not "mobile" any more... I've seen some implementation of depthwise conv on github, you can search "depthwise" to check them.

[qingzew]

@wjxiz1992 thank you

[allanpk716]

@shicai how can i not use cudnn when my caffe compile with cudnn ?

[shicai]

please refer to readme.md

[hana9090]

@shicai
I run faster rcnn and I have this error.
where should I uncomment this engine؟ which file?

[gargvikram07]

where should I uncomment this engine؟ which file?

[hana9090]

@gargvikram07 the error disappears when I use sudo before run the python file. When I search about the error many people suggest that you don't have admin privilege to use engine.

[lishaofeng]

@hana9090 it's work for me when i use sudo ./tools/demo.py

[tolotrasamuel]

@shicai Could you please explain where should I uncomment this engine? which file?

[shicai]

mobilenet_deploy.prototxt
just search engine in the file.

[shenyingying]

@shicai
good job,thanks for your share,but i have another question,when i use cuda10.01 RTX2080 cudnn7.5 caffe run forward,it throw the question still

I0731 16:55:33.251256 31878 layer_factory.hpp:77] Creating layer inception_3a/pool
I0731 16:55:33.251263 31878 net.cpp:120] Creating Layer inception_3a/pool
I0731 16:55:33.251266 31878 net.cpp:442] inception_3a/pool <- pool2/3x3_s2_pool2/3x3_s2_0_split_3
I0731 16:55:33.251271 31878 net.cpp:416] inception_3a/pool -> inception_3a/pool
F0731 16:55:33.252310 31878 cudnn_pooling_layer.cpp:12] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR

but i can't find the engine in model.prototxt.
thanks for your replay

[yuxwind]

Actually, I was out of GPU memory. After killing some application, the error is fixed.