error == cudaSuccess (11 vs. 0) invalid argument

[songjun54cm]

running rcnn_demo, error occurred when f = caffe('forward', batches(j));
the full message is :

F1205 14:31:18.047643 17619 relu_layer.cu:27] Check failed: error == cudaSuccess (11 vs. 0) invalid argument

My GPU is Tesla K20 the compute capability is 3.5.
And Caffe have been installed with all the caffe test successfully passed...
the version of caffe is the latest caffe release(the release candidate for Caffe 1.0) with head 737ea5e

don't know what is wrong....
need help.....

[dreadlord1984]

Have you solved this problem? Or what's the reason of this situation?
I meet the same error in fintuning the trained model just as describe here . http://caffe.berkeleyvision.org/gathered/examples/finetune_flickr_style.html.

I used the caffe version 0.99. And run successfully in other training task, just failed finetuning the trained model with different classes output.
My error:

math_functions.cpp:90] Check failed: error == cudaSuccess (11 vs. 0) invalid argument
*** Check failure stack trace: ***
@ 0x7fca72b8edaa (unknown)
@ 0x7fca72b8ece4 (unknown)
@ 0x7fca72b8e6e6 (unknown)
@ 0x7fca72b91687 (unknown)
@ 0x4b6dec caffe::caffe_copy<>()
@ 0x49ff4f caffe::SGDSolver<>::ComputeUpdateValue()
@ 0x49b6d3 caffe::Solver<>::Solve()
@ 0x41046e caffe::Solver<>::Solve()
@ 0x40de4d train()
@ 0x40f508 main
@ 0x7fca6ff3bec5 (unknown)
@ 0x40d6e9 (unknown)
Aborted (core dumped)

[songjun54cm]

Finally, I make it run by changing the batch size from 256 to 30.
to do this you need to change the value 'batch' in 'rcnn_model.mat' and the 'input_dim' in prototxt file.
some similar discussion can be found at #10

[zhenni]

I added the following lines to commands using multiple “arch” flags in Nvidia's NVCC compiler, and the error does not occur anymore.

-gencode=arch=compute_20,code=\"sm_20,compute_20\" -gencode=arch=compute_30,code=\"sm_30,compute_30\" -gencode=arch=compute_35,code=\"sm_35,compute_35\" -gencode=arch=compute_50,code=\"sm_50,compute_50\" 

[zachluo]

@zhenni Thanks! It solved my problem.

[wanfangla]

when I do "make runtest -j8" , I meet the same problem
math_functions.cpp:91 Check failed: error == cudaSuccess ( 11 vs. 0) invalid argument
......
make: *** [runtest](core dumped)
anyone solved this problem, I need some help

[zachluo]

@wanfangla please refer to the solution of @zhenni , it solved my problem the same with yours

[aimreant]

well, I've met the same problem with yours..
However, I dont understand how to do
where should I add these flags:

-gencode=arch=compute_20,code="sm_20,compute_20" -gencode=arch=compute_30,code="sm_30,compute_30" -gencode=arch=compute_35,code="sm_35,compute_35" -gencode=arch=compute_50,code="sm_50,compute_50"

not actually understand..
@zachluo @zhenni need some help, plz..

[leejiajun]

@zhenni @zachluo I still get this error.

CUDA_ARCH := -gencode=arch=compute_20,code=sm_20
-gencode=arch=compute_30,code=sm_30
-gencode=arch=compute_35,code=sm_35
-gencode=arch=compute_50,code=sm_50

[IdiosyncraticDragon]

@zhenni I met this problem too, and your advice was workable for me! Thank you!

[zhenni]

@aimreant @leejiajun @wanfangla
Although the response is a little late....it seems there still some people will have the similar problem.

I think this error may be caused by the difference between the compiled cuda-arch and your cud-arch of your gpu. Please check your architecture of your gpu and modify the corresponding lines in Makefile.config.
If you use cmake, maybe you can change your cud-arch flag to All (Do not remember so clearly, check -DCUDA_ARCH_NAME=ALL) and check cmake/Cuda.cmake (maybe also need to add some arch in this file). If you use VisualStudio, find the flags for NVCC Compiler and modify them.)

# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
		-gencode arch=compute_20,code=sm_21 \
		-gencode arch=compute_30,code=sm_30 \
		-gencode arch=compute_35,code=sm_35 \
		-gencode arch=compute_50,code=sm_50 \
		-gencode arch=compute_52,code=sm_52 \
		-gencode arch=compute_60,code=sm_60 \
		-gencode arch=compute_61,code=sm_61 \
		-gencode arch=compute_61,code=compute_61

[unicoe]

@zhenni Thank you! It solved my problem.

[aliericcantona]

I did the following steps still I have this error:
F1017 17:33:33.041625 870 sgd_solver.cu:19] Check failed: error == cudaSuccess (11 vs. 0) invalid argument

1. GeForce 980 therefore I set -gencode to 52 in Makefile.config:
   -gencode arch=compute_52,code=sm_52
   -gencode arch=compute_52,code=compute_52
2. make runtest -j8 passed successfully
3. changed the batch size from 256 to 30.
   but when I am trying to train a DNN network e.g. https://chunml.github.io/ChunML.github.io/project/Training-Your-Own-Data-On-Caffe/

That error pops up. Any idea?

[WUhailing]

any suggestions for fixing error 'pooling_layer.cu:379] Check failed: error == cudaSuccess (11 vs. 0) invalid argument'?

[stomachacheGE]

any suggestions for fixing error 'pooling_layer.cu:379] Check failed: error == cudaSuccess (11 vs. 0) invalid argument'?

Hi, did you solve the problem? Any hints?