Check failed: status == CURAND_STATUS_SUCCESS (201 vs. 0) CURAND_STATUS_LAUNCH_FAILURE

[Feynman27]

Hi,

I'm attempting to run pretrain.py on the mnist dataset out of the box, but within the function pretrain_main(), I hit an error affiliated with Cuda. The trace is below:

F0203 19:28:13.953368  9487 math_functions.cu:394] Check failed: status == CURAND_STATUS_SUCCESS (201 vs. 0)  CURAND_STATUS_LAUNCH_FAILURE
*** Check failure stack trace: ***
    @     0x7fb2a5554daa  (unknown)
    @     0x7fb2a5554ce4  (unknown)
    @     0x7fb2a55546e6  (unknown)
    @     0x7fb2a5557687  (unknown)
    @           0x4ae358  caffe::caffe_gpu_rng_uniform()
    @           0x4faff4  caffe::DropoutLayer<>::Forward_gpu()
    @           0x46a16b  caffe::Net<>::ForwardFromTo()
    @           0x46a597  caffe::Net<>::ForwardPrefilled()
    @           0x4aadae  caffe::Solver<>::Solve()
    @           0x417f62  train()
    @           0x4118f1  main
    @     0x7fb2a1f69f45  (unknown)
    @           0x416997  (unknown)
    @              (nil)  (unknown)
Aborted (core dumped)

I'm running from the docker image. Maybe it has something to do with the cuda version? The image uses CUDA 7.0.

[Feynman27]

Modifying the Dockerfile to use cuda8.0 fixed the problem:

FROM bfolkens/docker-opencv:3.1.0-cuda8.0-cudnn5

I had to remove import cv from dec.py since this isn't included in opencv3. Other than that, everything seems to be working fine.

[raymondsa]

I had the same error when i run faster CNN+WIN10+MATLAB2015+CUDA 8.0 GPU GTX1080.
The solution was:
https://github.com/ShaoqingRen/caffe/tree/faster-R-CNN