Caffe not working on GPU 0, but works fine on GPU 1

[alfredox10]

We have an issue popping up in our multi-gpu systems. For some reason running python where we use the caffemodel files to detect objects returns:
CUBLAS_STATUS_SUCCESS (14 vs. 0) CUBLAS_STATUS_INTERNAL_ERROR

I have verified this is not a memory issue because the process only uses about 500 MB on our 3 GB gpus, and GPU 0 only uses 20-30% memory on idle.

Ubuntu 14.04, Caffe (latest-master), cudnn 3, CUDA 7.5, NVIDIA 352, GTX 780 Ti

full log: http://pastebin.com/a5ZYU83q

[seanbell]

Can you give more information? From https://github.com/BVLC/caffe/blob/master/CONTRIBUTING.md:

When reporting a bug, it's most helpful to provide the following information, where applicable:

• What steps reproduce the bug?
• Can you reproduce the bug using the latest master, compiled with the DEBUG make option?
• What hardware and operating system/distribution are you running?
• If the bug is a crash, provide the backtrace (usually printed by Caffe; always obtainable with gdb).

Also, can you look at other Issues that mention CUBLAS_STATUS_INTERNAL_ERROR; are they relevant? (e.g. #2334 or #770)

[Lzc6996]

@alfredox10 can u share your solutions for this problem?

[alfredox10]

I believe it was a driver mismatch error. This error was persistent for a long time, and it only went away after we updated one machine to the latest drivers and another was completely reset and everything was reinstalled.

[Lzc6996]

think you for your help.

[williford]

@alfredox10 I'm closing since you seem to have solved the issued.