Caffe Window on 3 GPUs

[smithdir101]

Hi All,

I am dowload the caffe window version from https://github.com/BVLC/caffe/tree/windows.

Curretenly, I have 3 GPUS on my dell machine all of them Titan X.

I run this command and I get these error:

Please see the attached image.

If you can help me it would be great.

Thanks,

Jhon

[pavlejosipovic]

Did you try to run UT (test_all) project to verify that everything is setup correctly?
Next you can try running sample training on a single GPU.

If both pass, you will have to start debugging this assertion you are getting..

[SvenTwo]

I also had problems with 3 GPUs; this pull request fixed it for me: #3586 (not sure if it's applicable to current master though)

[smithdir101]

I solved this issue by activating p2p mode for each of 3 cards.
Use this command c:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi.exe -fdm 1 in order to activate TCC driver mode, not WDDM one. Restart is required! In case your display is connected to the GPU that is going to be in TCC mode, take into account it will operate in 800x600 mode only! It is a good idea to have additional "simple" GPU in order to connect system's display.

Next thing to consider. You cannot use more than 2 GPUs connected to one CPU unless you use tricks like 2GPU on one board (K80) or PCI bus switch on motherboard (there are vendors available). Therefore people often build 2 CPUs system in order to be able to use 4 GPUs. The catch here is that inter-CPU bus QPI still has very high latency relative to inter-PCI bus connections making work of 4 GPUs way less effective comparing to 4GPU's that are connected to the same one CPU.