When GPU device id > 0, error: cudaSuccess (77 vs. 0) an illegal memory access was encountered

[EvanWeiner]

After following all the instructions, and running the demo, I get:

F1004 19:15:23.296233 20261 syncedmem.hpp:18] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered

I tried with CUDNN on and off. I'm using TitanX. Any advice?

[EvanWeiner]

I found this error will occur if gpu device id > 0

@oh233 , why if I set gpu device id > 0, I will get this error? I tried different GPUs, different machines, different environments, still the same error. Only if I select device id = 0, then it will work. Please help!

[EvanWeiner]

To reproduce, just run: ./tools/demo.py --gpu 1 on a system with 2 or more GPUs.

[ferranrigual]

Any updates on this issue? We would REALLY like to parallelize our trainings. Thank you.

[HaozhiQi]

This is an incorrect setting in the demo.py. The cfg.GPU_ID is not reset in your case. Therefore the nms will output some incorrect values that cause illegal memory accessing.

It will not cause errors in train_net/test_net since cfg.GPU_ID is set accordingly in those scripts.

This is fixed in bc9a269 .

[qinhaifangpku]

@oh233 hi~ I wonder why the cfg.GPU_ID would influence the values output from the nms?
thank you in advance !

[MartinPlantinga]

This is also the case for Affordance Net. There under ./tools/demo.py the line cfg.GPU_ID = args.gpu_id has to be added.