rgnn.py error when multiple GPUs are used

[drchangliu]

It took over almost 50 minutes for one epoch of rgnn.py to run on a 1080Ti GPU. So we tried to use to use multiple GPUs as allowed by the baseline code.

python rgnn.py --device "0,1,2,3" and python rgnn.py --device "0,1,2,3" both provided the following error. Any suggestions? Thanks!

https://i.imgur.com/CaMzRis.png

[weihua916]

The slow speed is probably due to the slow disk read. Each epoch took about 10 minutes on our end, using single GeForce RTX 2080 GPU. I suggest you see whether your GPU is fully utilized or not.

For multi-GPU training, please modify the code following this (change gpus here), but I do not think the current bottleneck is GPU.

[shangyihao]

Hi~weihua,
Due to the limitation of cpu memory, we also tried to multiple GPUS as allowed by the baseline code, but failed to modify the batch size from single gpu to multiple GPUs.
Have your team tried multiple gpus training, and then give us some details or suggestions.
Thanks very much

[weihua916]

We haven't tried multi-GPU training. You may find this comment by @jwkirchenbauer useful.