Why is the first GPU over-consumed?

[xlnwel]

I run the code with the following command, spotting that there are 9 processes occupying the first GPU. Why would that be the case?

python3 train.py --gpu_devices 0,1,2,3 --num_actor_devices 3 --num_actors 3 --training_device 3

The initialization logs look fine to me

Here's a snapshot of the result of nvidia-smi

[daochenzha]

@xlnwel Thanks for the feedback. It seems that the first GPU got a copy of all the processes. It may be caused by the system and CUDA version etc. Do you mind sharing more information, such as your system, GPUs, etc.?

[xlnwel]

Hi, Thanks for replying.

I use 4 1080ti GPUs. The system is Ubuntu 20.04.2 LTS. More CUDA info are listed below

[daochenzha]

@xlnwel Unfortunately, I can not find an exact explanation for this. My guess is when launching the sub-process, it somehow uses device 0 so it occupies some memory in device 0 (for initializing CUDA). However, in our code, there is no operation specifically on device 0 in the act function. I am not sure about the exact reason. Most systems and CUDA versions should not have this issue. I would suggest trying other systems/CUDA versions.

[xlnwel]

Thanks for your response, @daochenzha. Yeah, I've also read the related part of the code without any conclusion. Thanks anyway.