Imbalance in Multi-GPU training

[NOTGOOOOD]

Hello,

When I train the model using multiple GPUs, I find there has multiple processes on the main thread (rank 0).

I haven't found out what's causing it at the moment.

Run command as below:

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4  --use_env main.py

My code main.py as below:

from rfdetr import RFDETRLarge

model = RFDETRLarge()
model.train(dataset_dir="mydataset/", epochs=10, batch_size=4, grad_accum_steps=4, lr=1e-4, output_dir="exp_train/")

nvidia-smi shows below:

[isaacrob-roboflow]

that's a little weird, it also looks like it's using more / different GPUs than it was allocated .. can you show the whole processes panel? are you sure you're only running one training?

[NOTGOOOOD]

Hi, @isaacrob-roboflow !

I'm sure only one training is running. I have tried it agin.

By the way, I'm using the develop branch.

[isaacrob-roboflow]

torch distributed launch is not recommended in newer pytorch. can you try launching with this instead?

torchrun --standalone --nproc_per_node=4

[NOTGOOOOD]

Sorry for the late reply, the problem still exists when I running this script. I don’t know if this problem is caused by the python/torch environment.

CUDA_VISIBLE_DEVICES=4,5,6,7  torchrun --standalone --nproc_per_node=4  main.py