All tensors must be on devices[0]: 0

[Ice-cool]

🐛 Describe the bug

For https://github.com/hpcaitech/ColossalAI-Examples/tree/main/image/resnet, when use python -m torch.distributed.launch --nproc_per_node 2 --master_addr localhost --master_port 29500 run_resnet_cifar10_with_engine.py, there is an error that All tensors must be on devices[0]: 0

Environment

torch=1.8.1

[FrankLeeeee]

Hi, did you build colossalai from source?

[Ice-cool]

yes, using nproc_per_node 1 is ok. but nproc_per_node 2 will meet this error.

[FrankLeeeee]

Ok, let me try to reproduce this error. May I know which GPU you are using and how many GPUs are available on your machine?

[Ice-cool]

Tesla-V100 and 2GPUs are avaliable in my machine.

[FrankLeeeee]

Got, let me try to reproduce this issue. I will get back to you soon!

[FrankLeeeee]

Hi, sorry for my late reply. We only got A100 machines so it took a while for me to look for a V100 machine. This bug can be reproduced on torch 1.8 but not torch 1.10. This bug is due to an optional argument in pytorch DistributedDataParallel. This bug will be fixed in #194 .