Single Node distributed multiprocessing training is not working

[kuanhan1]

Hi, I wanted to train a model on imagenet with single node distributed multiprocssing. Here is the error log:

Start training
Start training
Start training
Start training
Traceback (most recent call last):
File "main_multiprocessing_distributed.py", line 483, in
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
File "/home/libi/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 141, in spawn
while not spawn_context.join():
File "/home/libi/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 91, in join
raise Exception(msg)
Exception:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/home/libi/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 11, in _wrap
fn(i, *args)
File "/home/han424/projects/PCN/PCN_imagenet/main_multiprocessing_distributed.py", line 254, in main_worker
train(train_loader, model, criterion, optimizer, epoch, args)
File "/home/han424/projects/PCN/PCN_imagenet/main_multiprocessing_distributed.py", line 289, in train
for i, (input, target) in enumerate(train_loader):
File "/home/libi/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 819, in iter
return _DataLoaderIter(self)
File "/home/libi/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 560, in init
w.start()
File "/home/libi/anaconda3/lib/python3.7/multiprocessing/process.py", line 110, in start
'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children

I would be thankful to any advice on it.

[yrsun]

I had the same problem. Sometimes the program seems have deadlock in def train at:
257: for i, (input, target) in enumerate(train_loader):

Sometimes I got output like this:
Use GPU: 1 for training
Use GPU: 2 for training
Use GPU: 0 for training
Use GPU: 3 for training
=> creating model 'resnet50'
=> creating model 'resnet50'
=> creating model 'resnet50'
=> creating model 'resnet50'
Traceback (most recent call last):
File "main.py", line 389, in
main()
File "main.py", line 106, in main
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
File "/home/yiru/.local/lib/python3.5/site-packages/torch/multiprocessing/spawn.py", line 146, in spawn
while not spawn_context.join():
File "/home/yiru/.local/lib/python3.5/site-packages/torch/multiprocessing/spawn.py", line 101, in join
raise Exception(msg)
Exception:

-- Process 2 terminated with the following error:
Traceback (most recent call last):
File "/home/yiru/.local/lib/python3.5/site-packages/torch/multiprocessing/spawn.py", line 11, in _wrap
fn(i, *args)
File "/home/yiru/src/pytorch_raw/examples/imagenet/main.py", line 229, in main_worker
train(train_loader, model, criterion, optimizer, epoch, args)
File "/home/yiru/src/pytorch_raw/examples/imagenet/main.py", line 257, in train
for i, (input, target) in enumerate(train_loader):
File "/home/yiru/.local/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 501, in iter
return _DataLoaderIter(self)
File "/home/yiru/.local/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 289, in init
w.start()
File "/usr/lib/python3.5/multiprocessing/process.py", line 103, in start
'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children

My command line is:
python3 main.py -a resnet50 --lr 0.01 --dist-url 'tcp://192.168.100.1:1234' --dist-backend 'nccl' --multiprocessing-distributed ~/imagenet/

I have researched for some solutions, but none of them worked for me.
I have tried:
(A) -j 0, this seems to work, but the program randomly results to broken pipe and forces my server to reboot in the following lines:
266: output = model(input)
277: loss.backward()
(B) updated to the latest openCV, no help
(C) pin_memory=False, no help

I really appreciate if anybody can help us.

[yrsun]

Hi, I finally figured out a solution. Maybe you can try this:
CUDA_VISIBLE_DEVICES=0,1 python3 main.py -j 0 -a resnet50 --lr 0.01 --epochs 20 --dist-url 'tcp://192.168.100.1:9999' --gpu-number 2 --dist-backend 'nccl' --multiprocessing-distributed ~/imagenet/

The trick part is: CUDA_VISIBLE_DEVICES=0 for 1GPU
CUDA_VISIBLE_DEVICES=0,1 for 2GPUs

Please let me know if this solution can help you.

Good luck

[mosjoker]

maybe num_works(default: 4) > 0 cause it ?

[sunkevin1214]

Hi, I finally figured out a solution. Maybe you can try this:
CUDA_VISIBLE_DEVICES=0,1 python3 main.py -j 0 -a resnet50 --lr 0.01 --epochs 20 --dist-url 'tcp://192.168.100.1:9999' --gpu-number 2 --dist-backend 'nccl' --multiprocessing-distributed ~/imagenet/

The trick part is: CUDA_VISIBLE_DEVICES=0 for 1GPU
CUDA_VISIBLE_DEVICES=0,1 for 2GPUs

Please let me know if this solution can help you.

Good luck

This answer cannot work for me, is there any other solution? Because the error is "AttributeError: module 'torch.multiprocessing' has no attribute 'spawn'", I guess maybe the python version problem? I am using pytorch1.0 and python3.7