ProcessGroupNCCL does not support scatter

[aoranwu]

When trying running DLRM distributedly using mpirun with NCCL as the backend there is a runtime error "ProcessGroupNCCL does not support scatter". Is there anyway to solve that?

Thanks!

[YazhiGao]

can u paste the complete trace?

[aoranwu]

The trace is as follows:
Traceback (most recent call last):
File "dlrm_s_pytorch.py", line 1056, in
Z = dlrm_wrap(X, lS_o, lS_i, use_gpu, device)
File "dlrm_s_pytorch.py", line 915, in dlrm_wrap
lS_i
File "/home/wisr/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "dlrm_s_pytorch.py", line 331, in forward
return self.distributed_forward(dense_x, lS_o, lS_i)
File "dlrm_s_pytorch.py", line 401, in distributed_forward
a2a_req = ext_dist.alltoall(ly, self.n_emb_per_rank)
File "/home/wisr/aoranwu/dlrm/extend_distributed.py", line 439, in alltoall
output = All2All_Scatter_Req.apply(a2ai, *inputs)
File "/home/wisr/aoranwu/dlrm/extend_distributed.py", line 225, in forward
req = dist.scatter(out_tensor, scatter_list if i == my_rank else [], src=i, async_op=True)
File "/home/wisr/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1380, in scatter
work = _default_pg.scatter(output_tensors, input_tensors, opts)
RuntimeError: ProcessGroupNCCL does not support scatter
Traceback (most recent call last):
File "dlrm_s_pytorch.py", line 1056, in
Z = dlrm_wrap(X, lS_o, lS_i, use_gpu, device)
File "dlrm_s_pytorch.py", line 915, in dlrm_wrap
lS_i
File "/home/wisr/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "dlrm_s_pytorch.py", line 331, in forward
return self.distributed_forward(dense_x, lS_o, lS_i)
File "dlrm_s_pytorch.py", line 401, in distributed_forward
a2a_req = ext_dist.alltoall(ly, self.n_emb_per_rank)
File "/home/wisr/aoranwu/dlrm/extend_distributed.py", line 439, in alltoall
output = All2All_Scatter_Req.apply(a2ai, *inputs)
File "/home/wisr/aoranwu/dlrm/extend_distributed.py", line 225, in forward
req = dist.scatter(out_tensor, scatter_list if i == my_rank else [], src=i, async_op=True)
File "/home/wisr/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1380, in scatter
work = _default_pg.scatter(output_tensors, input_tensors, opts)
RuntimeError: ProcessGroupNCCL does not support scatter

[YazhiGao]

All2All_Scatter_Req

The trace is as follows:
Traceback (most recent call last):
File "dlrm_s_pytorch.py", line 1056, in
Z = dlrm_wrap(X, lS_o, lS_i, use_gpu, device)
File "dlrm_s_pytorch.py", line 915, in dlrm_wrap
lS_i
File "/home/wisr/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "dlrm_s_pytorch.py", line 331, in forward
return self.distributed_forward(dense_x, lS_o, lS_i)
File "dlrm_s_pytorch.py", line 401, in distributed_forward
a2a_req = ext_dist.alltoall(ly, self.n_emb_per_rank)
File "/home/wisr/aoranwu/dlrm/extend_distributed.py", line 439, in alltoall
output = All2All_Scatter_Req.apply(a2ai, *inputs)
File "/home/wisr/aoranwu/dlrm/extend_distributed.py", line 225, in forward
req = dist.scatter(out_tensor, scatter_list if i == my_rank else [], src=i, async_op=True)
File "/home/wisr/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1380, in scatter
work = _default_pg.scatter(output_tensors, input_tensors, opts)
RuntimeError: ProcessGroupNCCL does not support scatter
Traceback (most recent call last):
File "dlrm_s_pytorch.py", line 1056, in
Z = dlrm_wrap(X, lS_o, lS_i, use_gpu, device)
File "dlrm_s_pytorch.py", line 915, in dlrm_wrap
lS_i
File "/home/wisr/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "dlrm_s_pytorch.py", line 331, in forward
return self.distributed_forward(dense_x, lS_o, lS_i)
File "dlrm_s_pytorch.py", line 401, in distributed_forward
a2a_req = ext_dist.alltoall(ly, self.n_emb_per_rank)
File "/home/wisr/aoranwu/dlrm/extend_distributed.py", line 439, in alltoall
output = All2All_Scatter_Req.apply(a2ai, *inputs)
File "/home/wisr/aoranwu/dlrm/extend_distributed.py", line 225, in forward
req = dist.scatter(out_tensor, scatter_list if i == my_rank else [], src=i, async_op=True)
File "/home/wisr/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1380, in scatter
work = _default_pg.scatter(output_tensors, input_tensors, opts)
RuntimeError: ProcessGroupNCCL does not support scatter

this is because NCCL indeed does not support scatter I believe? If you can update your pytorch to the latest, the code should be directly usingt the all2all primitive. For scatter, you have to use MPI guess. But we will update the documentation for better debugging.

[YazhiGao]

see pytorch/pytorch#47291

[mnaumovfb]

It looks like this is resolved.

[LukeLIN-web]

see this photo, https://pytorch.org/docs/stable/distributed.html