You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The src parameter of torch.distributed.broadcast should be of type int, indicating the root from which to broadcast. However, in the above code, the passed parameter is a list of all ranks in the data parallel group.
The above code snippet is from the DistributedDataParallel class in megatron/core/distributed/distributed_data_parallel.py.
To Reproduce
N/A
Expected behavior
N/A
Stack trace/logs
N/A
Environment (please complete the following information):
Describe the bug
The
src
parameter oftorch.distributed.broadcast
should be of typeint
, indicating the root from which to broadcast. However, in the above code, the passed parameter is a list of all ranks in the data parallel group.The above code snippet is from the
DistributedDataParallel
class inmegatron/core/distributed/distributed_data_parallel.py
.To Reproduce
N/A
Expected behavior
N/A
Stack trace/logs
N/A
Environment (please complete the following information):
Proposed fix
The
rank-0
of the data parallel group should be passed in.Additional context
N/A
The text was updated successfully, but these errors were encountered: