Distributed training all-reduce order

[zhiqi-0]

Hi,

I'm just wondering if there is a potential issue for all-reduce order when both data parallelism and tensor model parallelism are enabled during training. With torch DDP, both tensor model parallelism and data parallelism use all-reduce, and are launched on different streams. While the execution order is determined by hardware, will it cause hanging in some cases like:

GPU1: [MP] all-reduce -> [DP] all-reduce
GPU2: [DP] all-reduce -> [MP] all-reduce

From issues discussed here, I think it may be unsafe for undetermined order.

[github-actions]

Marking as stale. No activity in 60 days. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

Marking as stale. No activity in 60 days.

[fwyc0573]

I have encountered the same problem. Can anyone provide some opinions?

[deepakn94]

@fwyc0573 are you seeing a hang? Can you describe the setting, perhaps provide an example command line, and also paste the last couple of lines in the logs?

[github-actions]

Marking as stale. No activity in 60 days.