New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[distributed] calling nccl reduce with inconsistent dst hangs #39706

Open

ssnl opened this issue Jun 9, 2020 · 1 comment

Labels

module: deadlock module: nccl oncall: distributed triaged

Collaborator

ssnl commented Jun 9, 2020 •

edited by pytorch-probot bot

Not sure if intended or avoidable, but if dst is inconsistent across ranks, reduce finishes, but future kernels seems to hang. E.g.,

import torch
torch.distributed.init_process_group('nccl', init_method='tcp://localhost:10402', world_size=2, rank=0)
x = torch.zeros(3, device=0)
torch.distributed.reduce(x, 0, torch.distributed.ReduceOp.SUM)
print(x)

import torch
torch.distributed.init_process_group('nccl', init_method='tcp://localhost:10402', world_size=2, rank=1)
x = torch.zeros(3, device=1)
torch.distributed.reduce(x, 1, torch.distributed.ReduceOp.SUM)
print(x)

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar

The text was updated successfully, but these errors were encountered:

albanD added oncall: distributed module: nccl module: deadlock triaged labels

Contributor

mrshenli commented Jun 9, 2020

rohan-varma mentioned this issue

ProcessGroup wrapper that aids collectives desynchronization #22071

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment