[NCCL] Support NCCL Send/Recv #44921

mingzhe09088 · 2020-09-18T00:08:52Z

Stack from ghstack:

[NCCL] support send/recv to/from self when communicator is created on demand #45873 [NCCL] support send/recv to/from self when communicator is created on demand
[NCCL] create NCCL communicator for send/recv on demand #44922 [NCCL] create NCCL communicator for send/recv on demand
[NCCL] Support NCCL Send/Recv #44921 [NCCL] Support NCCL Send/Recv

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context.

Differential Revision: D23709848

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

dr-ci · 2020-09-18T00:12:35Z

💊 CI failures summary and remediations

As of commit d98ef7e (more details on the Dr. CI page):

Commit d98ef7e was recently pushed. Waiting for builds...

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 53 times.

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

Pull Request resolved: #44921 This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. ghstack-source-id: 112391583 Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/)

torch/distributed/distributed_c10d.py

torch/testing/_internal/distributed/distributed_test.py

torch/distributed/distributed_c10d.py

torch/testing/_internal/distributed/distributed_test.py

torch/csrc/distributed/c10d/init.cpp

torch/distributed/distributed_c10d.py

torch/lib/c10d/ProcessGroupNCCL.hpp

torch/testing/_internal/distributed/distributed_test.py

torch/lib/c10d/ProcessGroupNCCL.cpp

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Todo: add more tests. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

Pull Request resolved: #44921 This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. ghstack-source-id: 112545024 Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/)

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

torch/distributed/distributed_c10d.py

torch/lib/c10d/ProcessGroupNCCL.cpp

torch/distributed/distributed_c10d.py

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

Pull Request resolved: #44921 This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. ghstack-source-id: 112722354 Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D23709848/)!

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

Pull Request resolved: #44921 This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. ghstack-source-id: 112735516 Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D23709848/)!

pritamdamania87

LGTM, @jiayisuse can you take another look at the PR too?

torch/distributed/distributed_c10d.py

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

Pull Request resolved: #44921 This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. ghstack-source-id: 112769784 Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D23709848/)!

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

Pull Request resolved: #44921 This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. ghstack-source-id: 112835901 Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D23709848/)!

codecov · 2020-09-24T23:00:45Z

Codecov Report

Merging #44921 into gh/mingzhe09088/1/base will decrease coverage by 0.15%.
The diff coverage is 22.04%.

@@                    Coverage Diff                     @@
##           gh/mingzhe09088/1/base   #44921      +/-   ##
==========================================================
- Coverage                   68.32%   68.16%   -0.16%     
==========================================================
  Files                         410      410              
  Lines                       52981    53165     +184     
==========================================================
+ Hits                        36199    36240      +41     
- Misses                      16782    16925     +143

Impacted Files	Coverage Δ
torch/distributed/distributed_c10d.py	`27.83% <20.45%> (-0.52%)`	⬇️
.../testing/_internal/distributed/distributed_test.py	`29.55% <21.98%> (-0.59%)`	⬇️
torch/distributed/__init__.py	`88.88% <100.00%> (+1.38%)`	⬆️
torch/testing/_internal/expecttest.py	`78.57% <0.00%> (+1.02%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ab88c3...d98ef7e. Read the comment docs.

torch/distributed/distributed_c10d.py

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

Pull Request resolved: #44921 This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. ghstack-source-id: 112935178 Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D23709848/)!

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

facebook-github-bot · 2020-10-06T02:14:03Z

This pull request has been merged in 59083d6.

#65601) Summary: Pull Request resolved: #65601 I believe this feature was supported one year ago: #44921 #Closes: #65525 ghstack-source-id: 138918961 Test Plan: N/A Reviewed By: pritamdamania87, mingzhe09088 Differential Revision: D31163535 fbshipit-source-id: 9321a0a5137a3e265e2b54bd78730ac28c7acd55

[WIP][NCCL] Support NCCL Send/Recv

7d522da

This diff adds support for Process Group point-to-point operations on NCCL backend based on ncclSend/ncclRecv. See #43995 for more context. Differential Revision: [D23709848](https://our.internmc.facebook.com/intern/diff/D23709848/) [ghstack-poisoned]

mingzhe09088 requested review from apaszke, mrshenli, pietern, pritamdamania87, rohan-varma and zhaojuanmao as code owners September 18, 2020 00:08

mingzhe09088 mentioned this pull request Sep 18, 2020

[NCCL] create NCCL communicator for send/recv on demand #44922

Closed

pritamdamania87 suggested changes Sep 18, 2020

View reviewed changes

jiayisuse reviewed Sep 21, 2020

View reviewed changes

mingzhe09088 changed the title ~~[WIP][NCCL] Support NCCL Send/Recv~~ [NCCL] Support NCCL Send/Recv Sep 21, 2020

mingzhe09088 mentioned this pull request Sep 22, 2020

[NCCL][Test Only] test send/recv on OSS #45140

Closed

pritamdamania87 reviewed Sep 22, 2020

View reviewed changes

torch/distributed/distributed_c10d.py Outdated Show resolved Hide resolved

torch/lib/c10d/ProcessGroupNCCL.cpp Show resolved Hide resolved

torch/distributed/distributed_c10d.py Outdated Show resolved Hide resolved

pritamdamania87 approved these changes Sep 23, 2020

View reviewed changes

torch/distributed/distributed_c10d.py Outdated Show resolved Hide resolved

torch/distributed/distributed_c10d.py Outdated Show resolved Hide resolved

torch/distributed/distributed_c10d.py Show resolved Hide resolved

torch/distributed/distributed_c10d.py Outdated Show resolved Hide resolved

pritamdamania87 reviewed Sep 23, 2020

View reviewed changes

torch/distributed/distributed_c10d.py Outdated Show resolved Hide resolved

jiayisuse approved these changes Sep 24, 2020

View reviewed changes

mrshenli reviewed Sep 25, 2020

View reviewed changes

torch/distributed/distributed_c10d.py Outdated Show resolved Hide resolved

torch/distributed/distributed_c10d.py Outdated Show resolved Hide resolved

This was referenced Sep 25, 2020

[NCCL][Test Only] extra group #45363

Closed

[NCCL][Test Only] no change #45320

Closed

[Test Only] comment out a2a #45679

Closed

mingzhe09088 mentioned this pull request Oct 5, 2020

[NCCL] support send/recv to/from self when communicator is created on demand #45873

Closed

facebook-github-bot closed this in 59083d6 Oct 6, 2020

facebook-github-bot added the merged label Oct 6, 2020

mruberry added the Merged label Oct 28, 2020

wayi1 mentioned this pull request Sep 24, 2021

Update distributed.rst to show that CUDA send/recv on GPU is supported #65601

Closed

yaox12 mentioned this pull request Nov 2, 2022

Replace third_party/nccl with PyTorch's NCCL backend dmlc/dgl#4590

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NCCL] Support NCCL Send/Recv #44921

[NCCL] Support NCCL Send/Recv #44921

mingzhe09088 commented Sep 18, 2020 •

edited

dr-ci bot commented Sep 18, 2020 •

edited

pritamdamania87 left a comment

codecov bot commented Sep 24, 2020 •

edited

facebook-github-bot commented Oct 6, 2020

[NCCL] Support NCCL Send/Recv #44921

[NCCL] Support NCCL Send/Recv #44921

Conversation

mingzhe09088 commented Sep 18, 2020 • edited

dr-ci bot commented Sep 18, 2020 • edited

💊 CI failures summary and remediations

pritamdamania87 left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 24, 2020 • edited

Codecov Report

facebook-github-bot commented Oct 6, 2020

mingzhe09088 commented Sep 18, 2020 •

edited

dr-ci bot commented Sep 18, 2020 •

edited

codecov bot commented Sep 24, 2020 •

edited