[c10d] Increment sequence numbers on collectives. #55718

rohan-varma · 2021-04-09T22:57:26Z

Stack from ghstack:

[c10d] Increment sequence numbers on collectives. #55718 [c10d] Increment sequence numbers on collectives.

Increments sequence numbers when ProcessGroupGloo::enqueue or
ProcessGroupNCCL::collective is run, which is a common call all collectives
make. The next step will be to log these along with other collective info in
debug mode as well as integrating them with the process group wrapper.

Differential Revision: D27690690

Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/) [ghstack-poisoned]

facebook-github-bot · 2021-04-09T22:57:30Z

💊 CI failures summary and remediations

As of commit 5a61cfb (more details on the Dr. CI page):

2/2 failures possibly* introduced in this PR
- 1/2 non-scanned failure(s)

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build (1/1)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

/builder/manywheel/build_libtorch.sh: line 188: hexdump: command not found

++ echo '/pytorch /pytorch /pytorch ~/project
5a61cfb02ab1c548cf0346b99002ce5d7d1f32c5'
++ [[ gcc5.4_cxx11-abi == *\c\x\x\1\1\-\a\b\i* ]]
++ LIBTORCH_ABI=cxx11-abi-
++ set -x
++ mkdir -p /tmp/libtorch_housecpu
+++ cat debug/libtorch_cpu.so.dbg
+++ gzip -c
+++ tail -c8
+++ hexdump -n4 -e '"%x"'
/builder/manywheel/build_libtorch.sh: line 188: hexdump: command not found
++ CRC32=


Exited with code exit status 1

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/) ghstack-source-id: 126216756 Pull Request resolved: #55718

wayi1 · 2021-04-12T23:31:48Z

test/distributed/test_c10d.py

+
+        if dist.get_world_size(process_group) > 2:
+            # Test when certain ranks don't call collectives
+            if dist.get_rank(process_group) not in [0, 1]:


It may be better to also add a non-contiguous number like 3 in the list of [0, 1].

wayi1 · 2021-04-12T23:34:41Z

test/distributed/test_c10d.py

+
+    @skip_if_lt_x_gpu(4)
+    @requires_nccl()
+    def test_sequence_num_incremented_nccl_subgroup(self):


Nit: Create a helper function to avoid the boilerplate code here, which is same as test_sequence_num_incremented_gloo_subgroup? It seems that only one arg differs in these two tests.

wayi1

Thanks for improving the debuggability!

Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/) [ghstack-poisoned]

Pull Request resolved: #55718 Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. ghstack-source-id: 126779220 Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/)

Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/) [ghstack-poisoned]

Pull Request resolved: #55718 Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. ghstack-source-id: 126819376 Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/)

test/distributed/test_c10d.py

Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/) [ghstack-poisoned]

Pull Request resolved: #55718 Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. ghstack-source-id: 126896607 Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/)

Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/) [ghstack-poisoned]

Pull Request resolved: #55718 Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. ghstack-source-id: 126971120 Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/)

Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/) [ghstack-poisoned]

Pull Request resolved: #55718 Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. ghstack-source-id: 127033533 Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/)

Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/) [ghstack-poisoned]

Pull Request resolved: #55718 Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. ghstack-source-id: 127099736 Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/)

Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/) [ghstack-poisoned]

Pull Request resolved: #55718 Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. ghstack-source-id: 127215077 Differential Revision: [D27690690](https://our.internmc.facebook.com/intern/diff/D27690690/)

rohan-varma · 2021-04-23T16:51:03Z

ROCM succeeded - https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-linux-bionic-rocm4.1-py3.6-build/5174/console

facebook-github-bot · 2021-04-23T20:19:25Z

This pull request has been merged in 7ff1990.

Summary: Pull Request resolved: pytorch#55718 Increments sequence numbers when ProcessGroupGloo::enqueue or ProcessGroupNCCL::collective is run, which is a common call all collectives make. The next step will be to log these along with other collective info in debug mode as well as integrating them with the process group wrapper. ghstack-source-id: 127215077 Test Plan: CI Reviewed By: SciPioneer Differential Revision: D27690690 fbshipit-source-id: cb284b7c760763b7c0f814a41f06656fabf806d6

rohan-varma requested review from H-Huang, mingzhe09088, mrshenli, pritamdamania87, wayi1 and zhaojuanmao as code owners April 9, 2021 22:57

facebook-github-bot added oncall: distributed Add this issue/PR to distributed oncall triage queue cla signed labels Apr 9, 2021

This was referenced Apr 9, 2021

[c10d] monitored_barrier: make other ranks fail when one rank doesn't reach the barrier #55197

Closed

[c10d] sequence number in process group #55319

Closed

wayi1 reviewed Apr 12, 2021

View reviewed changes

wayi1 approved these changes Apr 12, 2021

View reviewed changes

zhaojuanmao approved these changes Apr 19, 2021

View reviewed changes

test/distributed/test_c10d.py Outdated Show resolved Hide resolved

rohan-varma mentioned this pull request Apr 20, 2021

[c10d] Add debug level field in ProcessGroup #56530

Closed

rohan-varma added 2 commits April 20, 2021 17:35

facebook-github-bot closed this in 7ff1990 Apr 23, 2021

facebook-github-bot added the Merged label Apr 23, 2021

facebook-github-bot deleted the gh/rohan-varma/287/head branch April 27, 2021 14:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[c10d] Increment sequence numbers on collectives. #55718

[c10d] Increment sequence numbers on collectives. #55718

rohan-varma commented Apr 9, 2021 •

edited

facebook-github-bot commented Apr 9, 2021 •

edited

wayi1 Apr 12, 2021

wayi1 Apr 12, 2021

wayi1 left a comment

rohan-varma commented Apr 23, 2021

facebook-github-bot commented Apr 23, 2021

[c10d] Increment sequence numbers on collectives. #55718

[c10d] Increment sequence numbers on collectives. #55718

Conversation

rohan-varma commented Apr 9, 2021 • edited

facebook-github-bot commented Apr 9, 2021 • edited

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build (1/1)

wayi1 Apr 12, 2021

Choose a reason for hiding this comment

wayi1 Apr 12, 2021

Choose a reason for hiding this comment

wayi1 left a comment

Choose a reason for hiding this comment

rohan-varma commented Apr 23, 2021

facebook-github-bot commented Apr 23, 2021

rohan-varma commented Apr 9, 2021 •

edited

facebook-github-bot commented Apr 9, 2021 •

edited