[Gloo] Support work-level timeouts in ProcessGroupGloo

[osalpekar]

Stack from ghstack:

• [Gloo] Tests for Gloo Async Work Wait-level Timeouts #41265 [Gloo] Tests for Gloo Async Work Wait-level Timeouts
• [Gloo] Support work-level timeouts in ProcessGroupGloo #40948 [Gloo] Support work-level timeouts in ProcessGroupGloo
• [NCCL] Tests for WorkNCCL::wait with Timeouts #40947 [NCCL] Tests for WorkNCCL::wait with Timeouts
• [NCCL] Support Wait Timeout in ProcessGroupNCCL #40946 [NCCL] Support Wait Timeout in ProcessGroupNCCL
• [NCCL] Add timeout to ProcessGroup Work Wait #40944 [NCCL] Add timeout to ProcessGroup Work Wait

Add work-level timeouts to ProcessGroupGloo. This uses the timeout support in waitSend and waitRecv functions from Gloo's unbound_buffer construct.

Differential Revision: D22173763

[dr-ci]

💊 CI failures summary and remediations

As of commit 0c14c90 (more details on the Dr. CI page):

---

• 4/4 failures possibly* introduced in this PR
  • 1/4 non-CircleCI failure(s)

---

🕵️ 3 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test (1/3)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jul 15 20:05:30 FAIL [0.032s]: test_interpolate_linear_1d_alert_nondeterministic_cuda (__main__.TestNN)

Jul 15 20:05:28   test_sparse_default_std (__main__.TestNNInit) ... ok (0.007s) 
Jul 15 20:05:28   test_sparse_only_works_on_2d_inputs (__main__.TestNNInit) ... ok (0.001s) 
Jul 15 20:05:29   test_trunc_normal (__main__.TestNNInit) ... ok (0.703s) 
Jul 15 20:05:30   test_uniform (__main__.TestNNInit) ... ok (0.904s) 
Jul 15 20:05:30   test_xavier_normal (__main__.TestNNInit) ... ok (0.105s) 
Jul 15 20:05:30   test_xavier_normal_errors_on_inputs_smaller_than_2d (__main__.TestNNInit) ... ok (0.001s) 
Jul 15 20:05:30   test_xavier_uniform (__main__.TestNNInit) ... ok (0.079s) 
Jul 15 20:05:30   test_xavier_uniform_errors_on_inputs_smaller_than_2d (__main__.TestNNInit) ... ok (0.001s) 
Jul 15 20:05:30  
Jul 15 20:05:30 ====================================================================== 
Jul 15 20:05:30 FAIL [0.032s]: test_interpolate_linear_1d_alert_nondeterministic_cuda (__main__.TestNN) 
Jul 15 20:05:30 ---------------------------------------------------------------------- 
Jul 15 20:05:30 Traceback (most recent call last): 
Jul 15 20:05:30   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 777, in wrapper 
Jul 15 20:05:30     method(*args, **kwargs) 
Jul 15 20:05:30   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 777, in wrapper 
Jul 15 20:05:30     method(*args, **kwargs) 
Jul 15 20:05:30   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 674, in efail_fn_no_device 
Jul 15 20:05:30     return efail_fn(slf, None, *args, **kwargs) 
Jul 15 20:05:30   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 665, in efail_fn 
Jul 15 20:05:30     slf.fail('expected a non-deterministic error, but it was not raised') 

pytorch_windows_vs2019_py36_cuda10.1_test2 (2/3)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

AssertionError: expected a non-deterministic error, but it was not raised

  test_xavier_uniform_errors_on_inputs_smaller_than_2d (__main__.TestNNInit) ... ok (0.014s) 
 
====================================================================== 
FAIL [0.026s]: test_interpolate_linear_1d_alert_nondeterministic_cuda (__main__.TestNN) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_device_type.py", line 674, in efail_fn_no_device 
    return efail_fn(slf, None, *args, **kwargs) 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_device_type.py", line 665, in efail_fn 
    slf.fail('expected a non-deterministic error, but it was not raised') 
AssertionError: expected a non-deterministic error, but it was not raised 
 
---------------------------------------------------------------------- 
Ran 1800 tests in 729.033s 
 
FAILED (failures=1, skipped=90, expected failures=4) 
 
Generating XML reports... 
Generated XML report: test-reports\python-unittest\TEST-PackedSequenceTest-20200715200851.xml 
Generated XML report: test-reports\python-unittest\TEST-TestAddRelu-20200715200851.xml 
Generated XML report: test-reports\python-unittest\TEST-TestAvgPool-20200715200851.xml 

pytorch_windows_vs2019_py36_cuda10.1_test1 (3/3)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

AssertionError: expected a non-deterministic error, but it was not raised

  test_xavier_uniform_errors_on_inputs_smaller_than_2d (__main__.TestNNInit) ... ok (0.016s) 
 
====================================================================== 
FAIL [0.025s]: test_interpolate_linear_1d_alert_nondeterministic_cuda (__main__.TestNN) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_device_type.py", line 674, in efail_fn_no_device 
    return efail_fn(slf, None, *args, **kwargs) 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_device_type.py", line 665, in efail_fn 
    slf.fail('expected a non-deterministic error, but it was not raised') 
AssertionError: expected a non-deterministic error, but it was not raised 
 
---------------------------------------------------------------------- 
Ran 1800 tests in 654.907s 
 
FAILED (failures=1, skipped=90, expected failures=4) 
 
Generating XML reports... 
Generated XML report: test-reports\python-unittest\TEST-PackedSequenceTest-20200715193432.xml 
Generated XML report: test-reports\python-unittest\TEST-TestAddRelu-20200715193432.xml 
Generated XML report: test-reports\python-unittest\TEST-TestAvgPool-20200715193432.xml 

---

ci.pytorch.org: 1 failed

• Failed: pr/pytorch-linux-xenial-rocm3.5.1-py3.6

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 23 times.

[mrshenli]

Shall we add a test for this?

[osalpekar]

Shall we add a test for this?

Added tests in the next PR in this stack (#41265)

[facebook-github-bot]

This pull request has been merged in b979129.