Enable test_distributed for ROCm but only with nccl backend [REDUX] #32551

jithunnair-amd · 2020-01-23T20:23:43Z

This is a redux of the original PR #28814 which was reverted in PR #29736 due to test_DistributedDataParallel being suspected as being flaky. Further investigation revealed it wasn't flakiness, but a bug in the PyTorch source code which has been now fixed in PR #32356. This PR is another attempt at enabling the test_distributed unit test suite only for the nccl backend.

kostmo · 2020-01-23T21:06:58Z

💊 CircleCI build failures summary and remediations

As of commit 1e35a0d:

1/1 failures introduced in this PR

Detailed failure analysis

One may explore the probable reasons each build failed interactively on the Dr. CI website.

🕵️ 1 new failure recognized by patterns

The following build failures do not appear to be due to upstream breakage:

pytorch_linux_xenial_py2_7_9_test (1/1)

Step: "Test" (full log | pattern match details)

Feb 06 23:11:08 RuntimeError: test_dataloader failed!

Feb 06 23:11:08 Ran 62 tests in 84.445s 
Feb 06 23:11:08  
Feb 06 23:11:08 FAILED (failures=1, skipped=8) 
Feb 06 23:11:08  
Feb 06 23:11:08 Generating XML reports... 
Feb 06 23:11:08 Traceback (most recent call last): 
Feb 06 23:11:08   File "test/run_test.py", line 486, in <module> 
Feb 06 23:11:08     main() 
Feb 06 23:11:08   File "test/run_test.py", line 479, in main 
Feb 06 23:11:08     raise RuntimeError(message) 
Feb 06 23:11:08 RuntimeError: test_dataloader failed! 
Feb 06 23:11:08 + cleanup 
Feb 06 23:11:08 + retcode=1 
Feb 06 23:11:08 + set +x 
Feb 06 23:11:08 =================== sccache compilation log =================== 
Feb 06 23:11:08 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/tmp/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/tmp/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" } 
Feb 06 23:11:08  
Feb 06 23:11:08 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Feb 06 23:11:08 Compile requests                 52 
Feb 06 23:11:08 Compile requests executed        30 
Feb 06 23:11:08 Cache hits                       23

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

This comment has been revised 5 times.

jithunnair-amd · 2020-01-24T17:11:40Z

Successful CI Run 1: 2 failing checks: clang-tidy and pytorch_macos_10_13_py3_test. Neither failure seems to be related to the PR changes.

jithunnair-amd · 2020-01-24T17:11:59Z

@pytorchbot retest this please

bddppq · 2020-01-28T17:52:48Z

Test is failing https://ci.pytorch.org/jenkins/job/pytorch-builds/job/py3.6-clang7-rocmdeb-ubuntu16.04-trigger/18270

17:51:01 ======================================================================
17:51:01 ERROR: test_broadcast_coalesced_nccl (__main__.CommTest)
17:51:01 ----------------------------------------------------------------------
17:51:01 Traceback (most recent call last):
17:51:01   File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 175, in wrapper
17:51:01     self._join_processes(fn)
17:51:01   File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 256, in _join_processes
17:51:01     self._check_return_codes(elapsed_time)
17:51:01   File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 275, in _check_return_codes
17:51:01     raise RuntimeError('Process {} terminated or timed out after {} seconds'.format(i, elapsed_time))
17:51:01 RuntimeError: Process 1 terminated or timed out after 156.1109459400177 seconds

jithunnair-amd · 2020-01-28T17:56:14Z

Test is failing https://ci.pytorch.org/jenkins/job/pytorch-builds/job/py3.6-clang7-rocmdeb-ubuntu16.04-trigger/18270

17:51:01 ======================================================================
17:51:01 ERROR: test_broadcast_coalesced_nccl (__main__.CommTest)
17:51:01 ----------------------------------------------------------------------
17:51:01 Traceback (most recent call last):
17:51:01   File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 175, in wrapper
17:51:01     self._join_processes(fn)
17:51:01   File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 256, in _join_processes
17:51:01     self._check_return_codes(elapsed_time)
17:51:01   File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 275, in _check_return_codes
17:51:01     raise RuntimeError('Process {} terminated or timed out after {} seconds'.format(i, elapsed_time))
17:51:01 RuntimeError: Process 1 terminated or timed out after 156.1109459400177 seconds

Yes, thank you for putting the details here. I'm running the tests locally to see if I can reproduce the failure.

…due to known comgr issue (flaky?)

jithunnair-amd · 2020-01-31T17:23:19Z

Successful CI Run 2: 2 failing checks: pytorch_linux_xenial_py3_6_gcc5_4_build and pytorch_macos_10_13_py3_test. Neither failure seems to be related to the PR changes.

Testing yet again for good measure.

@pytorchbot retest this please.

jithunnair-amd · 2020-01-31T19:11:09Z

Successful CI Run 3: 2 failing checks: pytorch_linux_xenial_py3_6_gcc5_4_build and pytorch_macos_10_13_py3_test. Neither failure seems to be related to the PR changes.

@bddppq Seems ready for review and merge

facebook-github-bot

@ezyang is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

jithunnair-amd · 2020-02-06T17:15:18Z

@ezyang I notice there are some trivial conflicts in test/run_test.py. Would you like me to resolve them here, or will they be resolved while importing into Phabricator?

ezyang · 2020-02-06T22:26:51Z

If you could resolve them that would be great, sorry about the delay

bddppq · 2020-02-06T23:43:54Z

Trigger more runs to test flakiness:

jithunnair-amd · 2020-02-06T23:44:58Z

Nice, thanks!

jithunnair-amd · 2020-02-07T03:43:48Z

All 10 CI runs above passed.

bddppq

Thanks!

facebook-github-bot

@ezyang is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-02-11T01:20:19Z

@bddppq merged this pull request in 3c4cec5.

…ytorch#32551) Summary: This is a redux of the original PR pytorch#28814 which was reverted in PR pytorch#29736 due to test_DistributedDataParallel being suspected as being flaky. Further investigation revealed it wasn't flakiness, but a bug in the PyTorch source code which has been now fixed in PR pytorch#32356. This PR is another attempt at enabling the test_distributed unit test suite only for the nccl backend. Pull Request resolved: pytorch#32551 Differential Revision: D19729966 Pulled By: bddppq fbshipit-source-id: 12a0d850991a903cc7723d63693b6157071d7115

Enable test_distributed for ROCm but only with nccl backend [REDUX]

2f7b1e7

pytorchbot added the open source label Jan 23, 2020

jithunnair-amd added 2 commits January 23, 2020 20:35

Add/remove code related to properly skipping tests for ROCm

ac13a88

Remove unused reference to TEST_WITH_ROCM

e0d79fc

ngimel added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Jan 25, 2020

ngimel requested a review from bddppq January 25, 2020 23:27

ngimel added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jan 25, 2020

Skip test_broadcast_coalesced_nccl in test_c10d because it times out …

397e0dd

…due to known comgr issue (flaky?)

jithunnair-amd requested a review from ezyang February 3, 2020 17:46

ezyang approved these changes Feb 4, 2020

View reviewed changes

facebook-github-bot reviewed Feb 4, 2020

View reviewed changes

Merge branch 'master' into enable_test_distributed_nccl_backend_redux

1e35a0d

bddppq requested a review from xw285cornell February 6, 2020 23:45

bddppq approved these changes Feb 7, 2020

View reviewed changes

bddppq added the module: rocm AMD GPU support for Pytorch label Feb 7, 2020

facebook-github-bot reviewed Feb 7, 2020

View reviewed changes

jithunnair-amd mentioned this pull request Feb 10, 2020

Use C10_WARP_SIZE to fix functionality on HIP vs CUDA for batch_norm_backward_reduce #33098

Closed

facebook-github-bot closed this in 3c4cec5 Feb 10, 2020

facebook-github-bot added the merged label Feb 11, 2020

mruberry added the Merged label Oct 28, 2020

Enable test_distributed for ROCm but only with nccl backend [REDUX] #32551

Enable test_distributed for ROCm but only with nccl backend [REDUX] #32551

Uh oh!

Conversation

jithunnair-amd commented Jan 23, 2020

Uh oh!

kostmo commented Jan 23, 2020 • edited by dr-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CircleCI build failures summary and remediations

Detailed failure analysis

🕵️ 1 new failure recognized by patterns

pytorch_linux_xenial_py2_7_9_test (1/1)

Uh oh!

jithunnair-amd commented Jan 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jithunnair-amd commented Jan 24, 2020

Uh oh!

bddppq commented Jan 28, 2020

Uh oh!

jithunnair-amd commented Jan 28, 2020

Uh oh!

jithunnair-amd commented Jan 31, 2020

Uh oh!

jithunnair-amd commented Jan 31, 2020

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd commented Feb 6, 2020

Uh oh!

ezyang commented Feb 6, 2020

Uh oh!

bddppq commented Feb 6, 2020

Uh oh!

jithunnair-amd commented Feb 6, 2020

Uh oh!

jithunnair-amd commented Feb 7, 2020

Uh oh!

bddppq left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Feb 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

kostmo commented Jan 23, 2020 •

edited by dr-ci bot

Loading

jithunnair-amd commented Jan 24, 2020 •

edited

Loading