Better handling of Autograd+Fork errors.

[VitalyFedyunin]

Stack from ghstack:

• Better handling of Autograd+Fork errors. #33885 Better handing of Autograd+Fork errors.

Fixes: #32835
Fixes: #5834

Can not combine with CUDA's implementation as each of them requires individual std::once_flag as well as different forked_autograd_child functions. CUDA version relays to python module, autograd uses TORCH_CHECK to report error to python and cpp.

Differential Revision: D20144024

[albanD]

LGTM

[albanD]

nit: "fork" -> "fork-based multiprocessing"
Maybe some people use the multiprocessing package without knowing what fork is?

[dr-ci]

💊 CircleCI build failures summary and remediations

As of commit a3c88af:

None of the build failures appear to be your fault.

• 1/2 broken upstream at merge base 095de1e since Feb 27

  Please rebase on the viable/strict branch (expand for instructions)

  If your commit is newer than viable/strict, you can try basing on an older, stable commit:

  git fetch origin viable/strict
  git rebase --onto viable/strict $(git merge-base origin/master HEAD)
  

  If your commit is older than viable/strict:

  git fetch origin viable/strict
  git rebase viable/strict
  

  Check out the recency history of this "viable master" tracking branch.

• 1/2 recognized as flaky ❄️

  • Re-run these jobs?

Detailed failure analysis

One may explore the probable reasons each build failed interactively on the Dr. CI website.

❄️ 1 failure recognized as flaky

The following build failures have been detected as flaky and may not be your fault:

pytorch_linux_xenial_cuda10_1_cudnn7_py3_nogpu_test (1/1)

Step: "Test" (full log | pattern match details) ❄️

Feb 27 20:25:35 AssertionError: 12 not less than or equal to 1e-05 :

Feb 27 20:25:35 ---------------------------------------------------------------------- 
Feb 27 20:25:35 Traceback (most recent call last): 
Feb 27 20:25:35   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 175, in wrapper 
Feb 27 20:25:35     self._join_processes(fn) 
Feb 27 20:25:35   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 256, in _join_processes 
Feb 27 20:25:35     self._check_return_codes(elapsed_time) 
Feb 27 20:25:35   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 276, in _check_return_codes 
Feb 27 20:25:35     self.assertEqual(p.exitcode, first_process.exitcode) 
Feb 27 20:25:35   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 896, in assertEqual 
Feb 27 20:25:35     super(TestCase, self).assertLessEqual(abs(x - y), prec, message) 
Feb 27 20:25:35 AssertionError: 12 not less than or equal to 1e-05 :  
Feb 27 20:25:35  
Feb 27 20:25:35 ---------------------------------------------------------------------- 
Feb 27 20:25:35 Ran 84 tests in 171.177s 
Feb 27 20:25:35  
Feb 27 20:25:35 FAILED (failures=1, skipped=1) 
Feb 27 20:25:35  
Feb 27 20:25:35 Generating XML reports... 
Feb 27 20:25:35 Traceback (most recent call last): 
Feb 27 20:25:35   File "test/run_test.py", line 493, in <module> 
Feb 27 20:25:35     main() 

🚧 1 upstream failure recognized by patterns:

These builds matched patterns, but were probably caused by upstream breakages:

• pytorch_linux_xenial_py3_6_gcc5_4_ge_config_legacy_test from Feb 27

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

This comment has been revised 5 times.

[albanD]

Nice catch !

[facebook-github-bot]

@VitalyFedyunin merged this pull request in 877ab3a.