[1.5 Release][Dist Autograd][Better Engineering] Notify Workers on Failure during Distributed Autograd #34638

osalpekar · 2020-03-12T04:17:23Z

Summary:
Fixes: #27643

This PR manages notifying workers in the event of a failure during distributed autograd. Gracefully handles propagating errors across all nodes in the backward pass and sets state in the local autograd engines accordingly.

Test Plan: Added 2 new tests checking errors when they are thrown in an intermediate node during distributed autograd. Ensured that all existing distributed autograd tests pass.

Differential Revision: D20164420

facebook-github-bot · 2020-03-12T04:17:44Z

This pull request was exported from Phabricator. Differential Revision: D20164420

facebook-github-bot · 2020-03-12T06:39:43Z

This pull request was exported from Phabricator. Differential Revision: D20164420

dr-ci · 2020-03-12T09:44:03Z

💊 CircleCI build failures summary and remediations

As of commit 94cb64f (more details on the Dr. CI page):

1/2 failures introduced in this PR
1/2 broken upstream at merge base 064f628 on Mar 18 from 2:35pm to 7:57pm (10 commits; d927d58 - c747f09)
Please rebase on the viable/strict branch (expand for instructions)

Since your merge base is older than viable/strict, run these commands:
```
git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD
```
Check out the recency history of this "viable master" tracking branch.

🚧 1 upstream failure:

These were probably caused by upstream breakages:

pytorch_windows_vs2019_py36_cuda10.1_build on Mar 18 from 2:35pm to 7:57pm (10 commits; d927d58 - c747f09)

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

This comment has been revised 37 times.

facebook-github-bot · 2020-03-12T21:14:03Z

This pull request was exported from Phabricator. Differential Revision: D20164420

facebook-github-bot · 2020-03-12T22:28:00Z

This pull request was exported from Phabricator. Differential Revision: D20164420

facebook-github-bot · 2020-03-12T22:55:15Z

This pull request was exported from Phabricator. Differential Revision: D20164420

pritamdamania87

Awesome work!

torch/csrc/distributed/autograd/context/context.h

torch/csrc/distributed/autograd/context/container.h

torch/csrc/distributed/autograd/context/context.cpp

torch/csrc/distributed/autograd/rpc_messages/autograd_backward_failure_req.cpp

torch/testing/_internal/distributed/rpc/dist_autograd_test.py

torch/csrc/distributed/autograd/engine/dist_engine.h

torch/csrc/distributed/autograd/engine/dist_engine.cpp

torch/testing/_internal/distributed/rpc/dist_autograd_test.py

facebook-github-bot · 2020-03-16T23:29:18Z

This pull request was exported from Phabricator. Differential Revision: D20164420

torch/csrc/distributed/autograd/context/context.cpp

pritamdamania87 · 2020-03-17T00:27:19Z

torch/testing/_internal/distributed/rpc/dist_autograd_test.py

Can we add the number of DIST_AUTOGRAD_FAILURE_REQ messages received per context id to debug info and then verify here that each node gets one of these messages for all the other context ids? Can add this as a separate PR.

facebook-github-bot · 2020-03-18T06:25:11Z

This pull request was exported from Phabricator. Differential Revision: D20164420

…ilure during Distributed Autograd (pytorch#34638) Summary: Pull Request resolved: pytorch#34638 Fixes: pytorch#27643 This PR manages notifying workers in the event of a failure during distributed autograd. Gracefully handles propagating errors across all nodes in the backward pass and sets state in the local autograd engines accordingly. Test Plan: Added 2 new tests checking errors when they are thrown in an intermediate node during distributed autograd. Ensured that all existing distributed autograd tests pass. Differential Revision: D20164420 fbshipit-source-id: 5aada5544ed12cd7e24053ba3b93f8b9b38ba021

facebook-github-bot · 2020-03-18T19:54:15Z

This pull request was exported from Phabricator. Differential Revision: D20164420

facebook-github-bot · 2020-03-19T02:21:11Z

This pull request has been merged in 5f67c92.

ezyang · 2020-03-19T14:46:22Z

this broke lint

  {
    path: 'test/test_jit.py',
    start_line: 8424,
    end_line: 8424,
    start_column: 33,
    end_column: 33,
    annotation_level: 'failure',
    message: '[E999] SyntaxError: invalid syntax'
  },
  {
    path: 'torch/testing/_internal/distributed/rpc/dist_autograd_test.py',
    start_line: 1,
    end_line: 1,
    start_column: 1,
    end_column: 1,
    annotation_level: 'failure',
    message: "[F401] 'sys' imported but unused"
  }

osalpekar requested review from mrshenli, pritamdamania87 and zhaojuanmao as code owners March 12, 2020 04:17

facebook-github-bot added the fb-exported label Mar 12, 2020

osalpekar force-pushed the export-D20164420 branch from 1f61dec to f943c42 Compare March 12, 2020 06:39

osalpekar force-pushed the export-D20164420 branch from f943c42 to 3103de8 Compare March 12, 2020 21:13

osalpekar force-pushed the export-D20164420 branch from 3103de8 to 4890a80 Compare March 12, 2020 22:27

osalpekar force-pushed the export-D20164420 branch from 4890a80 to 7c01788 Compare March 12, 2020 22:55

pritamdamania87 suggested changes Mar 13, 2020

View reviewed changes

osalpekar force-pushed the export-D20164420 branch from 7c01788 to d26f333 Compare March 16, 2020 23:29

pritamdamania87 approved these changes Mar 17, 2020

View reviewed changes

osalpekar force-pushed the export-D20164420 branch from d26f333 to 17b1cd8 Compare March 18, 2020 06:25

osalpekar requested review from albanD and apaszke as code owners March 18, 2020 06:25

osalpekar force-pushed the export-D20164420 branch from 17b1cd8 to 94cb64f Compare March 18, 2020 19:54

facebook-github-bot closed this in 5f67c92 Mar 19, 2020

facebook-github-bot added the merged label Mar 19, 2020

mrshenli mentioned this pull request Mar 19, 2020

Bucketization (#7284) #34577

Closed

mruberry added the Merged label Oct 28, 2020

[1.5 Release][Dist Autograd][Better Engineering] Notify Workers on Failure during Distributed Autograd #34638

[1.5 Release][Dist Autograd][Better Engineering] Notify Workers on Failure during Distributed Autograd #34638

Uh oh!

Conversation

osalpekar commented Mar 12, 2020

Uh oh!

facebook-github-bot commented Mar 12, 2020

Uh oh!

facebook-github-bot commented Mar 12, 2020

Uh oh!

dr-ci bot commented Mar 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CircleCI build failures summary and remediations

🚧 1 upstream failure:

Uh oh!

facebook-github-bot commented Mar 12, 2020

Uh oh!

facebook-github-bot commented Mar 12, 2020

Uh oh!

facebook-github-bot commented Mar 12, 2020

Uh oh!

pritamdamania87 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

facebook-github-bot commented Mar 16, 2020

Uh oh!

Uh oh!

Uh oh!

pritamdamania87 Mar 17, 2020

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Mar 18, 2020

Uh oh!

facebook-github-bot commented Mar 18, 2020

Uh oh!

facebook-github-bot commented Mar 19, 2020

Uh oh!

ezyang commented Mar 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dr-ci bot commented Mar 12, 2020 •

edited

Loading