Add Context Manager for Disabling Multithreading in Backwards, use in aot autograd #86245

eellison · 2022-10-04T23:27:21Z

Stack from ghstack (oldest at bottom):

-> Add Context Manager for Disabling Multithreading in Backwards, use in aot autograd #86245

We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and FakeTensorMode getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity.

… aot autograd [ghstack-poisoned]

pytorch-bot · 2022-10-04T23:27:23Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86245

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures, 1 Pending

As of commit 67dd731:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

… aot autograd ghstack-source-id: e59889bd05a2df14235ca9849ebe42e44601eb0f Pull Request resolved: #86245

torch/autograd/grad_mode.py

torch/csrc/autograd/init.cpp

ezyang · 2022-10-05T03:05:38Z

This looks basically fine but deferring to @albanD for final review.

albanD

Sounds good to me.

torch/csrc/autograd/init.cpp

torch/autograd/grad_mode.py

c10/core/AutogradState.h

…rds, use in aot autograd" We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. [ghstack-poisoned]

… aot autograd ghstack-source-id: a608e1cba5489ab9e28a9c5e511567f89bf827c1 Pull Request resolved: #86245

…rds, use in aot autograd" We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. [ghstack-poisoned]

… aot autograd ghstack-source-id: 43cdfe558be6e25bc097c98e1858aa992619fb44 Pull Request resolved: #86245

albanD

Small error in doc, good to go otherwise!

albanD · 2022-10-05T17:51:59Z

docs/source/torch.rst

@@ -268,7 +268,7 @@ Examples::
    set_grad_enabled
    is_grad_enabled
    inference_mode
-    is_inference_mode_enabled
+    set_multithreading_enabled


This should be reverted.

good catch, thanks

…rds, use in aot autograd" We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. [ghstack-poisoned]

… aot autograd ghstack-source-id: c9480c096fe195b176079deeda239de7ca56d3b8 Pull Request resolved: #86245

albanD

SGTM!

…rds, use in aot autograd" We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. [ghstack-poisoned]

… aot autograd ghstack-source-id: b22baf5bbc1d8b3e9ab80eb1ddabdb7306f3c19f Pull Request resolved: #86245

eellison · 2022-10-05T21:23:43Z

@pytorchbot rebase

pytorchmergebot · 2022-10-05T21:25:19Z

@pytorchbot successfully started a rebase job. Check the current status here

…rds, use in aot autograd" We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. [ghstack-poisoned]

pytorchmergebot · 2022-10-05T21:25:32Z

Successfully rebased gh/eellison/335/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/86245)

… aot autograd ghstack-source-id: ba4a24c07c4028a214e19e37a4eea35ce34c57ae Pull Request resolved: #86245

eellison · 2022-10-06T03:26:10Z

@pytorchbot merge

pytorchmergebot · 2022-10-06T03:27:39Z

@pytorchbot successfully started a merge job. Check the current status here.
The merge job was triggered without a flag. This means that your change will be merged once all checks on your PR have passed (ETA: 0-4 Hours). If this is not the intended behavior, feel free to use some of the other merge options in the wiki.
Please reach out to the PyTorch DevX Team with feedback or questions!

github-actions · 2022-10-06T03:28:30Z

Hey @eellison.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

… aot autograd (#86245) (#86245) Summary: We were running into a few issues with running multithreaded backwards in aot_autograd: such as #86136, and `FakeTensorMode` getting into a weird state as a result of not executing functions completely sequentially. The multithreaded backwards is lost in translation when we trace out the backwards anyway, and adds a lot of additional complexity. Pull Request resolved: #86245 Approved by: https://github.com/albanD, https://github.com/yf225 Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/d04889323e2bc0b7321b76e564292565c88b9a5e Reviewed By: seemethere Differential Revision: D40167028 Pulled By: seemethere fbshipit-source-id: f427c71e528deaa494521a61fcbf789d1a964711

Add Context Manager for Disabling Multithreading in Backwards, use in…

45fecc3

… aot autograd [ghstack-poisoned]

eellison requested review from albanD and soulitzer as code owners October 4, 2022 23:27

facebook-github-bot added the cla signed label Oct 4, 2022

eellison added a commit that referenced this pull request Oct 4, 2022

Add Context Manager for Disabling Multithreading in Backwards, use in…

e8f0750

… aot autograd ghstack-source-id: e59889bd05a2df14235ca9849ebe42e44601eb0f Pull Request resolved: #86245

eellison requested review from ezyang and Chillee October 4, 2022 23:31

eellison commented Oct 4, 2022

View reviewed changes

torch/autograd/grad_mode.py Outdated Show resolved Hide resolved

ezyang reviewed Oct 5, 2022

View reviewed changes

torch/csrc/autograd/init.cpp Outdated Show resolved Hide resolved

ezyang reviewed Oct 5, 2022

View reviewed changes

torch/csrc/autograd/init.cpp Outdated Show resolved Hide resolved

albanD approved these changes Oct 5, 2022

View reviewed changes

torch/csrc/autograd/init.cpp Outdated Show resolved Hide resolved

torch/autograd/grad_mode.py Outdated Show resolved Hide resolved

torch/autograd/grad_mode.py Outdated Show resolved Hide resolved

c10/core/AutogradState.h Show resolved Hide resolved

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 5, 2022

eellison added a commit that referenced this pull request Oct 5, 2022

Add Context Manager for Disabling Multithreading in Backwards, use in…

6838f9d

… aot autograd ghstack-source-id: a608e1cba5489ab9e28a9c5e511567f89bf827c1 Pull Request resolved: #86245

eellison added a commit that referenced this pull request Oct 5, 2022

Add Context Manager for Disabling Multithreading in Backwards, use in…

085f9f4

… aot autograd ghstack-source-id: 43cdfe558be6e25bc097c98e1858aa992619fb44 Pull Request resolved: #86245

albanD reviewed Oct 5, 2022

View reviewed changes

eellison added a commit that referenced this pull request Oct 5, 2022

Add Context Manager for Disabling Multithreading in Backwards, use in…

b2ca3a0

… aot autograd ghstack-source-id: c9480c096fe195b176079deeda239de7ca56d3b8 Pull Request resolved: #86245

albanD approved these changes Oct 5, 2022

View reviewed changes

yanboliang mentioned this pull request Oct 5, 2022

TorchBench model tacotron2 failed due to assert device.type != "meta" pytorch/torchdynamo#1483

Closed

eellison added a commit that referenced this pull request Oct 5, 2022

Add Context Manager for Disabling Multithreading in Backwards, use in…

06a12dc

… aot autograd ghstack-source-id: b22baf5bbc1d8b3e9ab80eb1ddabdb7306f3c19f Pull Request resolved: #86245

yf225 approved these changes Oct 5, 2022

View reviewed changes

eellison removed the ciflow/trunk Trigger trunk jobs on your pull request label Oct 5, 2022

pytorchmergebot pushed a commit that referenced this pull request Oct 5, 2022

Add Context Manager for Disabling Multithreading in Backwards, use in…

bf1ecf2

… aot autograd ghstack-source-id: ba4a24c07c4028a214e19e37a4eea35ce34c57ae Pull Request resolved: #86245

pytorchmergebot added the Merged label Oct 6, 2022

pytorchmergebot closed this in d048893 Oct 6, 2022

eellison mentioned this pull request Oct 10, 2022

Turn on Data Dependent Throwing #86480

Closed

facebook-github-bot deleted the gh/eellison/335/head branch June 8, 2023 16:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Context Manager for Disabling Multithreading in Backwards, use in aot autograd #86245

Add Context Manager for Disabling Multithreading in Backwards, use in aot autograd #86245

eellison commented Oct 4, 2022 •

edited

pytorch-bot bot commented Oct 4, 2022 •

edited

ezyang commented Oct 5, 2022

albanD left a comment

albanD left a comment

albanD Oct 5, 2022

eellison Oct 5, 2022

albanD left a comment

eellison commented Oct 5, 2022

pytorchmergebot commented Oct 5, 2022

pytorchmergebot commented Oct 5, 2022

eellison commented Oct 6, 2022

pytorchmergebot commented Oct 6, 2022

github-actions bot commented Oct 6, 2022

Add Context Manager for Disabling Multithreading in Backwards, use in aot autograd #86245

Add Context Manager for Disabling Multithreading in Backwards, use in aot autograd #86245

Conversation

eellison commented Oct 4, 2022 • edited

pytorch-bot bot commented Oct 4, 2022 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86245

✅ No Failures, 1 Pending

ezyang commented Oct 5, 2022

albanD left a comment

Choose a reason for hiding this comment

albanD left a comment

Choose a reason for hiding this comment

albanD Oct 5, 2022

Choose a reason for hiding this comment

eellison Oct 5, 2022

Choose a reason for hiding this comment

albanD left a comment

Choose a reason for hiding this comment

eellison commented Oct 5, 2022

pytorchmergebot commented Oct 5, 2022

pytorchmergebot commented Oct 5, 2022

eellison commented Oct 6, 2022

pytorchmergebot commented Oct 6, 2022

github-actions bot commented Oct 6, 2022

eellison commented Oct 4, 2022 •

edited

pytorch-bot bot commented Oct 4, 2022 •

edited