[dynamo, nested graph breaks] add nested graph break tests #144516

williamwen42 · 2025-01-10T00:08:37Z

Stack from ghstack (oldest at bottom):

Note: nested graph break tests (and wrapped tests) are xfailed/skipped for now - we will iteratively enable the tests as more of the nested graph break implementation is complete.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @Lucaskabela

Differential Revision: D81084809

[ghstack-poisoned]

pytorch-bot · 2025-01-10T00:08:41Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/144516

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Pending, 1 Unrelated Failure

As of commit 1e9764f with merge base 4e630f0 ():

NEW FAILURE - The following job has failed:

trunk / linux-jammy-rocm-py3.10 / test (default, 2, 2, linux.rocm.gpu.gfx942.1) (gh)
test_cuda.py::TestCudaMallocAsync::test_allocator_settings

FLAKY - The following job failed but was likely due to flakiness present on trunk:

trunk / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 1, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (similar failure)
test_cuda.py::TestCudaMallocAsync::test_allocator_settings

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 2e25912 Pull Request resolved: #144516

github-actions · 2025-03-11T06:39:38Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

ghstack-source-id: eef6717 Pull Request resolved: pytorch/pytorch#144516

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

ghstack-source-id: f415878 Pull Request resolved: #144516

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Pull Request resolved: #159817 Approved by: https://github.com/anijain2305 ghstack dependencies: #157971, #159281, #144516, #159329, #159678

Fix comments to reflect that we no longer codegen cells to be sent to resume function as inputs - they are instead codegen'd after the unsupported instruction in order to build resume functions that are closures. Also simplify some codegen. Pull Request resolved: #160138 Approved by: https://github.com/anijain2305 ghstack dependencies: #157971, #159281, #144516, #159329, #159678, #159817

Nested continuation function code objects are now unique w.r.t. stack trace below (and including) the current code object. Without this change, e.g. in the added test, `f3` would be recompiled on the second graph break. Followup: we can skip guards on continuation functions. Pull Request resolved: #159786 Approved by: https://github.com/anijain2305 ghstack dependencies: #157971, #159281, #144516, #159329, #159678, #159817, #160138

atalman · 2025-08-26T20:38:27Z

@pytorchmergebot revert -c ghfirst -m "failing internal tests"

pytorchmergebot · 2025-08-26T20:39:59Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-08-26T20:40:20Z

@williamwen42 your PR has been successfully reverted.

…144516)" This reverts commit 9a756c2. Reverted #144516 on behalf of https://github.com/atalman due to failing internal tests ([comment](#144516 (comment)))

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]

williamwen42 · 2025-08-26T23:02:47Z

@williamwen42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

atalman · 2025-08-27T02:59:09Z

@pytorchmergebot merge -f "we can reland this PR"

pytorchmergebot · 2025-08-27T03:00:40Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Summary: Note: nested graph break tests (and wrapped tests) are xfailed/skipped for now - we will iteratively enable the tests as more of the nested graph break implementation is complete. X-link: pytorch/pytorch#144516 Approved by: https://github.com/anijain2305 Reviewed By: atalman Differential Revision: D81084809 fbshipit-source-id: fd577714ac22a7544883ac7f82be4c7e9781dc4a

…44516) Pull Request resolved: pytorch#144516 Approved by: https://github.com/anijain2305 ghstack dependencies: pytorch#157971, pytorch#159281

…pytorch#159329) e.g. this graph breaks once now: ```python import torch torch._dynamo.config.nested_graph_breaks = True def inner(x): x = x + 1 torch._dynamo.graph_break() return x + 2 @torch.compile(backend="eager") def outer(x): return inner(x) print(outer(torch.ones(3))) ``` Pull Request resolved: pytorch#159329 Approved by: https://github.com/anijain2305 ghstack dependencies: pytorch#157971, pytorch#159281, pytorch#144516

…anagers (pytorch#159678) Pull Request resolved: pytorch#159678 Approved by: https://github.com/anijain2305 ghstack dependencies: pytorch#157971, pytorch#159281, pytorch#144516, pytorch#159329

Pull Request resolved: pytorch#159817 Approved by: https://github.com/anijain2305 ghstack dependencies: pytorch#157971, pytorch#159281, pytorch#144516, pytorch#159329, pytorch#159678

…160138) Fix comments to reflect that we no longer codegen cells to be sent to resume function as inputs - they are instead codegen'd after the unsupported instruction in order to build resume functions that are closures. Also simplify some codegen. Pull Request resolved: pytorch#160138 Approved by: https://github.com/anijain2305 ghstack dependencies: pytorch#157971, pytorch#159281, pytorch#144516, pytorch#159329, pytorch#159678, pytorch#159817

…ch#159786) Nested continuation function code objects are now unique w.r.t. stack trace below (and including) the current code object. Without this change, e.g. in the added test, `f3` would be recompiled on the second graph break. Followup: we can skip guards on continuation functions. Pull Request resolved: pytorch#159786 Approved by: https://github.com/anijain2305 ghstack dependencies: pytorch#157971, pytorch#159281, pytorch#144516, pytorch#159329, pytorch#159678, pytorch#159817, pytorch#160138

…ytorch#144516)" This reverts commit 9a756c2. Reverted pytorch#144516 on behalf of https://github.com/atalman due to failing internal tests ([comment](pytorch#144516 (comment)))

…44516) Note: nested graph break tests (and wrapped tests) are xfailed/skipped for now - we will iteratively enable the tests as more of the nested graph break implementation is complete. Differential Revision: [D81084809](https://our.internmc.facebook.com/intern/diff/D81084809) Pull Request resolved: pytorch#144516 Approved by: https://github.com/anijain2305

add nested graph break tests

40e526d

[ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: dynamo labels Jan 10, 2025

williamwen42 added a commit that referenced this pull request Jan 10, 2025

add nested graph break tests

cb31079

ghstack-source-id: 2e25912 Pull Request resolved: #144516

williamwen42 marked this pull request as draft January 10, 2025 00:16

github-actions bot added the Stale label Mar 11, 2025

williamwen42 removed the Stale label Mar 11, 2025

Update on "add nested graph break tests"

04e409b

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

williamwen42 mentioned this pull request Apr 10, 2025

[dynamo, nested graph breaks] small fixes to resume function generation #151056

Closed

williamwen42 changed the title ~~add nested graph break tests~~ [dynamo, nested graph breaks] add nested graph break tests Apr 10, 2025

Update on "[dynamo, nested graph breaks] add nested graph break tests"

0415dc9

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Divigroup-RAP pushed a commit to Divigroup-RAP/PYTORCH that referenced this pull request Apr 22, 2025

add nested graph break tests

3ceaebf

ghstack-source-id: eef6717 Pull Request resolved: pytorch/pytorch#144516

Update on "[dynamo, nested graph breaks] add nested graph break tests"

59b736c

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

williamwen42 mentioned this pull request May 14, 2025

[dynamo, nested graph breaks] refactor codegen to minimize NULL codegen'ing #153510

Closed

Update on "[dynamo, nested graph breaks] add nested graph break tests"

01b5875

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "[dynamo, nested graph breaks] add nested graph break tests"

09c01a2

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "[dynamo, nested graph breaks] add nested graph break tests"

b1ab7a0

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "[dynamo, nested graph breaks] add nested graph break tests"

5a419d0

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

williamwen42 added a commit that referenced this pull request May 15, 2025

add nested graph break tests

c31d0e0

ghstack-source-id: f415878 Pull Request resolved: #144516

Update on "[dynamo, nested graph breaks] add nested graph break tests"

a3f4af1

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "[dynamo, nested graph breaks] add nested graph break tests"

a2d97b2

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

williamwen42 mentioned this pull request May 17, 2025

[dynamo, nested graph breaks] remove block stack graph break in output_graph #153772

Closed

pytorchmergebot pushed a commit that referenced this pull request Aug 26, 2025

[dynamo, nested graph breaks] support nested closures (#159817)

ef0ef6f

Pull Request resolved: #159817 Approved by: https://github.com/anijain2305 ghstack dependencies: #157971, #159281, #144516, #159329, #159678

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Aug 26, 2025

pytorchmergebot reopened this Aug 26, 2025

williamwen42 added 2 commits August 26, 2025 15:50

Update on "[dynamo, nested graph breaks] add nested graph break tests"

842c520

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]

Update on "[dynamo, nested graph breaks] add nested graph break tests"

1e9764f

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 26, 2025

pytorchmergebot added the merging label Aug 27, 2025

pytorchmergebot closed this in 8b78ba0 Aug 27, 2025

pytorchmergebot removed the merging label Aug 27, 2025

github-actions bot deleted the gh/williamwen42/196/head branch September 27, 2025 02:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[dynamo, nested graph breaks] add nested graph break tests #144516

[dynamo, nested graph breaks] add nested graph break tests #144516

Uh oh!

williamwen42 commented Jan 10, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Jan 10, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Mar 11, 2025

Uh oh!

atalman commented Aug 26, 2025

Uh oh!

pytorchmergebot commented Aug 26, 2025

Uh oh!

pytorchmergebot commented Aug 26, 2025

Uh oh!

williamwen42 commented Aug 26, 2025

Uh oh!

atalman commented Aug 27, 2025

Uh oh!

pytorchmergebot commented Aug 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[dynamo, nested graph breaks] add nested graph break tests #144516

[dynamo, nested graph breaks] add nested graph break tests #144516

Uh oh!

Conversation

williamwen42 commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/144516

❌ 1 New Failure, 3 Pending, 1 Unrelated Failure

Uh oh!

github-actions bot commented Mar 11, 2025

Uh oh!

atalman commented Aug 26, 2025

Uh oh!

pytorchmergebot commented Aug 26, 2025

Uh oh!

pytorchmergebot commented Aug 26, 2025

Uh oh!

williamwen42 commented Aug 26, 2025

Uh oh!

atalman commented Aug 27, 2025

Uh oh!

pytorchmergebot commented Aug 27, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

williamwen42 commented Jan 10, 2025 •

edited

Loading

pytorch-bot bot commented Jan 10, 2025 •

edited

Loading