[dynamo] skip frame recursively when no graph is traced #144712

williamwen42 · 2025-01-13T22:21:21Z

Stack from ghstack (oldest at bottom):

-> [dynamo] skip frame recursively when no graph is traced #144712

Fixes #144360. cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @ydwu4

I'm considering refactoring the code flag logic in eval_frame (i.e. SKIP_CODE, SKIP_CODE_RECURSIVE, cache_limit_hit_fag, skip_frame_recursive_flag) to make things better defined and to have a cleaner, more consistent implementation.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames

[ghstack-poisoned]

pytorch-bot · 2025-01-13T22:21:24Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/144712

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 37 New Failures, 3 Unrelated Failures

As of commit 0732b6d with merge base a94ec0a ():

NEW FAILURES - The following jobs have failed:

inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
YituTechConvBert
inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
mixer_b16_224
inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
xcit_large_24_p8_224
inductor / unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_cpp_wrapper, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
moco
Lint / lintrunner-noclang / linux-job (gh)
RuntimeError: Command docker exec -t ac18edae90bfd296cf512e1decfef82be4e9702e14cceceb44556e006555328e /exec failed with exit code 1
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 2, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
dynamo/test_recompiles.py::RecompileTests::test_no_recursive_compile_after_cache_limit_hit
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 3, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
dynamo/test_higher_order_ops.py::HigherOrderOpVmapGuardTests::test_linearize_recompiles
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 4, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesFunctionTests::test_truth_dynamic_shapes
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 5, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
dynamo/test_ctx_manager.py::ContextlibContextManagerTests::test_graph_break_after___enter__
pull / linux-focal-cuda12.4-py3.10-gcc9-sm89 / test (default, 3, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
dynamo/test_higher_order_ops.py::HigherOrderOpVmapGuardTests::test_linearize_recompiles
pull / linux-focal-cuda12.4-py3.10-gcc9-sm89 / test (default, 4, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesFunctionTests::test_truth_dynamic_shapes
pull / linux-focal-cuda12.4-py3.10-gcc9-sm89 / test (default, 5, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
dynamo/test_modules.py::OptimizedModuleTest::test_recompile_limit_on_freed_module
pull / linux-focal-py3_9-clang9-xla / test (xla, 1, 1, lf.linux.12xlarge) (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/get-workflow-job-id'. Did you forget to run actions/checkout before running your local action?
pull / linux-focal-py3.13-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
dynamo/test_functions.py::FunctionTests::test_truth
pull / linux-focal-py3.13-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
dynamo/test_higher_order_ops.py::HigherOrderOpVmapGuardTests::test_linearize_recompiles
pull / linux-focal-py3.13-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesFunctionTests::test_truth_dynamic_shapes
pull / linux-focal-py3.13-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
dynamo/test_recompiles.py::RecompileTests::test_no_recursive_compile_after_cache_limit_hit
pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 1, 3, lf.linux.2xlarge) (gh)
test_fx.py::TestFX::test_symbolic_trace_assert
pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 2, 3, lf.linux.2xlarge) (gh)
test_autograd.py::TestAutograd::test_checkpointing_without_reentrant_saved_object_identity
pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 3, 3, lf.linux.2xlarge) (gh)
test_optim.py::TestOptimRenewedCPU::test_foreach_matches_forloop_AdamW_cpu_float64
pull / linux-focal-py3.9-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
dynamo/test_higher_order_ops.py::HigherOrderOpVmapGuardTests::test_linearize_recompiles
pull / linux-focal-py3.9-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesFunctionTests::test_truth_dynamic_shapes
pull / linux-focal-py3.9-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
dynamo/test_ctx_manager.py::ContextlibContextManagerTests::test_graph_break_after___enter__
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 1, 3, lf.linux.2xlarge) (gh)
test_fx.py::TestFX::test_symbolic_trace_assert
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 2, 3, lf.linux.2xlarge) (gh)
test_optim.py::TestOptimRenewedCPU::test_foreach_matches_forloop_AdamW_cpu_float64
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 3, 3, lf.linux.2xlarge) (gh)
test_jit.py::TestModuleContainers::test_module_properties
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, lf.linux.4xlarge) (gh)
dynamo/test_recompiles.py::RecompileTests::test_no_recursive_compile_after_cache_limit_hit
pull / linux-jammy-py3.10-clang15-asan / test (default, 3, 6, lf.linux.4xlarge) (gh)
dynamo/test_higher_order_ops.py::HigherOrderOpVmapGuardTests::test_linearize_recompiles
pull / linux-jammy-py3.10-clang15-asan / test (default, 4, 6, lf.linux.4xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesFunctionTests::test_truth_dynamic_shapes
pull / linux-jammy-py3.10-clang15-asan / test (default, 5, 6, lf.linux.4xlarge) (gh)
dynamo/test_ctx_manager.py::ContextlibContextManagerTests::test_graph_break_after___enter__
pull / linux-jammy-py3.10-clang15-asan / test (default, 6, 6, lf.linux.4xlarge) (gh)
dynamo/test_functions.py::FunctionTests::test_truth
pull / linux-jammy-py3.9-gcc11 / test (default, 1, 5, lf.linux.2xlarge) (gh)
dynamo/test_functions.py::FunctionTests::test_truth
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, lf.linux.2xlarge) (gh)
dynamo/test_ctx_manager.py::ContextlibContextManagerTests::test_graph_break_after___enter__
pull / linux-jammy-py3.9-gcc11 / test (default, 3, 5, lf.linux.2xlarge) (gh)
dynamo/test_higher_order_ops.py::HigherOrderOpVmapGuardTests::test_linearize_recompiles
pull / linux-jammy-py3.9-gcc11 / test (default, 4, 5, lf.linux.2xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesFunctionTests::test_truth_dynamic_shapes
pull / linux-jammy-py3.9-gcc11 / test (default, 5, 5, lf.linux.2xlarge) (gh)
dynamo/test_recompiles.py::RecompileTests::test_no_recursive_compile_after_cache_limit_hit
pull / linux-jammy-py3.9-gcc11 / test (numpy_2_x, 1, 1, lf.linux.2xlarge) (gh)
dynamo/test_functions.py::FunctionTests::test_truth

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
Process completed with exit code 1.
inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
Process completed with exit code 1.

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

pull / linux-jammy-py3-clang12-executorch / test (executorch, 1, 1, lf.linux.2xlarge, unstable) (gh) (#144480)
backends/xnnpack/test/ops/test_conv1d.py::TestConv1d::test_qs8_conv1d_batchnorm_seq

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: cec7638 Pull Request resolved: #144712

Temporary fix to #144360. Turning the config on globally will cause a bunch of tests to fail, which needs to be addressed in followups. I had a previous attempt at #144712, but this is a more complicated change and will likely be absorbed into work to refactor Dynamo's exception handling. Pull Request resolved: #144885 Approved by: https://github.com/jansel

github-actions · 2025-03-16T20:34:38Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

williamwen42 · 2025-03-17T20:57:17Z

Can be re-implemented with eval frame code actions.

[dynamo] skip frame recursively when no graph is traced

0732b6d

[ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: dynamo labels Jan 13, 2025

williamwen42 added a commit that referenced this pull request Jan 13, 2025

[dynamo] skip frame recursively when no graph is traced

f0a686f

ghstack-source-id: cec7638 Pull Request resolved: #144712

williamwen42 added the topic: not user facing topic category label Jan 13, 2025

williamwen42 marked this pull request as draft January 15, 2025 19:51

williamwen42 mentioned this pull request Jan 15, 2025

[dynamo] add option to not skip on empty graph #144885

Closed

github-actions bot added the Stale label Mar 16, 2025

williamwen42 closed this Mar 17, 2025

github-actions bot deleted the gh/williamwen42/197/head branch April 20, 2025 02:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[dynamo] skip frame recursively when no graph is traced #144712

[dynamo] skip frame recursively when no graph is traced #144712

Uh oh!

williamwen42 commented Jan 13, 2025 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Jan 13, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Mar 16, 2025

Uh oh!

williamwen42 commented Mar 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

[dynamo] skip frame recursively when no graph is traced #144712

[dynamo] skip frame recursively when no graph is traced #144712

Uh oh!

Conversation

williamwen42 commented Jan 13, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/144712

❌ 37 New Failures, 3 Unrelated Failures

Uh oh!

github-actions bot commented Mar 16, 2025

Uh oh!

williamwen42 commented Mar 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

williamwen42 commented Jan 13, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jan 13, 2025 •

edited

Loading