[Traceable FSDP2][Animesh's PR #125340] [dynamo][fsdp] Track FSDPNNModuleVariable for mutations #129045

yf225 · 2024-06-19T07:18:40Z

This is a copy of Animesh's work in #125340, with very small changes to the unit test. It's needed sooner for the Traceable FSDP2 work, so I copy it here and will work through landing it.

Stack from ghstack (oldest at bottom):

-> [Traceable FSDP2][Animesh's PR #125340] [dynamo][fsdp] Track FSDPNNModuleVariable for mutations #129045

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @chauhang @d4l3k @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng

[ghstack-poisoned]

pytorch-bot · 2024-06-19T07:18:43Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129045

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (13 Unrelated Failures)

As of commit 1ed17c1 with merge base 3a18577 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
'ul'

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128903)
ImportError: attempted relative import with no known parent package
inductor / cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128903)
ImportError: attempted relative import with no known parent package
inductor / cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128902)
ImportError: attempted relative import with no known parent package
inductor / cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128902)
ImportError: attempted relative import with no known parent package
inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128901)
ImportError: attempted relative import with no known parent package
inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128901)
ImportError: attempted relative import with no known parent package
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128929)
ImportError: attempted relative import with no known parent package
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128929)
ImportError: attempted relative import with no known parent package
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128931)
ImportError: attempted relative import with no known parent package
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128931)
ImportError: attempted relative import with no known parent package
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128932)
ImportError: attempted relative import with no known parent package
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128932)
ImportError: attempted relative import with no known parent package

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…ariable's param/buffer/submodule is mutated ghstack-source-id: 304e0000f1df32eea04b0de1319e9e6459b62227 Pull Request resolved: #129045

anijain2305

I think we should do this - #125340

Feel free to copy and open a PR if urgent. I can otherwise try to land it by Friday.

[ghstack-poisoned]

anijain2305 · 2024-06-19T23:51:37Z

test/distributed/test_dynamo_distributed.py

@@ -484,6 +516,17 @@ def test_fsdp_aot_eager(self):
            outputs = fsdp_m(inputs)
            self.assertTrue(same(correct_outputs, outputs))

+    @skip_if_lt_x_gpu(1)
+    @unittest.skipIf(not has_triton(), "Inductor+gpu needs triton and recent GPU arch")


We can xfail the test also, if nopython=True does not work today.

Or mabe remove nopython=True and add a comment that we will work on it later.

[ghstack-poisoned]

ghstack-source-id: 75d0ce85eef6411fa890b37a7c5768440b98aa2e Pull Request resolved: #129045

yf225 · 2024-06-20T04:00:49Z

@pytorchbot merge -f "unrelated failures"

pytorchmergebot · 2024-06-20T04:02:28Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Update

ac89143

[ghstack-poisoned]

yf225 added a commit that referenced this pull request Jun 19, 2024

[Traceable FSDP2][Dynamo] Track side effect if UnspecializedNNModuleV…

beaae9e

…ariable's param/buffer/submodule is mutated ghstack-source-id: 304e0000f1df32eea04b0de1319e9e6459b62227 Pull Request resolved: #129045

pytorch-bot bot added ciflow/inductor module: dynamo labels Jun 19, 2024

yf225 changed the title ~~[Traceable FSDP2][Dynamo] Track side effect if UnspecializedNNModuleVariable's param/buffer/submodule is mutated~~ [Traceable FSDP2] [Dynamo] Track side effect if UnspecializedNNModuleVariable's param/buffer/submodule is mutated Jun 19, 2024

yf225 requested a review from anijain2305 June 19, 2024 07:19

anijain2305 requested changes Jun 19, 2024

View reviewed changes

yf225 changed the title ~~[Traceable FSDP2] [Dynamo] Track side effect if UnspecializedNNModuleVariable's param/buffer/submodule is mutated~~ [Animesh's PR #125340] [dynamo][fsdp] Track FSDPNNModuleVariable for mutations Jun 19, 2024

Update

fc37a4b

[ghstack-poisoned]

pytorch-bot bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Jun 19, 2024

yf225 requested a review from anijain2305 June 19, 2024 23:42

anijain2305 reviewed Jun 19, 2024

View reviewed changes

anijain2305 approved these changes Jun 19, 2024

View reviewed changes

Update

7d0084a

[ghstack-poisoned]

Update

e79602f

[ghstack-poisoned]

Update

24f72c4

[ghstack-poisoned]

Update

1ed17c1

[ghstack-poisoned]

yf225 added a commit that referenced this pull request Jun 20, 2024

[dynamo][fsdp] Ensure that fsdp module can support __setattr__

aad5cfd

ghstack-source-id: 75d0ce85eef6411fa890b37a7c5768440b98aa2e Pull Request resolved: #129045

anijain2305 mentioned this pull request Jun 20, 2024

[dynamo][fsdp] Track FSDPNNModuleVariable for mutations #125340

Closed

pytorchmergebot added the merging label Jun 20, 2024

pytorchmergebot closed this in ad2593c Jun 20, 2024

pytorchmergebot added Merged and removed merging labels Jun 20, 2024

yf225 changed the title ~~[Animesh's PR #125340] [dynamo][fsdp] Track FSDPNNModuleVariable for mutations~~ [Traceable FSDP2][Animesh's PR #125340] [dynamo][fsdp] Track FSDPNNModuleVariable for mutations Jun 27, 2024

github-actions bot deleted the gh/yf225/48/head branch July 28, 2024 02:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Traceable FSDP2][Animesh's PR #125340] [dynamo][fsdp] Track FSDPNNModuleVariable for mutations #129045

[Traceable FSDP2][Animesh's PR #125340] [dynamo][fsdp] Track FSDPNNModuleVariable for mutations #129045

yf225 commented Jun 19, 2024 •

edited

Loading

pytorch-bot bot commented Jun 19, 2024 •

edited

Loading

anijain2305 left a comment

anijain2305 Jun 19, 2024

anijain2305 Jun 19, 2024

yf225 commented Jun 20, 2024

pytorchmergebot commented Jun 20, 2024

[Traceable FSDP2][Animesh's PR #125340] [dynamo][fsdp] Track FSDPNNModuleVariable for mutations #129045

[Traceable FSDP2][Animesh's PR #125340] [dynamo][fsdp] Track FSDPNNModuleVariable for mutations #129045

Conversation

yf225 commented Jun 19, 2024 • edited Loading

pytorch-bot bot commented Jun 19, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129045

✅ You can merge normally! (13 Unrelated Failures)

anijain2305 left a comment

Choose a reason for hiding this comment

anijain2305 Jun 19, 2024

Choose a reason for hiding this comment

anijain2305 Jun 19, 2024

Choose a reason for hiding this comment

yf225 commented Jun 20, 2024

pytorchmergebot commented Jun 20, 2024

Merge started

yf225 commented Jun 19, 2024 •

edited

Loading

pytorch-bot bot commented Jun 19, 2024 •

edited

Loading