[DCP][state_dict] DCP state_dict cannot correctly find FQN when the leaf module is wrapped by FSDP #115592

fegin · 2023-12-11T21:39:53Z

Summary: The original logic has an incorrect assumption that there is at one object name left when traversing the module tree. This is not correct when the leaf module is wrapped by FSDP.

Test Plan: CI

Differential Revision: D52049293

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225

pytorch-bot · 2023-12-11T21:39:57Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/115592

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit db2d941 with merge base c70f995 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

periodic / macos-12-py3-x86-64 / build (gh)

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

periodic / linux-focal-rocm5.7-py3.8 / test (distributed, 1, 2, linux.rocm.gpu, unstable) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2023-12-11T21:40:04Z

This pull request was exported from Phabricator. Differential Revision: D52049293

wz337 · 2023-12-11T23:40:11Z

test/distributed/checkpoint/test_state_dict.py

@@ -181,7 +191,7 @@ def test_fsdp(self) -> None:
            {
                "use_orig_params": [True, False],
                "use_composable": [True, False],
-                "use_dtensor": [True, False],
+                "use_dtensor": [True,False],


This seems unintentional?

…eaf module is wrapped by FSDP (pytorch#115592) Summary: The original logic has an incorrect assumption that there is at one object name left when traversing the module tree. This is not correct when the leaf module is wrapped by FSDP. Test Plan: CI Differential Revision: D52049293

facebook-github-bot · 2023-12-11T23:50:54Z

This pull request was exported from Phabricator. Differential Revision: D52049293

…eaf module is wrapped by FSDP (pytorch#115592) Summary: The original logic has an incorrect assumption that there is at one object name left when traversing the module tree. This is not correct when the leaf module is wrapped by FSDP. Test Plan: CI Differential Revision: D52049293

facebook-github-bot · 2023-12-11T23:51:37Z

This pull request was exported from Phabricator. Differential Revision: D52049293

…eaf module is wrapped by FSDP (pytorch#115592) Summary: The original logic has an incorrect assumption that there is at one object name left when traversing the module tree. This is not correct when the leaf module is wrapped by FSDP. Test Plan: CI Differential Revision: D52049293

facebook-github-bot · 2023-12-11T23:52:24Z

This pull request was exported from Phabricator. Differential Revision: D52049293

facebook-github-bot · 2023-12-11T23:53:17Z

This pull request was exported from Phabricator. Differential Revision: D52049293

…eaf module is wrapped by FSDP (pytorch#115592) Summary: The original logic has an incorrect assumption that there is at one object name left when traversing the module tree. This is not correct when the leaf module is wrapped by FSDP. Test Plan: CI Differential Revision: D52049293

wz337

LGTM!

…eaf module is wrapped by FSDP (pytorch#115592) Summary: The original logic has an incorrect assumption that there is at one object name left when traversing the module tree. This is not correct when the leaf module is wrapped by FSDP. Test Plan: CI Reviewed By: wz337 Differential Revision: D52049293

facebook-github-bot · 2023-12-12T03:14:15Z

This pull request was exported from Phabricator. Differential Revision: D52049293

…eaf module is wrapped by FSDP (pytorch#115592) Summary: The original logic has an incorrect assumption that there is at one object name left when traversing the module tree. This is not correct when the leaf module is wrapped by FSDP. Test Plan: CI Reviewed By: wz337 Differential Revision: D52049293

facebook-github-bot · 2023-12-12T03:15:02Z

This pull request was exported from Phabricator. Differential Revision: D52049293

facebook-github-bot · 2023-12-12T19:20:23Z

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

pytorchmergebot · 2023-12-12T19:22:15Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…eaf module is wrapped by FSDP (pytorch#115592) Summary: The original logic has an incorrect assumption that there is at one object name left when traversing the module tree. This is not correct when the leaf module is wrapped by FSDP. Test Plan: CI Differential Revision: D52049293 Pull Request resolved: pytorch#115592 Approved by: https://github.com/wz337

#119691) Co-authored-by: Chien-Chin Huang <chienchin@fb.com> resolved: #115592

facebook-github-bot added the fb-exported label Dec 11, 2023

github-actions bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Dec 11, 2023

fegin added the topic: not user facing topic category label Dec 11, 2023

fegin requested review from wz337 and LucasLLC December 11, 2023 22:30

fegin added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 11, 2023

wz337 reviewed Dec 11, 2023

View reviewed changes

fegin force-pushed the export-D52049293 branch from 5ebd7b8 to ee40a3b Compare December 11, 2023 23:50

fegin force-pushed the export-D52049293 branch from ee40a3b to 4b3ee0c Compare December 11, 2023 23:51

fegin force-pushed the export-D52049293 branch from 4b3ee0c to 72e3d30 Compare December 11, 2023 23:52

fegin requested a review from wz337 December 11, 2023 23:52

fegin force-pushed the export-D52049293 branch from 72e3d30 to 4187e7b Compare December 11, 2023 23:53

wz337 approved these changes Dec 12, 2023

View reviewed changes

fegin force-pushed the export-D52049293 branch from 4187e7b to 53c4d33 Compare December 12, 2023 03:14

fegin force-pushed the export-D52049293 branch from 53c4d33 to db2d941 Compare December 12, 2023 03:14

fegin added the ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR label Dec 12, 2023

pytorchmergebot added the merging label Dec 12, 2023

pytorchmergebot added the Merged label Dec 12, 2023

pytorchmergebot closed this in d954ef2 Dec 12, 2023

pytorchmergebot removed the merging label Dec 12, 2023

atalman added this to the 2.2.1 milestone Jan 17, 2024

mvpatel2000 mentioned this pull request Feb 6, 2024

[v2.2.1] Release Tracker #119295

Closed

Skylion007 mentioned this pull request Feb 12, 2024

[DCP][state_dict] DCP state_dict cannot correctly find FQN when the l… #119691

Merged

atalman pushed a commit that referenced this pull request Feb 14, 2024

[DCP][state_dict] DCP state_dict cannot correctly find FQN when the l… (

445b0f9

#119691) Co-authored-by: Chien-Chin Huang <chienchin@fb.com> resolved: #115592

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DCP][state_dict] DCP state_dict cannot correctly find FQN when the leaf module is wrapped by FSDP #115592

[DCP][state_dict] DCP state_dict cannot correctly find FQN when the leaf module is wrapped by FSDP #115592

fegin commented Dec 11, 2023 •

edited by pytorch-bot bot

pytorch-bot bot commented Dec 11, 2023 •

edited

facebook-github-bot commented Dec 11, 2023

wz337 Dec 11, 2023

facebook-github-bot commented Dec 11, 2023

facebook-github-bot commented Dec 11, 2023

facebook-github-bot commented Dec 11, 2023

facebook-github-bot commented Dec 11, 2023

wz337 left a comment

facebook-github-bot commented Dec 12, 2023

facebook-github-bot commented Dec 12, 2023

facebook-github-bot commented Dec 12, 2023

pytorchmergebot commented Dec 12, 2023

[DCP][state_dict] DCP state_dict cannot correctly find FQN when the leaf module is wrapped by FSDP #115592

[DCP][state_dict] DCP state_dict cannot correctly find FQN when the leaf module is wrapped by FSDP #115592

Conversation

fegin commented Dec 11, 2023 • edited by pytorch-bot bot

pytorch-bot bot commented Dec 11, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/115592

✅ You can merge normally! (2 Unrelated Failures)

facebook-github-bot commented Dec 11, 2023

wz337 Dec 11, 2023

Choose a reason for hiding this comment

facebook-github-bot commented Dec 11, 2023

facebook-github-bot commented Dec 11, 2023

facebook-github-bot commented Dec 11, 2023

facebook-github-bot commented Dec 11, 2023

wz337 left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Dec 12, 2023

facebook-github-bot commented Dec 12, 2023

facebook-github-bot commented Dec 12, 2023

pytorchmergebot commented Dec 12, 2023

Merge started

fegin commented Dec 11, 2023 •

edited by pytorch-bot bot

pytorch-bot bot commented Dec 11, 2023 •

edited