[FSDP] Add set_state_dict_type API to setup state_dict_type without using context manager #86243

fegin · 2022-10-04T23:12:34Z

Stack from ghstack (oldest at bottom):

-> [FSDP] Add set_state_dict_type API to setup state_dict_type without using context manager #86243

FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. set_state_dict_type allows users to do so.

Differential Revision: D40083670

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu

…sing context manager FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/) [ghstack-poisoned]

pytorch-bot · 2022-10-04T23:12:36Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86243

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 Failures

As of commit 4de33ef:

The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…sing context manager FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/) ghstack-source-id: 169371556 Pull Request resolved: #86243

rohan-varma

LGTM, just looks like there is a valid lint. Thanks for adding this!

rohan-varma · 2022-10-04T23:55:03Z

torch/distributed/fsdp/fully_sharded_data_parallel.py

+
+        prev_state_dict_type = None
+        prev_state_dict_config = None
+        # Use default config a state_dict config is not set.


"if a state_dict config is not set"

…e without using context manager" FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/) [ghstack-poisoned]

…sing context manager Pull Request resolved: #86243 FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. ghstack-source-id: 169453751 Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/)

…e without using context manager" FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/) [ghstack-poisoned]

…sing context manager Pull Request resolved: #86243 FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. ghstack-source-id: 170308364 Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/)

awgu · 2022-10-13T21:35:50Z

torch/distributed/fsdp/fully_sharded_data_parallel.py

@@ -2111,6 +2104,46 @@ def _get_training_state(
        )
        return next(iter(training_states))

+    @staticmethod
+    def set_state_dict_type(


Should we add a docstring?

…e without using context manager" FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/) [ghstack-poisoned]

…sing context manager Pull Request resolved: #86243 FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. ghstack-source-id: 170670792 Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/)

…e without using context manager" FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/) [ghstack-poisoned]

…sing context manager Pull Request resolved: #86243 FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. ghstack-source-id: 170765562 Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/)

fegin · 2022-10-18T22:34:21Z

@pytorchbot merge

pytorchmergebot · 2022-10-18T22:35:44Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2022-10-18T22:35:48Z

Merge failed

Reason: The following mandatory check(s) failed (Rule Distributed):

pull

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

…e without using context manager" FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/) [ghstack-poisoned]

…sing context manager Pull Request resolved: #86243 FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. ghstack-source-id: 170829792 Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/)

…e without using context manager" FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/) [ghstack-poisoned]

…sing context manager Pull Request resolved: #86243 FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. ghstack-source-id: 170860829 Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/)

…e without using context manager" FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/) [ghstack-poisoned]

…sing context manager Pull Request resolved: #86243 FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. ghstack-source-id: 170899517 Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/)

fegin · 2022-10-19T17:47:05Z

@pytorchbot merge

pytorchmergebot · 2022-10-19T17:48:33Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2022-10-19T18:18:55Z

Merge failed

Reason: The following mandatory check(s) failed (Rule Distributed):

pull

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

fegin · 2022-10-19T19:44:56Z

@pytorchbot merge -f "failing tests are not related"

pytorchmergebot · 2022-10-19T19:46:14Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

github-actions · 2022-10-19T19:46:59Z

Hey @fegin.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

…sing context manager (pytorch#86243) FSDP.state_dict_type is a context manager. However, users may want to decide what state_dict is going to used during initialization. `set_state_dict_type` allows users to do so. Differential Revision: [D40083670](https://our.internmc.facebook.com/intern/diff/D40083670/) Pull Request resolved: pytorch#86243 Approved by: https://github.com/rohan-varma

fegin requested review from mrshenli, zhaojuanmao, pritamdamania87, rohan-varma, mingzhe09088, H-Huang, awgu and kwen2501 as code owners October 4, 2022 23:12

pytorch-bot bot added the release notes: distributed (sharded) release notes category label Oct 4, 2022

facebook-github-bot added the cla signed label Oct 4, 2022

fegin mentioned this pull request Oct 4, 2022

[FSDP] Consolidate FSDP state_dict offload_to_cpu settings #86211

Closed

facebook-github-bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Oct 4, 2022

rohan-varma approved these changes Oct 4, 2022

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 4, 2022

fegin mentioned this pull request Oct 13, 2022

[FSDP] Introduce optim_state_dict() and load_optim_state_dict() #86912

Closed

awgu reviewed Oct 13, 2022

View reviewed changes

pytorchmergebot added the Merged label Oct 19, 2022

pytorchmergebot closed this in c08c799 Oct 19, 2022

facebook-github-bot deleted the gh/fegin/31/head branch June 8, 2023 17:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FSDP] Add set_state_dict_type API to setup state_dict_type without using context manager #86243

[FSDP] Add set_state_dict_type API to setup state_dict_type without using context manager #86243

fegin commented Oct 4, 2022 •

edited by pytorch-bot bot

pytorch-bot bot commented Oct 4, 2022 •

edited

rohan-varma left a comment

rohan-varma Oct 4, 2022

awgu Oct 13, 2022

fegin commented Oct 18, 2022

pytorchmergebot commented Oct 18, 2022

pytorchmergebot commented Oct 18, 2022

fegin commented Oct 19, 2022

pytorchmergebot commented Oct 19, 2022

pytorchmergebot commented Oct 19, 2022

fegin commented Oct 19, 2022

pytorchmergebot commented Oct 19, 2022

github-actions bot commented Oct 19, 2022

[FSDP] Add set_state_dict_type API to setup state_dict_type without using context manager #86243

[FSDP] Add set_state_dict_type API to setup state_dict_type without using context manager #86243

Conversation

fegin commented Oct 4, 2022 • edited by pytorch-bot bot

pytorch-bot bot commented Oct 4, 2022 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86243

❌ 4 Failures

rohan-varma left a comment

Choose a reason for hiding this comment

rohan-varma Oct 4, 2022

Choose a reason for hiding this comment

awgu Oct 13, 2022

Choose a reason for hiding this comment

fegin commented Oct 18, 2022

pytorchmergebot commented Oct 18, 2022

Merge started

pytorchmergebot commented Oct 18, 2022

Merge failed

fegin commented Oct 19, 2022

pytorchmergebot commented Oct 19, 2022

Merge started

pytorchmergebot commented Oct 19, 2022

Merge failed

fegin commented Oct 19, 2022

pytorchmergebot commented Oct 19, 2022

Merge started

github-actions bot commented Oct 19, 2022

fegin commented Oct 4, 2022 •

edited by pytorch-bot bot

pytorch-bot bot commented Oct 4, 2022 •

edited