Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Dist] Enable FSDP on CPU #112145

Closed
wants to merge 3 commits into from
Closed

[Dist] Enable FSDP on CPU #112145

wants to merge 3 commits into from

Conversation

rohan-varma
Copy link
Member

@rohan-varma rohan-varma commented Oct 26, 2023 •
edited
Loading

@rohan-varma
[Dist] Enable FSDP on CPU
adc214a 
Differential Revision: [D50688958](https://our.internmc.facebook.com/intern/diff/D50688958/)

[ghstack-poisoned]
@pytorch-bot pytorch-bot bot added the release notes: distributed (fsdp) release notes category label Oct 26, 2023
@pytorch-bot
Copy link

pytorch-bot bot commented Oct 26, 2023 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/112145

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 71911f2 with merge base dd24e92 (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@github-actions github-actions bot added the module: cpu CPU specific problem (e.g., perf, algorithm) label Oct 26, 2023
This was referenced Oct 26, 2023
Closed
Closed
rohan-varma added a commit that referenced this pull request Oct 26, 2023
@rohan-varma
[Dist] Enable FSDP on CPU
2bb5e63 
Differential Revision: [D50688958](https://our.internmc.facebook.com/intern/diff/D50688958/)

ghstack-source-id: 205424565
Pull Request resolved: #112145
fegin
fegin approved these changes Oct 27, 2023
View reviewed changes
Copy link
Contributor

@fegin fegin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

wanchaol
wanchaol reviewed Oct 27, 2023
View reviewed changes
torch/cpu/__init__.py
pass


class Event:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a comment to this class since it's technically a public API:

    """
    N.B. This class only exists to facilitate device-agnostic code
    """

Maybe also need to add a doc entry for this

@XilunWu XilunWu self-requested a review October 31, 2023 21:20
@rohan-varma
Update on "[Dist] Enable FSDP on CPU"
4c56310 
Differential Revision: [D50688958](https://our.internmc.facebook.com/intern/diff/D50688958/)

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10

[ghstack-poisoned]
rohan-varma added a commit that referenced this pull request Nov 6, 2023
@rohan-varma
[Dist] Enable FSDP on CPU
1f86eb8 
Pull Request resolved: #112145


ghstack-source-id: 206626309
@exported-using-ghexport

Differential Revision: [D50688958](https://our.internmc.facebook.com/intern/diff/D50688958/)
@rohan-varma
Update on "[Dist] Enable FSDP on CPU"
71911f2 
Differential Revision: [D50688958](https://our.internmc.facebook.com/intern/diff/D50688958/)

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10

[ghstack-poisoned]
rohan-varma added a commit that referenced this pull request Nov 6, 2023
@rohan-varma
[Dist] Enable FSDP on CPU
9d0aa7d 
Pull Request resolved: #112145


ghstack-source-id: 206641812
@exported-using-ghexport

Differential Revision: [D50688958](https://our.internmc.facebook.com/intern/diff/D50688958/)
@rohan-varma
Copy link
Member Author

@pytorchbot merge -f "Ci failures are flaky"

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

xuhancn pushed a commit to xuhancn/pytorch that referenced this pull request Nov 7, 2023
@rohan-varma @xuhancn
[Dist] Enable FSDP on CPU (pytorch#112145)
a7d98ee 
Differential Revision: [D50688958](https://our.internmc.facebook.com/intern/diff/D50688958/)

Pull Request resolved: pytorch#112145
Approved by: https://github.com/fegin
ghstack dependencies: pytorch#112144
@facebook-github-bot facebook-github-bot deleted the gh/rohan-varma/745/head branch November 10, 2023 15:24
Skylion007 pushed a commit to Skylion007/pytorch that referenced this pull request Nov 14, 2023
@rohan-varma @Skylion007
[Dist] Enable FSDP on CPU (pytorch#112145)
5531b9c 
Differential Revision: [D50688958](https://our.internmc.facebook.com/intern/diff/D50688958/)

Pull Request resolved: pytorch#112145
Approved by: https://github.com/fegin
ghstack dependencies: pytorch#112144
@awgu
Copy link
Contributor

awgu commented Nov 14, 2023

@rohan-varma Is this CPU FSDP only for testing, or is it numerically correct?

@rohan-varma
Copy link
Member Author

@awgu I mostly landed it for testing and because some users were asking to use it in their testing use cases, but what do you mean by numerically correct? While we haven't done any correctness testing, none of the logic in FSDP is changed?

@awgu awgu mentioned this pull request Nov 22, 2023
Closed
awgu added a commit that referenced this pull request Nov 22, 2023
@awgu
Update base for Update on "[FSDP] Added DDP parity test for CPU train…
461c777 
…ing"


This is a follow-up to #112145 to include a numerical parity test with DDP for CPU training. 
```
python -m pytest test/distributed/fsdp/test_fsdp_misc.py -k test_fsdp_cpu_training -s
```

We should follow-up on https://github.com/pytorch/pytorch/pull/112145/files#r1375102283 at some point too.

[ghstack-poisoned]
awgu added a commit that referenced this pull request Nov 22, 2023
@awgu
Update on "[FSDP] Added DDP parity test for CPU training"
f1c9b4c 
This is a follow-up to #112145 to include a numerical parity test with DDP for CPU training. 
```
python -m pytest test/distributed/fsdp/test_fsdp_misc.py -k test_fsdp_cpu_training -s
```

We should follow-up on https://github.com/pytorch/pytorch/pull/112145/files#r1375102283 at some point too.

[ghstack-poisoned]
awgu added a commit that referenced this pull request Nov 22, 2023
@awgu
Update base for Update on "[FSDP] Added DDP parity test for CPU train…
3c68130 
…ing"


This is a follow-up to #112145 to include a numerical parity test with DDP for CPU training. 
```
python -m pytest test/distributed/fsdp/test_fsdp_misc.py -k test_fsdp_cpu_training -s
```

We should follow-up on https://github.com/pytorch/pytorch/pull/112145/files#r1375102283 at some point too.

cc H-Huang kwen2501 wanchaol fegin fduwjj wz337 wconstab mrshenli zhaojuanmao rohan-varma kiukchung d4l3k lucasllc XilunWu tianyu-l

[ghstack-poisoned]
awgu added a commit that referenced this pull request Nov 22, 2023
@awgu
Update on "[FSDP] Added DDP parity test for CPU training"
99e090b 
This is a follow-up to #112145 to include a numerical parity test with DDP for CPU training. 
```
python -m pytest test/distributed/fsdp/test_fsdp_misc.py -k test_fsdp_cpu_training -s
```

We should follow-up on https://github.com/pytorch/pytorch/pull/112145/files#r1375102283 at some point too.

cc H-Huang kwen2501 wanchaol fegin fduwjj wz337 wconstab mrshenli zhaojuanmao rohan-varma kiukchung d4l3k lucasllc XilunWu tianyu-l

[ghstack-poisoned]
pytorchmergebot pushed a commit that referenced this pull request Nov 22, 2023
@awgu @pytorchmergebot
[FSDP] Added DDP parity test for CPU training (#114372)
e7726b5 
This is a follow-up to #112145 to include a numerical parity test with DDP for CPU training.
```
python -m pytest test/distributed/fsdp/test_fsdp_misc.py -k test_fsdp_cpu_training -s
```

We should follow-up on https://github.com/pytorch/pytorch/pull/112145/files#r1375102283 at some point too.

Pull Request resolved: #114372
Approved by: https://github.com/XilunWu
xunsongh pushed a commit to xunsongh/pytorch that referenced this pull request Nov 24, 2023
@awgu @xunsongh
[FSDP] Added DDP parity test for CPU training (pytorch#114372)
8fb7db7 
This is a follow-up to pytorch#112145 to include a numerical parity test with DDP for CPU training.
```
python -m pytest test/distributed/fsdp/test_fsdp_misc.py -k test_fsdp_cpu_training -s
```

We should follow-up on https://github.com/pytorch/pytorch/pull/112145/files#r1375102283 at some point too.

Pull Request resolved: pytorch#114372
Approved by: https://github.com/XilunWu
@awgu awgu mentioned this pull request Dec 1, 2023
Open
@rohan-varma rohan-varma mentioned this pull request Jan 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wanchaol wanchaol wanchaol left review comments

@fegin fegin fegin approved these changes

@mrshenli mrshenli Awaiting requested review from mrshenli

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@H-Huang H-Huang Awaiting requested review from H-Huang

@awgu awgu Awaiting requested review from awgu

@kwen2501 kwen2501 Awaiting requested review from kwen2501

@fduwjj fduwjj Awaiting requested review from fduwjj

@wz337 wz337 Awaiting requested review from wz337

@XilunWu XilunWu Awaiting requested review from XilunWu

Assignees
No one assigned
Labels
Merged module: cpu CPU specific problem (e.g., perf, algorithm) release notes: distributed (fsdp) release notes category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@rohan-varma @pytorchmergebot @awgu @fegin @wanchaol