Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FSDP] Add keep_low_precision_grads support when CPU offloading #86495

Closed
wants to merge 3 commits into from
Closed

[FSDP] Add keep_low_precision_grads support when CPU offloading #86495

wants to merge 3 commits into from

Conversation

awgu
Copy link
Contributor

@awgu awgu commented Oct 7, 2022 •
edited

Stack from ghstack:

When CPU offloading, FSDP uses _cpu_grad, not _saved_grad_shard. This adds support for keep_low_precision_grads for that case.

@pytorch-bot
Copy link

pytorch-bot bot commented Oct 7, 2022 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86495

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures, 5 Pending

As of commit 5fcc277:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added release notes: distributed (fsdp) release notes category labels Oct 7, 2022
awgu added a commit that referenced this pull request Oct 7, 2022
@awgu
[FSDP] Add keep_low_precision_grads support when CPU offloading
30e2f84 
ghstack-source-id: 7f7cd5b25a32452a396adeb434ae58d31b579812
Pull Request resolved: #86495
@awgu
Update on "[FSDP] Add keep_low_precision_grads support when CPU off…
e8a28b8 
…loading"


When CPU offloading, FSDP uses `_cpu_grad`, not `_saved_grad_shard`. This adds support for `keep_low_precision_grads` for that case.

[ghstack-poisoned]
awgu added a commit that referenced this pull request Oct 7, 2022
@awgu
[FSDP] Add keep_low_precision_grads support when CPU offloading
f0a24d5 
ghstack-source-id: 5211bbd049442bc53d7a3bd2db846b294627db72
Pull Request resolved: #86495
@rohan-varma rohan-varma self-requested a review October 7, 2022 23:24
rohan-varma
rohan-varma approved these changes Oct 7, 2022
View reviewed changes
Copy link
Member

@rohan-varma rohan-varma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 7, 2022
@awgu
Update on "[FSDP] Add keep_low_precision_grads support when CPU off…
5fcc277 
…loading"


When CPU offloading, FSDP uses `_cpu_grad`, not `_saved_grad_shard`. This adds support for `keep_low_precision_grads` for that case.

[ghstack-poisoned]
@awgu awgu mentioned this pull request Oct 8, 2022
Closed
@awgu
Copy link
Contributor Author

awgu commented Oct 8, 2022

@pytorchbot merge

@awgu awgu mentioned this pull request Oct 8, 2022
Closed
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions
Copy link

github-actions bot commented Oct 8, 2022

Hey @awgu.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

facebook-github-bot pushed a commit that referenced this pull request Oct 10, 2022
@awgu @facebook-github-bot
[FSDP] Add keep_low_precision_grads support when CPU offloading (#8…
7031e30 
…6495) (#86495)

Summary:
When CPU offloading, FSDP uses `_cpu_grad`, not `_saved_grad_shard`. This adds support for `keep_low_precision_grads` for that case.

Pull Request resolved: #86495
Approved by: https://github.com/rohan-varma

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/af9c6bc851cfc8fba9e4c71830b783cb34d92a05

Reviewed By: seemethere

Differential Revision: D40217972

Pulled By: seemethere

fbshipit-source-id: 4d5b595d9246dba4237bea7ca6c27b8cebf1beff
@facebook-github-bot facebook-github-bot deleted the gh/awgu/115/head branch June 8, 2023 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rohan-varma rohan-varma rohan-varma approved these changes

@mrshenli mrshenli Awaiting requested review from mrshenli

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@H-Huang H-Huang Awaiting requested review from H-Huang

@kwen2501 kwen2501 Awaiting requested review from kwen2501

@mingzhe09088 mingzhe09088 Awaiting requested review from mingzhe09088

Assignees
No one assigned
Labels
ciflow/trunk Trigger trunk jobs on your pull request Merged release notes: distributed (fsdp) release notes category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@awgu @pytorchmergebot @rohan-varma