-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Fix FSDP mixed precision semantics and add user warning #21361
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix FSDP mixed precision semantics and add user warning #21361
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #21361 +/- ##
=======================================
- Coverage 87% 87% -0%
=======================================
Files 269 269
Lines 23799 23791 -8
=======================================
- Hits 20618 20610 -8
Misses 3181 3181 |
|
doc failures are irrelevant and is due to `sphinx.errors.SphinxWarning: failed to reach any of the inventories with the following issues: Warning, treated as error: seems something is down again. |
What does this PR do?
Fixes #<issue_number>
Align FSDP "-mixed" precision modes with their "-true" counterparts, since parameters remain in full precision. Adds a warning for all reduced-precision modes indicating experimental status.
Before submitting
PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:
Reviewer checklist
📚 Documentation preview 📚: https://pytorch-lightning--21361.org.readthedocs.build/en/21361/