Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add privateuse1 in FSDP's sharded grad scaler #126971

Closed
wants to merge 1 commit into from
Closed

Add privateuse1 in FSDP's sharded grad scaler #126971

wants to merge 1 commit into from

Conversation

accelerate321
Copy link
Contributor

@accelerate321 accelerate321 commented May 23, 2024 •
edited by pytorch-bot bot
Loading

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented May 23, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126971

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7097927 with merge base f0366de (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (sharded) release notes category labels May 23, 2024
@bdhirsh bdhirsh requested review from awgu and weifengpy May 23, 2024 22:26
@bdhirsh bdhirsh added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label May 23, 2024
awgu
awgu approved these changes May 23, 2024
View reviewed changes
Copy link
Contributor

@awgu awgu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM!

@accelerate321
Copy link
Contributor Author

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 24, 2024
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@accelerate321 accelerate321 deleted the privateuse1_for_sharded_grad_scaler branch May 24, 2024 05:57
titaiwangms pushed a commit to titaiwangms/pytorch that referenced this pull request May 28, 2024
@accelerate321 @titaiwangms
Add privateuse1 in FSDP's sharded grad scaler (pytorch#126971)
415b28f 
1. add privateuse1 in FSDP's sharded grad scaler
2. support found_inf copy for more devices

Pull Request resolved: pytorch#126971
Approved by: https://github.com/awgu, https://github.com/weifengpy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@awgu awgu awgu approved these changes

@weifengpy weifengpy weifengpy approved these changes

Assignees
No one assigned
Labels
ciflow/trunk Trigger trunk jobs on your pull request Merged oncall: distributed Add this issue/PR to distributed oncall triage queue open source release notes: distributed (sharded) release notes category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@accelerate321 @pytorchmergebot @awgu @weifengpy @bdhirsh @pytorchbot