-
Notifications
You must be signed in to change notification settings - Fork 21.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FSDP] Add fast path for NO_SHARD
clip_grad_norm_()
#89137
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/89137
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit ab9f9de: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
ghstack-source-id: e5ad011e74f0dd5cf7a1e3d96db35682705aa2e1 Pull Request resolved: #89137
ghstack-source-id: e5ad011e74f0dd5cf7a1e3d96db35682705aa2e1 Pull Request resolved: pytorch#89137
[ghstack-poisoned]
ghstack-source-id: df801495fc5517d7420e151ad8a5807627e1f5cc Pull Request resolved: #89137
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks for tracking the follow up issues around improved testing!
# TODO: Gradient computation on CPU and GPU differ slightly causing | ||
# drift unrelated to `clip_grad_norm_()`. | ||
# https://github.com/pytorch/pytorch/issues/89133 | ||
return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A bit hacky, but would it help if we just ran DDP on CPU for this case if having this coverage is important?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does DDP work on CPU with NCCL backend?
ghstack-source-id: df801495fc5517d7420e151ad8a5807627e1f5cc Pull Request resolved: pytorch#89137
[ghstack-poisoned]
ghstack-source-id: 0dd6ff61ca05986e32893b082e76f4341f1af61e Pull Request resolved: #89137
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Pull Request resolved: pytorch#89137 Approved by: https://github.com/rohan-varma
Stack from ghstack:
NO_SHARD
clip_grad_norm_()
#89137 [FSDP] Add fast path forNO_SHARD
clip_grad_norm_()