-
Notifications
You must be signed in to change notification settings - Fork 21.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PT-D][Tensor parallelism] Add documentations for TP #94421
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/94421
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 8d99683: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
ghstack-source-id: 06caff66222400a86a803e30ca9d09afe24a6aba Pull Request resolved: #94421
[ghstack-poisoned]
ghstack-source-id: f105a1ade4ccac6ea9a8337d34e1830d12099132 Pull Request resolved: #94421
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
first pass, let's add the experimental line as this is prototype release.
:members: | ||
|
||
We also enabled 2D parallelism to integrate with ``FullyShardedDataParallel``. | ||
Users just need to call the following API explicitly: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I remembered we have a FSDP extension, Is TP automatically register the extension now?
Also, I wonder if we should give a small code snippet showing how the 2-D parallel look like
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The registrations is in the is_available
. Let me send a follow-up PR for this one.
|
||
|
||
.. currentmodule:: torch.distributed.tensor.parallel.fsdp | ||
.. autofunction:: is_available |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we really need to add this API to the doc? I remembered is_available is introduced when we are in tau, but since now it's pytorch I think fsdp should always be available?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, because of 2D hook registration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will send a follow-up PR to address the naming of this one.
This is far from completed and we will definitely polish it down the road. [ghstack-poisoned]
ghstack-source-id: 8fa4b414901367acdd0ac76b0844a1eb59ee5c67 Pull Request resolved: #94421
@pytorchbot rebase |
@pytorchbot successfully started a rebase job. Check the current status here |
This is far from completed and we will definitely polish it down the road. [ghstack-poisoned]
Successfully rebased |
ghstack-source-id: d03f0b1bf33d5f3f662d1f574a35828bbc336330 Pull Request resolved: #94421
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Stack from ghstack (oldest at bottom):
This is far from completed and we will definitely polish it down the road.