Update API GUIDE to include multi host training and add some colors #4706
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JackCaoG
commented
Feb 28, 2023
JackCaoG
force-pushed
the
JackCaoG/update_API_GUIDE
branch
from
4f328bf to
d56a9fa
Compare
cowanmeg
reviewed
Feb 28, 2023
vanbasten23
reviewed
Mar 1, 2023
vanbasten23
reviewed
Mar 1, 2023
will-cromar
reviewed
Mar 1, 2023
API_GUIDE.md
Outdated
| ### Running on Multiple XLA Hosts | ||
| Multi-host setup for different accelerators can be very different. This doc will talk about the device independent bits of multi-host training and will use the TPU + PJRT runtime(currently available on 1.13 and 2.x releases) as an example. | ||
|
|
||
| Let's assume you have the above mnist example from above section in a `train_mnist_xla.py`. If it is a single host multi device training, you would run it like |
There was a problem hiding this comment.
I think first you should show a smoke test, e.g. just dumping the real device IDs from each host with python -c before copying around a script.
There was a problem hiding this comment.
ok, I can add that
There was a problem hiding this comment.
I am having some trouble creating a pod to play with these command. For now I will leave this doc as "How it should work" and direct user to the pod user guide.. The major problem now is that user guide still uses XRT.
JackCaoG
force-pushed
the
JackCaoG/update_API_GUIDE
branch
from
50922fe to
21fef5f
Compare
vanbasten23
approved these changes
Mar 4, 2023
JackCaoG
added a commit
that referenced
this pull request
Mar 10, 2023
…4706) * Update API GUIDE to include multi host training and add some colors * address review comments
JackCaoG
added a commit
that referenced
this pull request
Mar 10, 2023
* Update API GUIDE to include multi host training and add some colors (#4706) * Update API GUIDE to include multi host training and add some colors * address review comments * Update README (#4734) * Update README * update user guide section title * Add public readme for torchdynamo (#4744) * Add public readme for torchdynamo * Update index file
mateuszlewko
pushed a commit
that referenced
this pull request
Mar 15, 2023
…4706) * Update API GUIDE to include multi host training and add some colors * address review comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
My goal is to make
API_GUIDEto be a good entry point for any firs time pytorch/xla user. We can put more technical details on different docs, butAPI_GUIDEshould include a big picture.rendered version in https://github.com/pytorch/xla/blob/JackCaoG/update_API_GUIDE/API_GUIDE.md#running-on-multiple-xla-devices-with-multi-processing.