New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Docs] Add examples for new object-based c10d APIs #43932
Conversation
Adds some basic examples to the documentation for each of the newly added object-based collectibves. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
Adds some basic examples to the documentation for each of the newly added object-based collectibves. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
💊 CI failures summary and remediationsAs of commit 926d912 (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group. This comment has been revised 44 times. |
Adds some basic examples to the documentation for each of the newly added object-based collectives. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
Pull Request resolved: #43932 Adds some basic examples to the documentation for each of the newly added object-based collectibves. ghstack-source-id: 111118006 Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/)
Adds some basic examples to the documentation for each of the newly added object-based collectives. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
Pull Request resolved: #43932 Adds some basic examples to the documentation for each of the newly added object-based collectibves. ghstack-source-id: 111180437 Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/)
Adds some basic examples to the documentation for each of the newly added object-based collectives. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
Pull Request resolved: #43932 Adds some basic examples to the documentation for each of the newly added object-based collectibves. ghstack-source-id: 111211263 Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/)
Adds some basic examples to the documentation for each of the newly added object-based collectives. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
Pull Request resolved: #43932 Adds some basic examples to the documentation for each of the newly added object-based collectibves. ghstack-source-id: 111268988 Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/)
Adds some basic examples to the documentation for each of the newly added object-based collectives. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
Pull Request resolved: #43932 Adds some basic examples to the documentation for each of the newly added object-based collectibves. ghstack-source-id: 111466860 Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/)
Adds some basic examples to the documentation for each of the newly added object-based collectives. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
Pull Request resolved: #43932 Adds some basic examples to the documentation for each of the newly added object-based collectibves. ghstack-source-id: 117637342 Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/)
Adds some basic examples to the documentation for each of the newly added object-based collectives. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
Pull Request resolved: #43932 Adds some basic examples to the documentation for each of the newly added object-based collectibves. ghstack-source-id: 117750674 Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/)
Codecov Report
@@ Coverage Diff @@
## gh/rohan-varma/162/base #43932 +/- ##
============================================================
+ Coverage 68.49% 80.76% +12.27%
============================================================
Files 413 1867 +1454
Lines 54478 201584 +147106
============================================================
+ Hits 37312 162814 +125502
- Misses 17166 38770 +21604 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks for adding this! Left some minor comments mainly about rendering and clarity.
>>> output = [None for _ in gather_objects] | ||
>>> dist.all_gather_object(output, gather_objects[dist.get_rank()]) | ||
>>> output | ||
['foo', 12, {1: 2}] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shall we verify if this line is rendered correctly?
gather_objects[dist.get_rank()], | ||
output if dist.get_rank() == 0 else None, | ||
dst=0 | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shall we also verify if these lines are rendered correctly?
>>> else: | ||
>>> objects = [None, None, None] | ||
>>> dist.broadcast_object_list(objects, src=0) | ||
>>> broadcast_objects |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this var is not defined?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, it should be just objects
.
Example:: | ||
>>> # Note: Process group initialization omitted on each rank. | ||
>>> import torch.distributed as dist | ||
>>> gather_objects = ["foo", 12, {1: 2}] # any picklable object |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this assumes world_size==3
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, I guess we could just add this as a comment to keep the example clear and concise.
>>> if dist.get_rank() == 0: | ||
>>> objects = ["foo", 12, {1: 2}] # any picklable object | ||
>>> else: | ||
>>> objects = [None, None, None] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does this have to be 3 Nones?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's mostly to point out that whatever is contained in objects
on non-src ranks, it doesn't matter, we will only look at the objects
from the src rank. The 2nd arg can be anything for non-src ranks, it is not considered. Will add a comment indicating so.
Adds some basic examples to the documentation for each of the newly added object-based collectives. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
Pull Request resolved: #43932 Adds some basic examples to the documentation for each of the newly added object-based collectibves. ghstack-source-id: 117956466 Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/)
Adds some basic examples to the documentation for each of the newly added object-based collectives. Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/) [ghstack-poisoned]
Pull Request resolved: #43932 Adds some basic examples to the documentation for each of the newly added object-based collectibves. ghstack-source-id: 117965966 Differential Revision: [D23441838](https://our.internmc.facebook.com/intern/diff/D23441838/)
@mrshenli Looks like the rendering is as expected after building the docs locally, see for example: |
This pull request has been merged in b77ca9e. |
Stack from ghstack:
Adds some basic examples to the documentation for each of the newly added
object-based collectives.
Differential Revision: D23441838