Skip to content

Commit

Permalink
Update base for Update on "scatter_object_list API for c10d"
Browse files Browse the repository at this point in the history
Closes #23232. As part of addressing #23232, this PR adds support for scatter_object_list which is an API to scatter arbitrary picklable objects to all the other ranks.

The implementation approach follows a similar approach as #42189. The result of the `scatter` is stored as the first element of `scatter_object_output_list`, and the src rank is expected to provide an input list `scatter_object_input_list` which contains the objects to scatter.

Note that this API requires 1 broadcast and 2 scatters. This is because we must communicate the maximum object size to be scattered, which only the src rank knows about. After that, we also need to communicate the objects themselves as well as the true sizes of the object.

Note that the API is designed to match the tensor-based collectives other than supporting async_op. For now, it is a blocking call. If we see demand to support async_op, we will have to make more progress on merging work/future to support this.

It only works for Gloo because NCCL doesn't support scatter.

Differential Revision: [D23430686](https://our.internmc.facebook.com/intern/diff/D23430686/)

**NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D23430686/)!

[ghstack-poisoned]
  • Loading branch information
rohan-varma committed Dec 4, 2020
2 parents b7aa999 + 0e4f9a7 commit 22d28da
Showing 0 changed files with 0 additions and 0 deletions.

0 comments on commit 22d28da

Please sign in to comment.