Fix distributed documentation for asynchronous collective Work objects #45709
Conversation
facebook-github-bot
added
the
oncall: distributed
Codecov Report
@@ Coverage Diff @@
## master #45709 +/- ##
=======================================
Coverage 68.25% 68.25%
=======================================
Files 410 410
Lines 53246 53246
=======================================
+ Hits 36343 36344 +1
+ Misses 16903 16902 -1
Continue to review full report at Codecov.
|
docs/source/distributed.rst
Outdated
| return anything | ||
|
|
||
| asynchronous operation - when ``async_op`` is set to True. The collective operation function | ||
| Every collective operation function supports the following two kinds of operations, depending on the setting of the ``async_op`` flag passed into the collective: |
There was a problem hiding this comment.
let's break this into shorter lines
docs/source/distributed.rst
Outdated
| further function calls utilizing the output of the collective call will behave as expected. For CUDA collectives, | ||
| function calls utilizing the output on the same CUDA stream will behave as expected. Users must take care of | ||
| synchronization under the scenario of running under different streams. For details on CUDA semantics such as stream | ||
| synchronization, see `cuda semantics <https://pytorch.org/docs/stable/autograd.html#profiler>`__. |
There was a problem hiding this comment.
cuda semantics points to profiler, is this intentional?
There was a problem hiding this comment.
Thanks for the catch! It should actually point to https://pytorch.org/docs/stable/notes/cuda.html
There was a problem hiding this comment.
@rohan-varma has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@rohan-varma merged this pull request in 154347d. |
Closes #42247. Clarifies some documentation related to
Workobject semantics (outputs of async collective functions). Clarifies the difference between CPU operations and CUDA operations (on Gloo or NCCL backend), and provides an example where the difference in CUDA operation's wait() semantics is necessary to understand for correct code.