Use MPI_Isend/MPI_Irecv to back send/recv #11630
Conversation
pietern
added
the
oncall: distributed
|
Both the Python tests and the MPI tests in torch/lib/c10d/test pass locally. |
torch/lib/c10d/ProcessGroupMPI.cpp
Outdated
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
torch/lib/c10d/ProcessGroupMPI.cpp
Outdated
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
torch/lib/c10d/ProcessGroupMPI.cpp
Outdated
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
The isCompleted function is changed to being non-const to accomodate setting some internal status on the work object in the case of completion. Previously, it was only checking a member field, but for the MPI backend it calls MPI_Test to poll for completion of an asynchronous request.
pietern
force-pushed
the
c10d-mpi-isend-irecv
branch
from
7092879 to
de50a59
Compare
|
Thanks. Addressed comments. |
There was a problem hiding this comment.
pietern has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
Flaky test for CPU only distributed tests where it should skip is a KP. |
| std::array<char, MPI_MAX_ERROR_STRING> buf; | ||
| int len = buf.size(); | ||
| MPI_CHECK(MPI_Error_string(status_.MPI_ERROR, buf.data(), &len)); | ||
| return std::runtime_error(std::string(buf.data(), len)); |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
5 participants
The isCompleted function is changed to being non-const to accomodate
setting some internal status on the work object in the case of
completion. Previously, it was only checking a member field, but for the
MPI backend it calls MPI_Test to poll for completion of an asynchronous
request.