Add flag to disable MPI threads - HOROVOD_MPI_THREADS_DISABLE #324
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jiezhang
approved these changes
Jun 20, 2018
horovod/common/operations.cc
Outdated
| // be used together with Horovod if multi-threaded MPI is installed. | ||
| auto mpi_threads_disable = std::getenv("HOROVOD_MPI_THREADS_DISABLE"); | ||
| int required; |
There was a problem hiding this comment.
This seems cleaner.
int required = MPI_THREAD_MULTIPLE;
if (mpi_threads_disable != nullptr && std::atoi(mpi_threads_disable) > 0) {
required = MPI_THREAD_FUNNELED;
}
jeevandev
approved these changes
Jun 20, 2018
horovod/common/operations.cc
Outdated
| // be used together with Horovod if multi-threaded MPI is installed. | ||
| auto mpi_threads_disable = std::getenv("HOROVOD_MPI_THREADS_DISABLE"); | ||
| int required; |
sblotner
approved these changes
Jun 20, 2018
docs/running.md
Outdated
| @@ -12,7 +12,9 @@ allows you to have a mixture of different NUMA configurations because the defaul | |||
| `-mca pml ob1` and `-mca btl ^openib` flags force the use of TCP for MPI communication. This avoids many multiprocessing | |||
| issues that Open MPI has with RDMA which typically result in segmentation faults. Using TCP for MPI does not have | |||
| noticeable performance impact since most of the heavy communication is done by NCCL, which will use RDMA via RoCE or | |||
| InfiniBand if they're available (see [Horovod on GPU](gpus.md)). | |||
| InfiniBand if they're available (see [Horovod on GPU](gpus.md)). Notable exception from this rule are models that heavily | |||
There was a problem hiding this comment.
exception --> exceptions
docs/running.md
Outdated
| @@ -12,7 +12,9 @@ allows you to have a mixture of different NUMA configurations because the defaul | |||
| `-mca pml ob1` and `-mca btl ^openib` flags force the use of TCP for MPI communication. This avoids many multiprocessing | |||
| issues that Open MPI has with RDMA which typically result in segmentation faults. Using TCP for MPI does not have | |||
| noticeable performance impact since most of the heavy communication is done by NCCL, which will use RDMA via RoCE or | |||
| InfiniBand if they're available (see [Horovod on GPU](gpus.md)). | |||
| InfiniBand if they're available (see [Horovod on GPU](gpus.md)). Notable exception from this rule are models that heavily | |||
| use `hvd.broadcast()` and `hvd.allgather()` operations. To make those operations use RDMA read [Open MPI with RDMA](#open-mpi-with-rdma) | |||
There was a problem hiding this comment.
"RDMA read [Open MPI...]" --> "RDMA, read the [Open MPI...]"
docs/running.md
Outdated
| @@ -39,6 +41,25 @@ $ mpirun -np 16 \ | |||
| python train.py | |||
| ``` | |||
|
|
|||
| ### Open MPI with RDMA | |||
|
|
|||
| As noted above, using TCP for MPI communication does not have any significant affects on performance in the majority of cases. | |||
docs/running.md
Outdated
| Models that make heavy use of `hvd.broadcast()` and `hvd.allgather()` operations are exceptions to that rule. | ||
|
|
||
| Default Open MPI `openib` BTL that provides RDMA functionality does not work well with MPI multi-threading. In order to use | ||
| RDMA with `openib`, multi-threading must be disabled via `-x HOROVOD_MPI_THREADS_DISABLE=1` option. See example below: |
There was a problem hiding this comment.
See example --> See the example
docs/running.md
Outdated
| python train.py | ||
| ``` | ||
|
|
||
| Other MPI RDMA implementations may or may not benefit from disabling multi-threading, please consult vendor documentation. |
There was a problem hiding this comment.
multi-threading, please --> multithreading, so please
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.