Add flag to disable MPI threads - HOROVOD_MPI_THREADS_DISABLE

docs/running.md

@@ -12,7 +12,9 @@ allows you to have a mixture of different NUMA configurations because the defaul
 `-mca pml ob1` and `-mca btl ^openib` flags force the use of TCP for MPI communication.  This avoids many multiprocessing
 issues that Open MPI has with RDMA which typically result in segmentation faults.  Using TCP for MPI does not have
 noticeable performance impact since most of the heavy communication is done by NCCL, which will use RDMA via RoCE or
-InfiniBand if they're available (see [Horovod on GPU](gpus.md)).
+InfiniBand if they're available (see [Horovod on GPU](gpus.md)).  Notable exceptions from this rule are models that heavily
+use `hvd.broadcast()` and `hvd.allgather()` operations.  To make those operations use RDMA, read the [Open MPI with RDMA](#open-mpi-with-rdma)
+section below.
 
 With the `-x` option you can specify (`-x NCCL_DEBUG=INFO`) or copy (`-x LD_LIBRARY_PATH`) an environment variable to all
 the workers.
@@ -39,6 +41,25 @@ $ mpirun -np 16 \
     python train.py
 ```
 
+### Open MPI with RDMA
+
+As noted above, using TCP for MPI communication does not have any significant effects on performance in the majority of cases.
+Models that make heavy use of `hvd.broadcast()` and `hvd.allgather()` operations are exceptions to that rule.
+
+Default Open MPI `openib` BTL that provides RDMA functionality does not work well with MPI multithreading.  In order to use
+RDMA with `openib`, multithreading must be disabled via `-x HOROVOD_MPI_THREADS_DISABLE=1` option.  See the example below:
+
+```bash
+$ mpirun -np 16 \
+    -H server1:4,server2:4,server3:4,server4:4 \
+    -bind-to none -map-by slot \
+    -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x HOROVOD_MPI_THREADS_DISABLE=1 -x PATH \
+    -mca pml ob1 \
+    python train.py
+```
+
+Other MPI RDMA implementations may or may not benefit from disabling multithreading, so please consult vendor documentation.
+
 ### Hangs due to SSH issues
 
 The host where `mpirun` is executed must be able to SSH to all other hosts without any prompts.

horovod/common/operations.cc

@@ -1179,10 +1179,19 @@ void BackgroundThreadLoop(HorovodGlobalState& state) {
   // Initialize MPI. This must happen on the background thread, since not all
   // MPI implementations support being called from multiple threads.
   //
-  // We will ask for multiple threads, so other libraries like mpi4py can
+  // In some cases MPI library has multi-threading support, but it slows down certain
+  // components, e.g. OpenIB BTL in OpenMPI gets disabled if MPI_THREAD_MULTIPLE is
+  // requested.
+  //
+  // By default, we will ask for multiple threads, so other libraries like mpi4py can
   // be used together with Horovod if multi-threaded MPI is installed.
+  auto mpi_threads_disable = std::getenv("HOROVOD_MPI_THREADS_DISABLE");
+  int required = MPI_THREAD_MULTIPLE;
+  if (mpi_threads_disable != nullptr && std::atoi(mpi_threads_disable) > 0) {
+    required = MPI_THREAD_FUNNELED;
+  }
   int provided;
-  MPI_Init_thread(NULL, NULL, MPI_THREAD_MULTIPLE, &provided);
+  MPI_Init_thread(NULL, NULL, required, &provided);
 
   // Create a private MPI communicator for Horovod to avoid collisions with
   // other threads using MPI.