MPI Failing to Bind Process to hwthread #7368
Description
Background information
Trying to run nccl-test with MPI and --bind-to hwthread
What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)
Ran mpiexec --version:
mpiexec (OpenRTE) 2.1.1
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
I installed OpenMPI using apt-get install, followed the instructions for quick install here: http://lsi.ugr.es/jmantas/pdp/ayuda/datos/instalaciones/Install_OpenMPI_en.pdf
Please describe the system on which you are running
- Operating system/version: Running Docker container w/ https://gitlab.com/nvidia/container-images/cuda/blob/master/dist/ubuntu18.04/10.2/base/Dockerfile
- Computer hardware: Nvidia DGX1
Details of the problem
--------------------------------------------------------------------------
WARNING: Open MPI tried to bind a process but failed. This is a
warning only; your job will continue, though performance may
be degraded.
Local host: caa484f9e63a
Application name: nccl-tests/build/all_reduce_perf
Error message: failed to bind memory
Location: rtc_hwloc.c:255
Ran the following command:
/usr/bin/mpirun --allow-run-as-root --bind-to hwthread -np 1 nccl-tests/build/all_reduce_perf -b 8 -e 2G -f 2 -g 1
Exact same setup worked on my local machine, running Ubuntu w/ a GTX 1070