Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
MPI_THREAD_MULTIPLE support broken #157
Support for MPI_THREAD_MULTIPLE is currently broken on trunk. Even very simple programs that just do an MPI_Init_thread with MPI_THREAD_MULTIPLE and then call MPI_Barrier() will hang.
Example of a backtrace when --mpi-preconnect_mpi 1 is passed to mpirun
Running without MPI_THREAD_MULTIPLE but against the same build works fine.
I think the problem is due to some changes between 1.6 and trunk in opal/threads/condition.h
Now in 1.6 OPAL_ENABLE_PROGRESS_THREADS is hardcoded by configure to be
In trunk OPAL_ENABLE_MULTI_THREADS is set to 1 if mpi threads are
So in trunk when MPI_THREAD_MULTIPLE is requrest in init, the
Between 1.6 and trunk OPAL_ENABLE_PROGRESS_THREADS seems to have
If I change a few of those OPAL_ENABLE_MULTI_THREADS to
Trac comment by cyeoh on 2012-08-15 09:46:08:
After further investigation it turns out that with --enable-event-thread-support AND --enable-orte-progress-threads (which is marked experimental) then the orte progress thread handles servicing the event thread and as a result the OOB messages establishing the ib connection succeeds.
However the test program still hangs as there is no thread to ensure progress for the btl openib. Normally btl_openib_component_progress is called as a callback by opal_progress, but with the main thread calling opal_condition_wait instead this never occurs.
Presumably the currently unfinished OMPI_ENABLE_PROGRESS_THREADS is eventually meant to fix this. However, in the meantime my suggestion (I'll attach a patch) is to reintroduce OPAL_ENABLE_PROGRESS_THREADS for opal/threads/condition.h and use it where OPAL_ENABLE_MULTI_THREADS currently selects between either using pthread_cond_wait or calling opal_progress in a loop and looking for the signal variable to change in the condition object.
This is strictly not needed for the orte components when --enable-orte-progress-threads is enabled, but there is no clean way to enable it just for uses by orte but not for ompi as its a compile time property.
Trac comment by rhc on 2012-08-16 00:16:23:
Replying to [comment:1 cyeoh]:
We already agreed in the last developer meeting that we shouldn't be forcibly turning off ompi_enable_progress_threads - that was done solely because we were failing all the threaded tests and nobody was doing anything about it. If the code is finally reaching the point of working, or at least people are willing to work on it, then by all means remove that line in configure.ac.
Trac comment by cyeoh on 2012-08-16 01:03:36:
If enabling --enable-mpi-thread-multiple isn't going to work (and it currently doesn't) when using MPI_THREAD_MULTIPLE unless event thread support and the orte progress thread are also enabled, then should we be getting configure to automatically enable those too if enable-mpi-thread-multiple is requested?
I think what is making things a bit difficult between 1.6 and trunk is that the opal thread implementation is automatically turned on when MPI thread multiple support is enabled. In 1.6 OPAL_ENABLE_MULTI_THREAD was turned on with mpi thread multiple support, but it didn't actually control anything - you had to turn on OPAL_ENABLE_PROGRESS_THREADS which was off by default.
So in 1.6 we could run with MPI_THREAD_MULTIPLE even though OMPI_ENABLE_PROGRESS_THREADS (which is incomplete) was disabled.
In trunk now we have to complete OMPI_ENABLE_PROGRESS_THREAD (which doesn't currently even compile) support before any threaded MPI program will work because otherwise there is nothing to progress the openib btl. Previously because the low level opal thread support was also disabled there was no assumption that there was a separate progress thread for the btls and rather than call pthread_cond_wait, opal_condition_wat would call opal_progress.
So I'm wondering if the current state is intentional - eg force the situation to get someone to complete OMPI_ENABLE_PROGRESS_THREAD support if the want to run with MPI_THREAD_MULTIPLE?
Trac comment by rhc on 2012-08-16 15:05:09:
Okay, to make things easier, I added a couple of checks in configuring so that we error out if orte-progress-threads are enabled without libevent thread support, and if ompi-progress-threads are enabled without orte-progress-threads.
I also checked and found that only one place didn't compile when OMPI progress threads are enabled. There was a variable ompi_progress_thread_count in ompi/request/req_wait.c that wasn't defined. It is only used in that file, so I statically defined it there and initialized it to zero. I have no idea if that's right, or if that will make things work - but at least everything now compiles.
We definitely want to at least -encourage- people to complete the progress thread code. We need async progress for MPI-3, which is a goal for the 1.7 series. So putting some pressure on would be good.
Hope that helps the debugging!
pushed a commit
Mar 5, 2015
Hello, what is the status of the bug? Debian has switched all their architectures to OpenMPI by default now, but the fact that OpenMPI has broken MPI_THREAD_MULTIPLE probably means it should switch back to MPICH?
The current version of OpenMPI on Debian is 1.10, do we have to assume that this bug is still open there?