MPI thread for active messages

[shiozaki]

(1) Figure out why MPI_Init_thread does not work properly with OpenMPI
(2) Optimize sleeptime__ (util/constants.h) with realistic calculations

[shiozaki]

MPI_Init_thread works just fine with MVAPITCH2 2.0a.

[jeffhammond]

OpenMPI does not properly support MPI_THREAD_MULTIPLE. It is hit or miss whether or not the BTL is thread-safe and thus one cannot use this mode with OpenMPI in a portable way (see recent discussion on their user list for details). The thread support in MPICH and its derivatives (MVAPICH, Intel MPI, CrayMPI, Blue Gene MPI, ...) is much better.

Depending on your use case, busy-polling or even continuous intermittent polling may not be necessary. In some cases, you may be able to leverage the behavior of the eager protocol in MPI_Send to achieve what you want. Some tuning may be required to make this work optimally.

[shiozaki]

Good to know the status of OpenMPI.

This is our (minimum-feature) implementation of active messages, in which buffer flush has to be done in the background. Currently there are many hard-wired buffer flush calls here and there, but it is cleaner to have a dedicated thread to manage it. The code is mainly used in parallel FCI that is based Robert Harrison's algorithm published in somewhere.

[shiozaki]

OK there is no much we can do with this. I close this issue with a label "won't-fix".