Need MPI_THREAD_MULTIPLE for backward compatibility

[eschnett]

It seems I need either MPI_THREAD_MULTIPLE or MPI_THREAD_SERIALIZED for backward compatibility, to gradually add HPX support to Cactus. I am using the MPI parcelport, and I tried simply calling some MPI functions. This seems to work many times, but I get strange hangs that could be explained by OpenMPI not being thread safe.

[hkaiser]

What do you mean by that? Do we need to add a preprocessor constant, do we need to initialize MPI in a particular way, or what else do you want us to do?

[eschnett]

By default, if MPI is initialized via MPI_Init, MPI routines can be called only from one of the threads. The alternative routine MPI_Init_thread accepts several "mode" selectors, including MPI_THREAD_MULTIPLE and MPI_THREAD_SERIALIZED.

MPI_THREAD_MULTIPLE allows all threads to make MPI calls. Unfortunately, e.g. OpenMPI does not seem to fully support this yet.

MPI_THREAD_SERIALIZED also allows all threads to make MPI calls, but requires that the application ensures that the calls are serialized, i.e. that only one thread at a time makes MPI calls. Using this would probably require (maybe global) state changes between "HPX uses MPI" and "application uses MPI".

(I just see that MPI_THREAD_SINGLE is the default, which states that the applications is single-threaded. With HPX, you probably should use at least MPI_THREAD_FUNNELED instead.)

[hkaiser]

Thanks for the explanations. HPX guarantees that the MPI calls are being made from exactly one thread only, thus MPI_THREAD_FUNNELED seems to be the right flag for now. I'll go ahead and make that change to the code base.

If you need to do MPI calls yourself then I don't see any way to synchronize HPX with an external application without creating a global contention point (as MPI_THREAD_MULTIPLE is not a viable option). Any suggestions?

[eschnett]

You could introduce a function that switches HPX between use-MPI and don't-use-MPI mode. This function could be local to a locality. Of course, performance would not be ideal.

Since Cactus has defined times at which it performs MPI calls, I would then wrap these calls by first disabling and later re-enabling MPI in HPX.

[sithhell]

You could introduce a function that switches HPX between use-MPI and
don't-use-MPI mode. This function could be local to a locality. Of course,
performance would not be ideal.

Since Cactus has defined times at which it performs MPI calls, I would
then wrap these calls by first disabling and later re-enabling MPI in HPX.

That's absolutely possible. Is suggest to introduce something that pauses
the parcel handling completely.

[hkaiser]

For now I'm going to commit the following change. I switched from MPI_Init to MPI_Init_thread. HPX passes either MPI_THREAD_MULTIPLE or MPI_THREAD_SINGLE to this function, where the default is MPI_THREAD_SINGLE (the docs for MPI_THREAD_FUNNELLED explicitly say that all MPI calls are done from the main thread, which is not the case in HPX).

The mode used with MPI_Init_thread can be controlled by the configuration setting hpx.parcel.mpi.multithreaded (0 is the default and forces MPI_THREAD_SINGLE, anything != 0 forces MPI_THREAD_MULTIPLE). As usual this setting can be pre-initialized using the environment variable HPX_PARCELPORT_MPI_MULTITHREADED.