Re-structure for use in multi-language scenarios

[AndrewGaspar]

rsmpi does not currently support multi-language MPI programs very well. This issue is tracking the work needed to make it more embed-able.

Outstanding Issues:

• Expose mpi::is_initialized
• Expose UserCommunicator::from_raw
• Support for external initialization of MPI

Original Comment

A scenario for rsmpi, which I imagine would be common, is to use it inside a static lib that is linked into a larger application. This larger application is likely written in another language like Fortran, C, or C++. The larger application is probably already initializing MPI with its preferred parameters. In this case, the Rust library could would not "own" the MPI initialization, and therefore would not call mpi::initialize*. However, it may still want to check if MPI has already been initialized, even if it is just to panic and tell the user that MPI needs to be initialized first.

Of course, ffi::MPI_Initialized is available, but it seems like making the Rust-native version of it available is pretty low-cost.

[bsteinb]

Yeah, when I started on rsmpi I structured it in a way that is more geared toward using it from a rust application and having rsmpi be the only interface through which MPI is used. I later realized that writing a rust library which can then be used from HPC applications written in the more traditional languages is also a likely scenario.

However, there will have to be some restructuring to get rsmpi into shape for that use case. This issue touches on one aspect of it.

The Universe type will have to take a different role, maybe the initialized environment that it stands for will become fully implicit and not have value representation just as it is in the official MPI bindings. There could still be something like a FinalizeGuard in cases where the user wants something that calls MPI_Finalize on drop.

Types like ...Communicator will need a from_raw constructor which check certain attributes of the underlying objects (intra-communicator vs. inter-communicator or persistent vs. non-persistent request).

Then of course objects like communicators that are passed in from another language could have different error handlers set. Right now, rsmpi assumes that communication operations crash on error, but that is not necessarily the case, so, in general, everything has to return Result just like (almost) everything returns int in the C bindings.

Issue #12 is also somewhat related to this.

If you don't mind, I will hijack this issue and make the title reflect the somewhat broader scope. It could then become a tracking issue for a few smaller issues that cover the different areas that need to be touched.

[AndrewGaspar]

Ah, I hadn't considered all of the potential issues here. If you do find a way to add Send + Sync traits to MPI objects statically depending on while MPI threading setting used, you may need an unsafe mechanism for choosing at runtime whether Universe is safe to use in a multi-threaded environment. The caller would be expected to know how MPI was initialized in their context.

[jedbrown]

I'm also interested in this mode of use. I'll note that a best practice for MPI libraries is for object constructors to all take an MPI_Comm. Implicitly using MPI_COMM_WORLD is generally discouraged. To simplify user code, PETSc has a PetscInitialize that processes run-time options and sets up a profiling session. If MPI has not been initialized, PETSc initializes it and takes responsibility for MPI_Finalize. This is a pretty common pattern and seems like not much work to support.

It looks like rsmpi likes to use RSMPI_ANY_TAG, but this is often problematic for a nontrivial class of algorithms. It's common for libraries to implement "tag dispensers" to prevent different components from interfering with each other. (And MPI_Comm_dup at the boundary with other libraries.) In PETSc, these are implemented using attributes so we don't have to dup again each time an MPI_Comm gets passed along a call stack that goes back and forth between PETSc and other code. All this to say that attributes will end up being useful for multi-language scenarios.