What is the exact value of MPI_Comm for processes under the same communicator.

[AtlantaPepsi]

So I see that type MPI_Comm is essentially just int, but what is the exact value of communicator variables returned by functions like, say, MPI_Cart_create?

I was thinking if I could just make communicator a static variable of a class where all objects are under the same communicator. I tried printing out the values of communicator for all 4 processes under the same 2x2 Cartesian communicator, turns out 3 of them had same value and one different from the others.

I tried to probe the source code to see the implementation, I'm not familiar with the C binding process, but seems that it ultimately calls cartCreate() declared here, whose definition I couldn't find.

Alternatively I also see it in the mpich-pip repo, the value of newly created Communicator is assigned here, but variable newcomm_ptr which is of type MPIR_Comm. But MPIR_Comm, defined here does not even have a member named handle.

It's pretty obvious I'm very lost here, any advice on how to navigate the MPICH source code would greatly help!

[hzhou]

You can think MPI_Comm as a pointer to the internal communicator structure MPIR_Comm. It is an integer because we use an internal object memory allocation system, but it is equivalent to a pointer. Each process does not share the address space, thus you should not expect the handle to have the same value for each process. They can coincidentally share the same value depending on the state of the internal memory allocation, but it is purely coincidental. The same handle can refer to objects with no relationship.

The handle is defined by the macro MPIR_OBJECT_HEADER.

[AtlantaPepsi]

Thank you for the clarifications. Inferring from this, the MPIX_Stream objects are also pointer to MPIR_Stream created by implementation routines in MPIR layer, right? Is there then a duplication function for MPI Streams like MPI_Comm_dup for communicators?
I remember years ago I've seen a wiki-like page explaining the different MPI prefix (like -U, -R) and layers in MPICH, does it still exist or is it completely replaced by github wiki? I couldn't find a similar section in github wiki.

[hzhou]

Yes, MPIX_Stream is also handle to internal MPIR_Stream objects. There isn't a dup function for MPIX Streams. Unlike communicator, streams are local objects, so you can just pass the handle around (within your process).

Every thing internal to MPICH is name spaced either with MPIR_ or MPID. MPID refers to lower device layer. Everything above device layer are usually in the MPIR namespace.

Our main developer documentation is still wiki.mpich.org. It's a bit ad-hoc. There doesn't seem to be a general namespace guide, but here is one for ch4-device layer: https://github.com/pmodels/mpich/blob/main/doc/wiki/notes/ch4_namespace.md

[AtlantaPepsi]

I think the wiki link before led to an actual wiki-ish page instead of the github wiki, what I saw was something like in this question, but very similar to the CH4 namespace page, must be modifications after CH4 i suppose