Big MPI---point-to-point considerations (MPI_Rank only)

[tonyskjellum]

Problem

For 64-bit clean functionality, convenience, and symmetry, the Big MPI principles being applied in Ticket #80 to collective operations should be applied to MPI more widely. In this case, we consider the idea that you might want more than 2^31 MPI ranks, hence needing a new data type, MPI_Rank.

Proposal

MPI needs to be 64-bit clean throughout.

Changes to the Text

MPI_Rank will replace int for ranks; support > 2^31 MPI processes in a communicator.

A separate ticket considers MPI_Count and miscellaneous concerns for point-to-point.

Impact on Implementations

No current API is impacted. New _X APIs for all point-to-point operations affected will be needed.

MPI implementations will have to be 64-bit clean inside since count*extent > 2^31 is already problematic for some implementations. New APIs will have to be added and the internals of MPI will have to be 64-bit capable for buffers and related issues.

Impact on Users

Users who opt in with the new API will be able to have communicators larger than 2^31. [MPI_Rank]

References

See also Ticket #80, #98, #99, #100

[jeffhammond]

I vigorously object to MPI_Rank. At the very least, this has nothing to do with the original problem and proposed solutions.

Furthermore, you need to propose new versions of a large number of functions that have nothing to do with large-count support. Below is a partial list.

• MPI_Comm_rank
• MPI_Comm_size
• MPI_Group_translate_ranks
• MPI_Group_incl
• MPI_Group_excl
• MPI_Group_range_incl
• MPI_Group_range_excl
• MPI_Cart_coords
• MPI_Graph_neighbors_count
• MPI_Graph_neighbors
• MPI_Cart_shift
• MPI_Win_lock and MPI_Win_unlock
• MPI_Win_shared_query
• MPI_Win_flush and MPI_Win_flush_local
• MPI_Type_create_darray

Please make the MPI_Rank proposal separate. If you insist on adding it to the large-count stuff, you will likely tank the whole thing, because I'm going to lobby full-time against anything that includes MPI_Rank.

If somebody builds a system that needs more than needs more than 2147483648 ranks, it is not unreasonable to expect them to move to ILP64 such that INT_MAX is sufficiently large and no MPI changes are required.

[jeffhammond]

Honestly, MPI_Rank is a reason to break backwards-compatibility because it will break ABI if it is implemented in the intended manner (something wider thanint).

You really need to consider how badly you want to break every piece of MPI software in the world today and if the nonsensical possibility of a machine that effectively supports more than 2147483648 ranks is worth it.

MPICH

/* The order of these elements must match that in mpif.h, mpi_f08_types.f90,
   and mpi_c_interface_types.f90 */
typedef struct MPI_Status {
    int count_lo;
    int count_hi_and_cancelled;
    int MPI_SOURCE;
    int MPI_TAG;
    int MPI_ERROR;
} MPI_Status;

Open-MPI

struct ompi_status_public_t {
    /* These fields are publicly defined in the MPI specification.
       User applications may freely read from these fields. */
    int MPI_SOURCE;
    int MPI_TAG;
    int MPI_ERROR;
    /* The following two fields are internal to the Open MPI
       implementation and should not be accessed by MPI applications.
       They are subject to change at any time.  These are not the
       droids you're looking for. */
    int _cancelled;
    size_t _ucount;
};
typedef struct ompi_status_public_t ompi_status_public_t;

[jeffhammond]

You will also need to revise the matching rules for how this works when users send using MPI_Rank and probe or recv using int. What is the semantic of this? Return an error code and expect the user to try again with the new function?

[tonyskjellum]

Jeff, we will consider that.

Process: Martin specifically asked that we consider the MPI_Rank option. We will specifically split the vote in a way to allow the forum to accept/reject this change for MPI-4. It was pointed out that endpoints and GPU-like devices and fine-grain accelerators could yield > 2^31 ranks in a communicator.

Technical: The problem with the heterogeneous use of the APIs will have to be fully considered like you're saying. Seems like, to allow this, protocols will carry an extra 32-bits of rank space. (Not my favorite answer, that's my straw answer. That means a tax on current performance.)

[tonyskjellum]

How about if I split this ticket now --- a) MPI_Count + miscellaneous; b) MPI_Rank ?

[jdinan]

There are several constants that can be passed through rank arguments -- e.g. MPI_ANY_SOURCE and MPI_PROC_NULL. We would potentially need to introduce _X versions of these constants or require implementations to define them in a way that is compatible with both int and MPI_Rank (in both C and Fortran language interfaces).