Non-blocking communicator/file constructors/destructors/etc, respectively to "Groups, Contexts, Communicators, Caching," "Topology," and "I/O" chapters

[tonyskjellum]

Problem

The standard functionality is not complete; some non-blocking now, some not.

Proposal

I. As a baseline goal, these functions will be proposed for addition to MPI-3.x:
Groups, Contexts, Communicators, and Caching Chapter:
Non-blocking:
1. both intra-comm and inter-comm
* MPI_COMM_ISPLIT
* MPI_COMM_ISPLIT_TYPE
* MPI_INTERCOMM_IMERGE
* MPI_INTERCOMM_ICREATE
* MPI_COMM_ICREATE
2. intra-comm only:
* MPI_COMM_ICREATE_GROUP
3. Destructor category:
* MPI_COMM_IFREE (see also MPI_COMM_IDISCONNECT)
4. Other:
* MPI_COMM_ISET_INFO

II. Topology chapter variants to be added to comment.
* MPI_CART_ICREATE
* MPI_GRAPH_ICREATE
* MPI_DIST_GRAPH_ICREATE
* MPI_DIST_GRAPH_ICREATE_ADJACENT

III. I/O Chapter

1. Non-block constructor:

• MPI_FILE_IOPEN

2. Non-blocking destructor:

• MPI_FILE_ICLOSE

3. Other file-level operations:

• MPI_FILE_IDELETE
• MPI_FILE_ISETSIZE
• MPI_FILE_IPREALLOCATE

4. Maybe (because they are collective):

• MPI_FILE_ISET_INFO
• MPI_FILE_ISET_VIEW

IV. Add non-blocking destructor to the "Process Creation and Management Chapter" (DPM)

• MPI_COMM_IDISCONNECT
  and clarify that both MPI_COMM_DISCONNECT and MPI_COMM_IDISCONNECT work on intracommunicators as well as intercommunicators.

Note: Dynamic process management functions (MPI_COMM_ACCEPT, MPI_COMM_CONNECT, MPI_COMM_SPAWN, MPI_COMM_SPAWN_MULTIPLE) are defined in the related Ticket #81. However, there is no proposed nonblocking equivalent for MPI_JOIN; see #13 for the proposal to deprecate MPI_COMM_JOIN, which is why we don't offer the nonblocking version in Ticket #81.

Other than the intentional duplication of MPI_COMM_IDISCONNECT with Ticket #81, this ticket is complementary to the remainder of Ticket #81 and #82

Changes to the Text

The chapters will be modified to provide explanations, definitions, and rationale for these added functions.

Impact on Implementations

This will require implementations to add these new functions; they are analogous to other functions already in the standard, and each should be incremental work.

Note: other proposals, such as for fault-tolerance, are exploring analogs to functions in this Chapter to be added by this proposal but missing in current MPI. A recent Fault Tolerance WG discussion revealed these omissions and opportunities for making MPI support fully nonblocking libraries and components more fully.

Impact on Users

Users will be able to write better and more completely nonblocking MPI programs and libraries. The design of the MPI standard will be more "orthogonal."

References

https://github.com/mpi-forum/mpi-standard

The associated PR is at: mpi-forum/mpi-standard#48

The RMA form of this proposal is Ticket #82

[dholmes-epcc-ed-ac-uk]

The MPI_COMM_IDUP_WITH_INFO function was added by issue #53 (pull request https://github.com/mpi-forum/mpi-standard/pull/14).

[tonyskjellum]

Removing MPI_COMM_IDUP_WITH_INFO from this set to remove duplication. We will combine issue #53 and this issue, as appropriate, in discussions and presentations.

[tonyskjellum]

Here is a preview of all the functionality proposed. It is not a pull request yet because we have not discussed it in either the Collective WG nor in a plenary. Comments most welcome.

mpi32-report-04feb18.pdf

[htorst]

I would like to remind us that the MPI-3 forum decided consciously to not standardize all these functions in their nonblocking version even though it is straight-forward and an obvious gap. The reasoning was to lower the barrier for implementer of MPI libraries and foster MPI-3's adoption. So we only standardized the most crucial functions comm_dup that enables to implement nonlocking libraries on top of MPI. All others can build on this and thus seem to be syntactic sugar or can you elaborate what is different from MPI-3?

Also, some of your function names exceed F77's maximum character limit ;-).

[jeffhammond]

Fortran 77 allowed a maximum of 6 character symbols so every single function in the standard violates this. What you refer to is the 72-column limit of fixed-source form that exists in every version of Fortran. But I am not certain that the problem here is worse than MPI_Rget_accumulate, which is the basis for example text on this subject.

[jeffhammond]

I would like to remind us that the MPI-3 forum decided consciously to not standardize all these functions in their nonblocking version even though it is straight-forward and an obvious gap. The reasoning was to lower the barrier for implementer of MPI libraries and foster MPI-3's adoption.

Given the adoption of MPI-3 has been pretty good, and the implementation effort of this type of feature does not seem that bad, we should focus on what makes sense and not get hung up on the adoption argument.

So we only standardized the most crucial functions comm_dup that enables to implement nonlocking libraries on top of MPI. All others can build on this and thus seem to be syntactic sugar or can you elaborate what is different from MPI-3?

They are clearly not syntactic sugar. You can’t implement nonblocking comm_free with anything in the standard right now. In any case, I won’t support this unless we add the RMA equivalents as well. I can actually implement nonblocking win_{create,allocate} using dynamic windows and nonblocking collectives but the performance overhead that results is a deal-breaker.

[tonyskjellum]

Jeff, Torsten et al, I appreciate that there is discussion and feedback vs. a big yawn :-) 1) I agree about the RMA equivalents. We can discuss this at the collective WG meeting in Portland. If there is no objective, I plan to write those uptoo, but I suggest that I would wish for guidance from others as part of that process. 2) The functions posed are not syntactic sugar, as you observe MPI_COMM_IFREE -- gives the opportunity for recovering resources that MPI_COMM_FREE does not. And, the lack of orthogonality in the standard is damning. They are all of the same class but do different things, and we have offered the user some nonblocking and some only blocking. [We don't disagree here.] 3) The MPI Forum often makes us come back several times to get things right. It took 15 years between the time I first proposed nonblocking collectives and persistent collectives a) for Torsten to get nonblocking collectives into MPI-3 [despite that IBM had nonblocking collectives in their vendor-specific library before we started MPI-1], b) for us to now come close to getting persistent collectives for MPI-3+/4/Next. That it was turned down before is no problem from my viewpoint either. The history of being conservative is well established. 4) Any limits of FORTRAN names, we can address as special naming. FORTRAN making something unstandardizable was a good excuse 25 years ago, but no longer. We work around, we don't use it as a reason not to standardize. The currently proposed API names of is editorial and conventional not normative from the point of view of the value of the operation performed... "What's in a name?" 5) With the fault-tolerance work, analogous nonblocking functions such as MPI_Comm_Ishrink are proposed. The closest to this without a fault is MPI_Comm_Icreate or MPI_Comm_Isplit_with_type. We cannot be in the business of standardizing the FT operations without the fault-free analogs, for standard to be logical, orthogonal, and complete. MPI_Comm_Ishrink == MPI_Comm_Isplit_with_type where the type is MPI_FAULT_CONDITION (or such), and the implementation picks the color as a function of whether or not the consensus is achieved that the MPI process lives. So, by looking at where ULFM's tickets go, we find the lack of the original functionality. By making the standard orthogonal, then we can stop looking at some new functionality in isolation, but recognie the duality of MPI_Comm_Isplit_with_type and MPI_Comm_Ishrink (the ULFM version of it, anyway), and begin to argue, wait, this is a special form of a collective we already have. Let's discuss if there is a way to make such operations come into the standard in way that closely relates them to functions we already have with only a simple side effect. A benefit thereof is that we say, hey "implicit colors in split" are a good idea. What ways can we get them. Well, machine properties are reognized (one key only); now we talk fault tolerance state, later maybe multiple ideas involve implicit colors in isplit :-) [This motivates me (both for collective chapter, and how to cast FT operations a bit differently than the ULFM approach... more atomically perhaps.] Much obliged. Regards, Tony

…

On Sun, Feb 4, 2018 at 1:34 PM, Jeff Hammond ***@***.***> wrote: > I would like to remind us that the MPI-3 forum decided consciously to not > standardize all these functions in their nonblocking version even though it > is straight-forward and an obvious gap. The reasoning was to lower the > barrier for implementer of MPI libraries and foster MPI-3's adoption. > Given the adoption of MPI-3 has been pretty good, and the implementation effort of this type of feature does not seem that bad, we should focus on what makes sense and not get hung up on the adoption argument. So we only standardized the most crucial functions comm_dup that enables to > implement nonlocking libraries on top of MPI. All others can build on this > and thus seem to be syntactic sugar or can you elaborate what is different > from MPI-3? > They are clearly not syntactic sugar. You can’t implement nonblocking comm_free with anything in the standard right now. In any case, I won’t support this unless we add the RMA equivalents as well. I can actually implement nonblocking win_{create,allocate} using dynamic windows and nonblocking collectives but the performance overhead that results is a deal-breaker. > -- Jeff Hammond ***@***.*** http://jeffhammond.github.io/ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA38icwiCVeBkWJSg5ENrD5YFHD1Cuq7ks5tRfg_gaJpZM4R4NbA> .

-- Anthony Skjellum, PhD skjellum@gmail.com Cell: +1-205-807-4968

[tonyskjellum]

1-sided Windows non-blocking constructors and destructor functionality now added as well.

mpi32-report-10feb18.pdf

[dholmes-epcc-ed-ac-uk]

What about MPI_Files? MPI_File_iopen, MPI_File_iclose, and so on?

[dholmes-epcc-ed-ac-uk]

The PDF that includes nonblocking window creation functions is a bit messed up - there are two chapters (11 & 12) entitled "One-sided Communication" with similar but different content.

The text for MPI_WIN_ICREATE in chapter 12 describes the MPI_WIN_ICREATE_DYNAMIC operation.

The text for MPI_WIN_ICREATE in chapter 11 includes:

It is erroneous to use the window win as an input argument to other MPI functions before the MPI_WIN_ICREATE operation completes.

This is a little ambiguous - it could be taken to mean "before the MPI_WIN_ICREATE function returns", which is too early. I guess this is supposed to mean "before the request is completed by a successful call to MPI_WAIT[_ALL] or MPI_TEST[_ALL] that returned flag = true" or some such similar wording. Counter-argument: this ambiguous wording is copy-pasted from MPI_COMM_IDUP.

There are a lot of cross-references that appear as ?? throughout.

[dholmes-epcc-ed-ac-uk]

Should there also be MPI_WIN_IFENCE (etc, for other synchronisation methods)?
Same reason as MPI_IBARRIER?

[jeffhammond]

@dholmes-epcc-ed-ac-uk "completed" means "waited upon (or equivalent)" in many other places in the standard, e.g.:

These operations are nonblocking: the call initiates the transfer, but the transfer may continue after the call returns. The transfer is completed, at the origin or both the origin and the target, when a subsequent synchronization call is issued by the caller on the involved window object.

Please don't try to add nonblocking RMA synchronization here. That needs to be handled by the RMA WG. We've been down that path before. Pavan and friends wrote a paper on it. I support the features but RMA needs to drive if after thinking through all the details.