Non-blocking communicator/file constructors/destructors/etc, respectively to "Groups, Contexts, Communicators, Caching," "Topology," and "I/O" chapters

[tonyskjellum]

Problem

The standard functionality is not complete; some non-blocking now, some not.

Proposal

I. As a baseline goal, these functions will be proposed for addition to MPI-3.x:
Groups, Contexts, Communicators, and Caching Chapter:
Non-blocking:
1. both intra-comm and inter-comm
* MPI_COMM_ISPLIT
* MPI_COMM_ISPLIT_TYPE
* MPI_INTERCOMM_IMERGE
* MPI_INTERCOMM_ICREATE
* MPI_COMM_ICREATE
2. intra-comm only:
* MPI_COMM_ICREATE_GROUP
3. Destructor category:
* MPI_COMM_IFREE (see also MPI_COMM_IDISCONNECT)
4. Other:
* MPI_COMM_ISET_INFO

II. Topology chapter variants to be added to comment.
* MPI_CART_ICREATE
* MPI_GRAPH_ICREATE
* MPI_DIST_GRAPH_ICREATE
* MPI_DIST_GRAPH_ICREATE_ADJACENT

III. I/O Chapter

1. Non-block constructor:

• MPI_FILE_IOPEN

2. Non-blocking destructor:

• MPI_FILE_ICLOSE

3. Other file-level operations:

• MPI_FILE_IDELETE
• MPI_FILE_ISETSIZE
• MPI_FILE_IPREALLOCATE

4. Maybe (because they are collective):

• MPI_FILE_ISET_INFO
• MPI_FILE_ISET_VIEW

IV. Add non-blocking destructor to the "Process Creation and Management Chapter" (DPM)

• MPI_COMM_IDISCONNECT
  and clarify that both MPI_COMM_DISCONNECT and MPI_COMM_IDISCONNECT work on intracommunicators as well as intercommunicators.

Note: Dynamic process management functions (MPI_COMM_ACCEPT, MPI_COMM_CONNECT, MPI_COMM_SPAWN, MPI_COMM_SPAWN_MULTIPLE) are defined in the related Ticket #81. However, there is no proposed nonblocking equivalent for MPI_JOIN; see #13 for the proposal to deprecate MPI_COMM_JOIN, which is why we don't offer the nonblocking version in Ticket #81.

Other than the intentional duplication of MPI_COMM_IDISCONNECT with Ticket #81, this ticket is complementary to the remainder of Ticket #81 and #82

Changes to the Text

The chapters will be modified to provide explanations, definitions, and rationale for these added functions.

Impact on Implementations

This will require implementations to add these new functions; they are analogous to other functions already in the standard, and each should be incremental work.

Note: other proposals, such as for fault-tolerance, are exploring analogs to functions in this Chapter to be added by this proposal but missing in current MPI. A recent Fault Tolerance WG discussion revealed these omissions and opportunities for making MPI support fully nonblocking libraries and components more fully.

Impact on Users

Users will be able to write better and more completely nonblocking MPI programs and libraries. The design of the MPI standard will be more "orthogonal."

References

https://github.com/mpi-forum/mpi-standard

The associated PR is at: mpi-forum/mpi-standard#48

The RMA form of this proposal is Ticket #82

[dholmes-epcc-ed-ac-uk]

The MPI_COMM_IDUP_WITH_INFO function was added by issue #53 (pull request mpi-forum/mpi-standard#14).

[tonyskjellum]

Removing MPI_COMM_IDUP_WITH_INFO from this set to remove duplication. We will combine issue #53 and this issue, as appropriate, in discussions and presentations.

[tonyskjellum]

Here is a preview of all the functionality proposed. It is not a pull request yet because we have not discussed it in either the Collective WG nor in a plenary. Comments most welcome.

mpi32-report-04feb18.pdf

[htorst]

I would like to remind us that the MPI-3 forum decided consciously to not standardize all these functions in their nonblocking version even though it is straight-forward and an obvious gap. The reasoning was to lower the barrier for implementer of MPI libraries and foster MPI-3's adoption. So we only standardized the most crucial functions comm_dup that enables to implement nonlocking libraries on top of MPI. All others can build on this and thus seem to be syntactic sugar or can you elaborate what is different from MPI-3?

Also, some of your function names exceed F77's maximum character limit ;-).

[jeffhammond]

Fortran 77 allowed a maximum of 6 character symbols so every single function in the standard violates this. What you refer to is the 72-column limit of fixed-source form that exists in every version of Fortran. But I am not certain that the problem here is worse than MPI_Rget_accumulate, which is the basis for example text on this subject.

[jeffhammond]

I would like to remind us that the MPI-3 forum decided consciously to not standardize all these functions in their nonblocking version even though it is straight-forward and an obvious gap. The reasoning was to lower the barrier for implementer of MPI libraries and foster MPI-3's adoption.

Given the adoption of MPI-3 has been pretty good, and the implementation effort of this type of feature does not seem that bad, we should focus on what makes sense and not get hung up on the adoption argument.

So we only standardized the most crucial functions comm_dup that enables to implement nonlocking libraries on top of MPI. All others can build on this and thus seem to be syntactic sugar or can you elaborate what is different from MPI-3?

They are clearly not syntactic sugar. You can’t implement nonblocking comm_free with anything in the standard right now. In any case, I won’t support this unless we add the RMA equivalents as well. I can actually implement nonblocking win_{create,allocate} using dynamic windows and nonblocking collectives but the performance overhead that results is a deal-breaker.

[tonyskjellum]

Jeff, Torsten et al, I appreciate that there is discussion and feedback vs. a big yawn :-) 1) I agree about the RMA equivalents. We can discuss this at the collective WG meeting in Portland. If there is no objective, I plan to write those uptoo, but I suggest that I would wish for guidance from others as part of that process. 2) The functions posed are not syntactic sugar, as you observe MPI_COMM_IFREE -- gives the opportunity for recovering resources that MPI_COMM_FREE does not. And, the lack of orthogonality in the standard is damning. They are all of the same class but do different things, and we have offered the user some nonblocking and some only blocking. [We don't disagree here.] 3) The MPI Forum often makes us come back several times to get things right. It took 15 years between the time I first proposed nonblocking collectives and persistent collectives a) for Torsten to get nonblocking collectives into MPI-3 [despite that IBM had nonblocking collectives in their vendor-specific library before we started MPI-1], b) for us to now come close to getting persistent collectives for MPI-3+/4/Next. That it was turned down before is no problem from my viewpoint either. The history of being conservative is well established. 4) Any limits of FORTRAN names, we can address as special naming. FORTRAN making something unstandardizable was a good excuse 25 years ago, but no longer. We work around, we don't use it as a reason not to standardize. The currently proposed API names of is editorial and conventional not normative from the point of view of the value of the operation performed... "What's in a name?" 5) With the fault-tolerance work, analogous nonblocking functions such as MPI_Comm_Ishrink are proposed. The closest to this without a fault is MPI_Comm_Icreate or MPI_Comm_Isplit_with_type. We cannot be in the business of standardizing the FT operations without the fault-free analogs, for standard to be logical, orthogonal, and complete. MPI_Comm_Ishrink == MPI_Comm_Isplit_with_type where the type is MPI_FAULT_CONDITION (or such), and the implementation picks the color as a function of whether or not the consensus is achieved that the MPI process lives. So, by looking at where ULFM's tickets go, we find the lack of the original functionality. By making the standard orthogonal, then we can stop looking at some new functionality in isolation, but recognie the duality of MPI_Comm_Isplit_with_type and MPI_Comm_Ishrink (the ULFM version of it, anyway), and begin to argue, wait, this is a special form of a collective we already have. Let's discuss if there is a way to make such operations come into the standard in way that closely relates them to functions we already have with only a simple side effect. A benefit thereof is that we say, hey "implicit colors in split" are a good idea. What ways can we get them. Well, machine properties are reognized (one key only); now we talk fault tolerance state, later maybe multiple ideas involve implicit colors in isplit :-) [This motivates me (both for collective chapter, and how to cast FT operations a bit differently than the ULFM approach... more atomically perhaps.] Much obliged. Regards, Tony

…

On Sun, Feb 4, 2018 at 1:34 PM, Jeff Hammond ***@***.***> wrote: > I would like to remind us that the MPI-3 forum decided consciously to not > standardize all these functions in their nonblocking version even though it > is straight-forward and an obvious gap. The reasoning was to lower the > barrier for implementer of MPI libraries and foster MPI-3's adoption. > Given the adoption of MPI-3 has been pretty good, and the implementation effort of this type of feature does not seem that bad, we should focus on what makes sense and not get hung up on the adoption argument. So we only standardized the most crucial functions comm_dup that enables to > implement nonlocking libraries on top of MPI. All others can build on this > and thus seem to be syntactic sugar or can you elaborate what is different > from MPI-3? > They are clearly not syntactic sugar. You can’t implement nonblocking comm_free with anything in the standard right now. In any case, I won’t support this unless we add the RMA equivalents as well. I can actually implement nonblocking win_{create,allocate} using dynamic windows and nonblocking collectives but the performance overhead that results is a deal-breaker. > -- Jeff Hammond ***@***.*** http://jeffhammond.github.io/ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA38icwiCVeBkWJSg5ENrD5YFHD1Cuq7ks5tRfg_gaJpZM4R4NbA> .

-- Anthony Skjellum, PhD skjellum@gmail.com Cell: +1-205-807-4968

[tonyskjellum]

1-sided Windows non-blocking constructors and destructor functionality now added as well.

mpi32-report-10feb18.pdf

[dholmes-epcc-ed-ac-uk]

What about MPI_Files? MPI_File_iopen, MPI_File_iclose, and so on?

[dholmes-epcc-ed-ac-uk]

The PDF that includes nonblocking window creation functions is a bit messed up - there are two chapters (11 & 12) entitled "One-sided Communication" with similar but different content.

The text for MPI_WIN_ICREATE in chapter 12 describes the MPI_WIN_ICREATE_DYNAMIC operation.

The text for MPI_WIN_ICREATE in chapter 11 includes:

It is erroneous to use the window win as an input argument to other MPI functions before the MPI_WIN_ICREATE operation completes.

This is a little ambiguous - it could be taken to mean "before the MPI_WIN_ICREATE function returns", which is too early. I guess this is supposed to mean "before the request is completed by a successful call to MPI_WAIT[_ALL] or MPI_TEST[_ALL] that returned flag = true" or some such similar wording. Counter-argument: this ambiguous wording is copy-pasted from MPI_COMM_IDUP.

There are a lot of cross-references that appear as ?? throughout.

[dholmes-epcc-ed-ac-uk]

Should there also be MPI_WIN_IFENCE (etc, for other synchronisation methods)?
Same reason as MPI_IBARRIER?

[jeffhammond]

@dholmes-epcc-ed-ac-uk "completed" means "waited upon (or equivalent)" in many other places in the standard, e.g.:

These operations are nonblocking: the call initiates the transfer, but the transfer may continue after the call returns. The transfer is completed, at the origin or both the origin and the target, when a subsequent synchronization call is issued by the caller on the involved window object.

Please don't try to add nonblocking RMA synchronization here. That needs to be handled by the RMA WG. We've been down that path before. Pavan and friends wrote a paper on it. I support the features but RMA needs to drive if after thinking through all the details.

[tonyskjellum]

Jeff, OK, thank you, we will rescind that. You specifically asked for it as a requirement for considering the baseline concept :-) No worries. Regards, Tony PS Nonblocking oonstructors and destructors seem to work well in RMA. Can you point me at the paper you mentioned.

[tonyskjellum]

Folks: I am going to split this out into a discussion with RMA group as Jeff suggests. Jeff motivated me to think about all this. I agree we will split the idea and present to the RMA WG, with the benefit also of their previous thinking, per Jeff.

[tonyskjellum]

Dan, Not sure what happened with the LaTeX; but given Jeff's input, I will split the issues so it can be a discussion with RMA WG. More soon. Remember, I am not trying to read anything on this in Portland. This is just opening the discussion. Would you work with help me with cleaning up the Win proposal? Which would be separated.

[jeffhammond]

@tonyskjellum I asked for "RMA equivalents" to communicator constructors and destructors, not synchronization operations, meaning MPI_Win_{create,allocate,free}*.

[jeffhammond]

@tonyskjellum Paper = http://www.mcs.anl.gov/papers/P5153-0614.pdf

[tonyskjellum]

After discussions with @dholmes-epcc-ed-ac-uk and Puri Bangalore, we have added the prospective non-blocking constructor, destructor, and collective operations that appear most needful of non-blocking variants. The proposed document will soon be updated with those APIs (as well as we-adding the parts of the API from 1-sided we removed this AM due to LaTeX bug).

[tonyskjellum]

Please see this update:

i) reincorporates the 1-sided proposed functions
ii) Has the proposed I/O operations for the first time
iii) amends the change log description for all operations in this ticket/issue

mpi32-report-13feb18-2144.pdf

[dholmes-epcc-ed-ac-uk]

Title of issue should now be more general than "Add missing [communicator] operations ..." - perhaps just "Add missing object constructors and destructors" with the chapter references moved into the "Changes to the text" section of the description?

[dholmes-epcc-ed-ac-uk]

The only new blocking routine MPI_SPLIT_WITH_INFO is an odd-one-out on this issue because it adds new functionality (the possibility of supplying an MPI_INFO to the split operation). Additional justification is needed for adding this new function (strictly function-pair, because you propose a nonblocking version too). Should that be a different issue? I can see why it is included in this issue - it is a missing communicator constructor. Either way, I feel it should be called out as different to the others and separately justified - what can be done with MPI_INFO that cannot be done with "color" and "key"? Why are you not proposing full orthogonality, i.e. a "_WITH_INFO" version of all communicator constructors? All Window and File constructors already have MPI_INFO arguments - why is MPI_COMM_SPLIT as special as MPI_COMM_DUP whereas the others are not?

[tonyskjellum]

Ok Anthony Skjellum, PhD 205-807-4968

…

On Feb 14, 2018, at 5:31 AM, Dan Holmes ***@***.***> wrote: Title of issue should now be more general than "Add missing [communicator] operations ..." - perhaps just "Add missing object constructors and destructors" with the chapter references moved into the "Changes to the text" section of the description? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[tonyskjellum]

Specific feedback came from the plenary presentation of Ticket #78, and one-on-one discussions thereafter:

1. Split the RMA functions to a separate ticket so that the RMA working group can weigh in. We will make the separate ticket and hand it off for their study.

2. Put the MPI_IDISCONNECT into Ticket #81. [We decided to duplicate it there.]

3. Consider again if we need stronger MPI_COMM_DESTROY/IDESTROY functions for even
   stronger statement of resource recovery etc. Would be a separate ticket if so.

4. Additionally, during the reading of Ticket #25, it was pointed out that MPI_COMM_SET_INFO is collective. Therefore, we need to consider adding MPI_COMM_ISET_INFO to this ticket.

Our goal is to review Ticket #78 at a virtual meeting (and allow Ticket #81 to trail along), and read Ticket #78 (and possibly #81) at the June, 2018 meeting in Austin. The others (not named yet), would follow in due course if they gain adherents and proponents.

[tonyskjellum]

The latest draft attached

1. MPI_COMM_ISET_INFO added
2. DPM functions removed
3. Updated and corrected the changes chapter

mpi32-report.pdf

[tonyskjellum]

Today, the associated pull request has been added for a plan to read at Austin meeting; pull request ahead of 2-week deadline. We may still update PR further before the two-week deadline.

[tonyskjellum]

mpi32-report-ticket78.pdf

This is the public copy.

[tonyskjellum]

We did the first reading attempt in Austin on June 13, 2018. There were a few issues raised, and quality improvements needed. We will fix these and present a new reading in Barcelona in September.

[tonyskjellum]

At the Persistence/Collective joint meeting today, we agreed that we need to justify the utility of these functions and establish community best practices.