New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Non-blocking communicator/file constructors/destructors/etc, respectively to "Groups, Contexts, Communicators, Caching," "Topology," and "I/O" chapters #78
Comments
The MPI_COMM_IDUP_WITH_INFO function was added by issue #53 (pull request https://github.com/mpi-forum/mpi-standard/pull/14). |
Removing MPI_COMM_IDUP_WITH_INFO from this set to remove duplication. We will combine issue #53 and this issue, as appropriate, in discussions and presentations. |
Here is a preview of all the functionality proposed. It is not a pull request yet because we have not discussed it in either the Collective WG nor in a plenary. Comments most welcome. |
I would like to remind us that the MPI-3 forum decided consciously to not standardize all these functions in their nonblocking version even though it is straight-forward and an obvious gap. The reasoning was to lower the barrier for implementer of MPI libraries and foster MPI-3's adoption. So we only standardized the most crucial functions comm_dup that enables to implement nonlocking libraries on top of MPI. All others can build on this and thus seem to be syntactic sugar or can you elaborate what is different from MPI-3? Also, some of your function names exceed F77's maximum character limit ;-). |
Fortran 77 allowed a maximum of 6 character symbols so every single
function in the standard violates this.
What you refer to is the 72-column limit of fixed-source form that exists
in every version of Fortran. But I am not certain that the problem here is
worse than MPI_Rget_accumulate, which is the basis for example text on this
subject.
|
I would like to remind us that the MPI-3 forum decided consciously to not
standardize all these functions in their nonblocking version even though it
is straight-forward and an obvious gap. The reasoning was to lower the
barrier for implementer of MPI libraries and foster MPI-3's adoption.
Given the adoption of MPI-3 has been pretty good, and the implementation
effort of this type of feature does not seem that bad, we should focus on
what makes sense and not get hung up on the adoption argument.
So we only standardized the most crucial functions comm_dup that enables to
implement nonlocking libraries on top of MPI. All others can build on this
and thus seem to be syntactic sugar or can you elaborate what is different
from MPI-3?
They are clearly not syntactic sugar. You can’t implement nonblocking
comm_free with anything in the standard right now.
In any case, I won’t support this unless we add the RMA equivalents as
well. I can actually implement nonblocking win_{create,allocate} using
dynamic windows and nonblocking collectives but the performance overhead
that results is a deal-breaker.
|
Jeff, Torsten et al, I appreciate that there is discussion and feedback vs.
a big yawn :-)
1) I agree about the RMA equivalents. We can discuss this at the collective
WG meeting in Portland.
If there is no objective, I plan to write those uptoo, but I suggest that I would wish for guidance from others as part of that process.
2) The functions posed are not syntactic sugar, as you observe
MPI_COMM_IFREE -- gives the opportunity for recovering resources that
MPI_COMM_FREE does not.
And, the lack of orthogonality in the standard is damning. They are all of
the same class but do different things, and we have offered the user some nonblocking and some only blocking. [We don't disagree here.]
3) The MPI Forum often makes us come back several times to get things
right. It took 15 years between the time I first proposed nonblocking collectives
and persistent collectives a) for Torsten to get nonblocking collectives
into MPI-3 [despite that IBM had nonblocking collectives in their
vendor-specific library before we started MPI-1],
b) for us to now come close to getting persistent collectives for
MPI-3+/4/Next. That it was turned down before is no problem from my
viewpoint either. The history of being conservative is well established.
4) Any limits of FORTRAN names, we can address as special naming. FORTRAN
making something unstandardizable was a good excuse 25 years ago, but no
longer.
We work around, we don't use it as a reason not to standardize. The
currently proposed API names of is editorial and conventional not
normative from the point of view of the value of the operation performed... "What's in a name?"
5) With the fault-tolerance work, analogous nonblocking functions such as
MPI_Comm_Ishrink are proposed. The closest to this without a fault is
MPI_Comm_Icreate or MPI_Comm_Isplit_with_type. We cannot be in the business
of standardizing the FT operations without the fault-free analogs, for
standard to be logical, orthogonal, and complete. MPI_Comm_Ishrink ==
MPI_Comm_Isplit_with_type where the type is MPI_FAULT_CONDITION (or such),
and the implementation picks the color as a function of whether or not the
consensus is achieved that the MPI process lives. So, by looking at where
ULFM's tickets go,
we find the lack of the original functionality. By making the standard
orthogonal, then we can stop looking at some new functionality in
isolation, but recognie the duality of MPI_Comm_Isplit_with_type and
MPI_Comm_Ishrink (the ULFM version of it, anyway), and begin to argue,
wait, this is a special form of a collective we already have. Let's
discuss if there is a way to make such operations come into the standard in
way that closely relates them to functions we already have with only a
simple side effect. A benefit thereof is that we say, hey "implicit colors
in split" are a good idea. What ways can we get them.
Well, machine properties are reognized (one key only); now we talk fault
tolerance state, later maybe multiple ideas involve implicit colors in
isplit :-) [This motivates me (both for collective chapter, and how to cast
FT operations a bit differently than the ULFM approach... more atomically perhaps.]
Much obliged.
Regards,
Tony
…On Sun, Feb 4, 2018 at 1:34 PM, Jeff Hammond ***@***.***> wrote:
> I would like to remind us that the MPI-3 forum decided consciously to not
> standardize all these functions in their nonblocking version even though
it
> is straight-forward and an obvious gap. The reasoning was to lower the
> barrier for implementer of MPI libraries and foster MPI-3's adoption.
>
Given the adoption of MPI-3 has been pretty good, and the implementation
effort of this type of feature does not seem that bad, we should focus on
what makes sense and not get hung up on the adoption argument.
So we only standardized the most crucial functions comm_dup that enables to
> implement nonlocking libraries on top of MPI. All others can build on
this
> and thus seem to be syntactic sugar or can you elaborate what is
different
> from MPI-3?
>
They are clearly not syntactic sugar. You can’t implement nonblocking
comm_free with anything in the standard right now.
In any case, I won’t support this unless we add the RMA equivalents as
well. I can actually implement nonblocking win_{create,allocate} using
dynamic windows and nonblocking collectives but the performance overhead
that results is a deal-breaker.
> --
Jeff Hammond
***@***.***
http://jeffhammond.github.io/
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#78 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AA38icwiCVeBkWJSg5ENrD5YFHD1Cuq7ks5tRfg_gaJpZM4R4NbA>
.
--
Anthony Skjellum, PhD
skjellum@gmail.com
Cell: +1-205-807-4968
|
1-sided Windows non-blocking constructors and destructor functionality now added as well. |
What about MPI_Files? MPI_File_iopen, MPI_File_iclose, and so on? |
The PDF that includes nonblocking window creation functions is a bit messed up - there are two chapters (11 & 12) entitled "One-sided Communication" with similar but different content. The text for MPI_WIN_ICREATE in chapter 12 describes the MPI_WIN_ICREATE_DYNAMIC operation. The text for MPI_WIN_ICREATE in chapter 11 includes:
This is a little ambiguous - it could be taken to mean "before the MPI_WIN_ICREATE function returns", which is too early. I guess this is supposed to mean "before the request is completed by a successful call to MPI_WAIT[_ALL] or MPI_TEST[_ALL] that returned flag = true" or some such similar wording. Counter-argument: this ambiguous wording is copy-pasted from MPI_COMM_IDUP. There are a lot of cross-references that appear as ?? throughout. |
Should there also be MPI_WIN_IFENCE (etc, for other synchronisation methods)? |
@dholmes-epcc-ed-ac-uk "completed" means "waited upon (or equivalent)" in many other places in the standard, e.g.:
Please don't try to add nonblocking RMA synchronization here. That needs to be handled by the RMA WG. We've been down that path before. Pavan and friends wrote a paper on it. I support the features but RMA needs to drive if after thinking through all the details. |
Jeff, OK, thank you, we will rescind that. You specifically asked for it
as a requirement for considering the baseline concept :-)
No worries.
Regards,
Tony
PS Nonblocking oonstructors and destructors seem to work well in RMA. Can
you point me at the paper you mentioned.
|
Folks: I am going to split this out into a discussion with RMA group as
Jeff suggests. Jeff motivated me to think about all this.
I agree we will split the idea and present to the RMA WG, with the benefit
also of their previous thinking, per Jeff.
|
Dan, Not sure what happened with the LaTeX; but given Jeff's input, I will
split the issues so it can be a discussion with RMA WG.
More soon.
Remember, I am not trying to read anything on this in Portland. This is
just opening the discussion.
Would you work with help me with cleaning up the Win proposal? Which
would be separated.
|
@tonyskjellum I asked for "RMA equivalents" to communicator constructors and destructors, not synchronization operations, meaning |
After discussions with @dholmes-epcc-ed-ac-uk and Puri Bangalore, we have added the prospective non-blocking constructor, destructor, and collective operations that appear most needful of non-blocking variants. The proposed document will soon be updated with those APIs (as well as we-adding the parts of the API from 1-sided we removed this AM due to LaTeX bug). |
Please see this update: i) reincorporates the 1-sided proposed functions |
Title of issue should now be more general than "Add missing [communicator] operations ..." - perhaps just "Add missing object constructors and destructors" with the chapter references moved into the "Changes to the text" section of the description? |
The only new blocking routine MPI_SPLIT_WITH_INFO is an odd-one-out on this issue because it adds new functionality (the possibility of supplying an MPI_INFO to the split operation). Additional justification is needed for adding this new function (strictly function-pair, because you propose a nonblocking version too). Should that be a different issue? I can see why it is included in this issue - it is a missing communicator constructor. Either way, I feel it should be called out as different to the others and separately justified - what can be done with MPI_INFO that cannot be done with "color" and "key"? Why are you not proposing full orthogonality, i.e. a "_WITH_INFO" version of all communicator constructors? All Window and File constructors already have MPI_INFO arguments - why is MPI_COMM_SPLIT as special as MPI_COMM_DUP whereas the others are not? |
Ok
Anthony Skjellum, PhD
205-807-4968
… On Feb 14, 2018, at 5:31 AM, Dan Holmes ***@***.***> wrote:
Title of issue should now be more general than "Add missing [communicator] operations ..." - perhaps just "Add missing object constructors and destructors" with the chapter references moved into the "Changes to the text" section of the description?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Today, the associated pull request has been added for a plan to read at Austin meeting; pull request ahead of 2-week deadline. We may still update PR further before the two-week deadline. |
This is the public copy. |
We did the first reading attempt in Austin on June 13, 2018. There were a few issues raised, and quality improvements needed. We will fix these and present a new reading in Barcelona in September. |
At the Persistence/Collective joint meeting today, we agreed that we need to justify the utility of these functions and establish community best practices. |
We will need to retarget for MPI-4.x |
Decided to focus this for MPI-5.0. This orthogonalizes the standard more, and provides a path to useful new functionality. So, we want to bring this back. @dholmes-epcc-ed-ac-uk : when we split between orthogonalization and key new functionality, we will explore splitting ticket, but not before. |
@wesbland @Wee-Free-Scot Hi, I am interested in pursuing this with Dan for MPI-5.0. We are re-reading the comments, and I will close the loop with him to see how to proceed. Can we make a ticket like this that cross-cuts many chapters, or should we do piecemeal? |
If you're the one planning to do the work across many chapters, than one ticket is great. If you are planning to rally the CCs from each chapter and having each one does as its own PR, then one ticket per chapter/PR is best. |
Problem
The standard functionality is not complete; some non-blocking now, some not.
Proposal
I. As a baseline goal, these functions will be proposed for addition to MPI-3.x:
Groups, Contexts, Communicators, and Caching Chapter:
Non-blocking:
1. both intra-comm and inter-comm
* MPI_COMM_ISPLIT
* MPI_COMM_ISPLIT_TYPE
* MPI_INTERCOMM_IMERGE
* MPI_INTERCOMM_ICREATE
* MPI_COMM_ICREATE
2. intra-comm only:
* MPI_COMM_ICREATE_GROUP
3. Destructor category:
* MPI_COMM_IFREE (see also MPI_COMM_IDISCONNECT)
4. Other:
* MPI_COMM_ISET_INFO
II. Topology chapter variants to be added to comment.
* MPI_CART_ICREATE
* MPI_GRAPH_ICREATE
* MPI_DIST_GRAPH_ICREATE
* MPI_DIST_GRAPH_ICREATE_ADJACENT
III. I/O Chapter
IV. Add non-blocking destructor to the "Process Creation and Management Chapter" (DPM)
and clarify that both MPI_COMM_DISCONNECT and MPI_COMM_IDISCONNECT work on intracommunicators as well as intercommunicators.
Note: Dynamic process management functions (MPI_COMM_ACCEPT, MPI_COMM_CONNECT, MPI_COMM_SPAWN, MPI_COMM_SPAWN_MULTIPLE) are defined in the related Ticket #81. However, there is no proposed nonblocking equivalent for MPI_JOIN; see #13 for the proposal to deprecate MPI_COMM_JOIN, which is why we don't offer the nonblocking version in Ticket #81.
Other than the intentional duplication of MPI_COMM_IDISCONNECT with Ticket #81, this ticket is complementary to the remainder of Ticket #81 and #82
Changes to the Text
The chapters will be modified to provide explanations, definitions, and rationale for these added functions.
Impact on Implementations
This will require implementations to add these new functions; they are analogous to other functions already in the standard, and each should be incremental work.
Note: other proposals, such as for fault-tolerance, are exploring analogs to functions in this Chapter to be added by this proposal but missing in current MPI. A recent Fault Tolerance WG discussion revealed these omissions and opportunities for making MPI support fully nonblocking libraries and components more fully.
Impact on Users
Users will be able to write better and more completely nonblocking MPI programs and libraries. The design of the MPI standard will be more "orthogonal."
References
https://github.com/mpi-forum/mpi-standard
The associated PR is at: mpi-forum/mpi-standard#48
The RMA form of this proposal is Ticket #82
The text was updated successfully, but these errors were encountered: