New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Persistent Collective Communication #25

Open
tonyskjellum opened this Issue Dec 8, 2015 · 38 comments

Comments

@tonyskjellum

tonyskjellum commented Dec 8, 2015

This ticket proposes non-blocking persistent collective operations be added to the standard.

For each API that currently supports a request as an out argument of a non-blocking collective operation, MPI will support an _init version.

Example: MPI_Bcast() [blocking, MPI-1] -> MPI_Ibcast(...,info,request[,ierr]) [MPI 3.x] -> MPI_Bcast_init(...,info,request[,ierr]) [new]

The attached text is prototypical standards draft text.

Some quick notes:

MPI persistent collective operation _inits are initialization calls and follow all the ordering rules already existing for posing nonblocking collective operations (this implies _init operations are local and non-blocking). Remember that the point-to-point init functions disallow communication whereas the persistent collective inits may communicate but must not rely on state of other MPI processes.

The info argument is an in argument that allows for a limited number of hints about how the persistent, nonblocking collective will be used, as defined in the text (collective chapter).

The request argument is an out argument in each process of the calling group of the communicator that can be used zero or more times to start a collective operation. Each such request must be started in all the processes of the underlying group of the communicator (they are deemed active after this). Each request must be completed (making them inactive) before another start is permitted. Each process in the group of the communicator must complete the operation.

We do not require that starts across a group for persistent collective operations serialize to the same order. This makes using MPI_STARTALL allowed.

MPI_Test/MPI_Wait operations (all) give completion information without destroying the persistent request. MPI_Request_free() is used as with persistent point-to-point operations to free up an inactive persistent collective operation. This is the same behavior as for persistent point-to-point requests.

There are some per-communicator info items also defined that make sense on a communicator scope vs. a per-operation scope, also defined in the collective chapter.

Ticket #83 specifies additional functionality for info keys for these persistent operations, and is a related ticket that should be considered in sequence or in parallel with this ticket.

The associated PR is at: mpi-forum/mpi-standard#29

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Dec 8, 2015

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Dec 8, 2015

These are the latest versions of the Topology and Collective chapters impacted by our proposal.

@dholmes-epcc-ed-ac-uk

This comment has been minimized.

Member

dholmes-epcc-ed-ac-uk commented May 23, 2016

Latest version of pdf:
mpi-report.pdf

@wesbland

This comment has been minimized.

Member

wesbland commented Jun 2, 2016

I had a few concerns that I raised on the PR. None of them were significant, but it would be good to take care of them before next week.

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Jun 2, 2016

OK, I''ll go look...

On Thu, Jun 2, 2016 at 8:12 AM, Wesley Bland notifications@github.com
wrote:

I had a few concerns that I raised on the PR. None of them were
significant, but it would be good to take care of them before next week.


You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#25 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AA38iY10TUUwgsySG3G4RqNoXxb69s_7ks5qHtakgaJpZM4GxSUF
.

Anthony Skjellum, PhD
skjellum@gmail.com
Cell: +1-205-807-4968

@schulzm

This comment has been minimized.

schulzm commented Jun 3, 2016

I added a few more comments, especially on the intro section.

@hritzdorf

This comment has been minimized.

hritzdorf commented Jun 3, 2016

Page 215, Line 5-9 of mpi-report.pdf specifies:
After initialization, all associated send buffers and buffers associated with input arguments (such as arrays of counts, displacements, or datatypes in the vector versions of the collectives) should not be modified, and all associated receive buffers should not be accessed, until the corresponding persistent request is freed.

I think, that the send buffers can be modified until MPI_Start/MPI_Startall and receive buffers can be accessed after the corresponding request is completed not freed.

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Jun 3, 2016

We do not allow these operations in Startall, but you are correct:

I would agree with this modified statement: I think, that the send buffers
can be modified until MPI_Start/ and receive buffers can be accessed after
the corresponding request is completed not freed.

Tony

On Fri, Jun 3, 2016 at 5:15 AM, Hubert Ritzdorf notifications@github.com
wrote:

Page 215, Line 5-9 of mpi-report.pdf specifies:
After initialization, all associated send buffers and buffers associated
with input arguments (such as arrays of counts, displacements, or datatypes
in the vector versions of the collectives) should not be modified, and all
associated receive buffers should not be accessed, until the corresponding
persistent request is freed.

I think, that the send buffers can be modified until
MPI_Start/MPI_Startall and receive buffers can be accessed after the
corresponding request is completed not freed.


You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#25 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AA38iTBIG5ZU5xmoLVDkbhwfjWEuFOf5ks5qH_7QgaJpZM4GxSUF
.

Anthony Skjellum, PhD
skjellum@gmail.com
Cell: +1-205-807-4968

@schulzm

This comment has been minimized.

schulzm commented Jun 3, 2016

I had the same comment - see comment on the PR - I think we have to allow this, otherwise we can only send the same data over and over again.

To be more precise, we can modify send buffers until Start and then again after the request is complete until the next Start. Similar for receive buffers.

@tonyskjellum: Can you clarify the statement that Startall is not allowed? I didn't get this from the text.

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Jun 3, 2016

I will check with Dan about start all ..
At one point we discussed a restriction

The wording about changing buffers is inadvertently restrictive --- we need to be able to change buffers when the requests are inactive ...

@dholmes-epcc-ed-ac-uk

This comment has been minimized.

Member

dholmes-epcc-ed-ac-uk commented Jun 6, 2016

As far as I recall, the “no buffer change” restriction is about preventing the user from changing the address of the buffer, not the content of the buffer.

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Jun 6, 2016

That's logical and what I remember too now that you mention

@schulzm

This comment has been minimized.

schulzm commented Jun 6, 2016

The user can't do that anyway - there is no way to access or change the pointer after the init calls - it also different to the terminology that we use in the rest of the standard (as far as I remember): changing a buffer means changing its contents (and that's a real restriction between start and completions and should be mentioned)

@RolfRabenseifner

This comment has been minimized.

RolfRabenseifner commented Sep 22, 2016

Hi Tony, Ryan, and Dan,

As we saw, the wording in the PT2PT Persistent Communication
chapter uses always "should" instead of "have to"/"must".

The meaning of "should" may be taken out of

http://www.iso.org/iso/how-to-write-standards.pdf

"In all clauses, you should be clear about
what is a requirement and what is a
recommendation or other statement.
ISO uses the following words to make
the distinction :
• Requirements : shall, shall not
• Recommendations : should, should not
• Permission : may, need not
• Possibility and capability : can, cannot"

Best regards
Rolf

Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner@hlrs.de
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

@wgropp

This comment has been minimized.

wgropp commented Sep 22, 2016

I’ve added this information and some related info to the instructions document (instr.tex), which I’ll push later today.

Bill

William Gropp
Acting Director and Chief Scientist, NCSA
Director, Parallel Computing Institute
Thomas M. Siebel Chair in Computer Science
University of Illinois Urbana-Champaign

@tonyskjellum

This comment has been minimized.

tonyskjellum commented May 29, 2017

The associated pull request is now: mpi-forum/mpi-standard#29

@wesbland

This comment has been minimized.

Member

wesbland commented May 29, 2018

@tonyskjellum / @dholmes-epcc-ed-ac-uk - Were there any "no no" changes made here that need to be read? Nothing is on the schedule, but I thought I remembered that there were a few needed after the reading (and the above comment seems to indicate that).

@tonyskjellum

This comment has been minimized.

tonyskjellum commented May 29, 2018

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Jun 6, 2018

@schulzm

This comment has been minimized.

schulzm commented Jun 14, 2018

Passed first vote during Austin Forum Meeting in June 2018

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Jun 14, 2018

This passed the first vote in Austin; we will have the second vote in Barcelona.

@puribangalore

This comment has been minimized.

puribangalore commented Sep 4, 2018

Here is the PDF file with minor edits to fix the API for second vote in Barcelona.
mpi32-report-ticket25-barcelona-vote-sep2018.pdf

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Sep 5, 2018

A request for no-no-votes and for the formal second vote were made before the two-week deadline just now.

@schulzm

This comment has been minimized.

schulzm commented Sep 21, 2018

Passed no/no vote and second vote at Barcelona meeting in Sep. 2018, ready to be merged into golden copy

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Sep 21, 2018

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Sep 23, 2018

It would now be apropos to discuss if there are any other places in the MPI standard besides Chapters 5 and 7 where persistent collective operations should be posed... we should advance such tickets (e.g., MPI I/O).

@schulzm

This comment has been minimized.

schulzm commented Sep 24, 2018

Vote tally: 16 yes, 0 abstain, 0 no - full results at https://www.mpi-forum.org/meetings/2018/09/votes

@wesbland

This comment has been minimized.

Member

wesbland commented Oct 5, 2018

Now that this is merged, it can be closed, right @schulzm / @tonyskjellum ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment