New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Info key(s) for Start Ordering Persistent Collective Communication #83

Open
tonyskjellum opened this Issue Mar 1, 2018 · 11 comments

Comments

@tonyskjellum

tonyskjellum commented Mar 1, 2018

Problem

We want to add one or more info keys, such as one that guarantees that the persistent collective operation will only be started in order across the group of its communicator. This concept was previously in Ticket #25, but removed. Other info keys relevant to persistent collective initialization may also be considered in addition to this one.

Note: this ticket actually describes the ordering rules for persistent collectives with and without this assertion with regard to all other collective communication on the same communicator. It is a pretty deep and interesting topic :-)

Proposal

Specify at least one info key for ordered-across-group in persistent collectives at communicator scope. Others TBD.

Changes to the Text

Appropriate text will be added that defines the info key and its semantics. If other keys are conceived of quickly, they may be added too. So far, just one info key,

Impact on Implementations

For the specific info key specified thus far, it is an assertion. Implementations can ignore this assertion, or exploit it for greater performance. The assertion is mpi_assert_strict_collective_ordering ; it requires that programs order persistent collectives[defined via a communicator that makes this assertion] under the same rules as other collectives. If an implementation can exploit this assertion for greater performance, it can do so, or do legally do nothing.

Impact on Users

Provides an info key; requires that the programs meet the restricted semantics if a given communicator with this assertion is used to generate persistent collective operations.

References

The pull request for this ticket is
mpi-forum/mpi-standard#91 . [Newly updated pull]

This ticket depends on Ticket #25.

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Mar 1, 2018

See page 217 of initial text...

mpi32-pers-infokeys-report-01mar18.pdf

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Mar 21, 2018

Verified today that this info key is at the communicator scope.

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Jun 6, 2018

mpi32-report-ticket83.pdf

This is the public copy.

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Jun 14, 2018

We need to make updates based on the reading attempt on June 13, 2018 (all minor). We will present a new reading in Barcelona.

@puribangalore

This comment has been minimized.

puribangalore commented Sep 5, 2018

Here is the latest PDF file for reading at Barcelona:
mpi32-report-ticket83-sep2018.pdf

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Sep 23, 2018

Outcome of reading

  1. Make clear that the info key provides a total ordering of all collective calls on a given communicator, not just a partial order involving Blocking/Nonblocking and another involving all Persistent. In other words, clarify the semantic impact.
  2. Refer to the semantic terms chapter for clear definition of start; do not restate it here.
    The semantic terms chapter is getting an additional virtual meeting before December f2f meeting.
  3. Add an example of what is "not legal" once we turn on this flag, for a given communicator. This will complement Puri's example(s) being worked for Ticket #90 .
  4. Make sure it is clear that MPI_Startall is just a part of the story, not the whole story.
@tonyskjellum

This comment has been minimized.

tonyskjellum commented Nov 1, 2018

Rudiments of the example/counter-examples:

Restrictions with key:

    persistent_op, persistent_op2 are specified on comm1

Rank 0:

    MPI_Bcast(...,comm1)
    MPI_Start(persistent_op)
    MPI_Reduce(,...,comm1)
    MPI_Start(persistent_op2)
    MPI_Gather(...,comm1)

All other ranks must do the exact same dispatch order for these collectives.

illegal:

    MPI_Bcast(...,comm1)
    MPI_IReduce(,...,comm1)
    MPI_Start(persistent_op)
    MPI_Start(persistent_op2)
    MPI_Gather(...,comm1)

Without key:

Rank 0:

    MPI_Bcast(...,comm1)
    MPI_Start(persistent_op)
    MPI_IReduce(,...,comm1)
    MPI_Start(persistent_op2)
    MPI_Gather(...,comm1)

Rank 1:

    MPI_Start(persistent_op)
    MPI_Bcast(...,comm1)
    MPI_IReduce(,...,comm1)
    MPI_Gather(...,comm1)
    MPI_Start(persistent_op2)

legal.

Rank N-1:
MPI_Bcast(...,comm1)
MPI_IReduce(,...,comm1)
MPI_Start(persistent_op2)
MPI_Gather(...,comm1)
MPI_Start(persistent_op)

Legal

Legal --> Other collectives still are issued in same order across group.

@tonyskjellum

This comment was marked as resolved.

tonyskjellum commented Nov 19, 2018

New info coming shortly for reading in December. Here is the promise it will be up soon.

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Nov 19, 2018

mpi32-report-ticket83-19nov18.pdf

Here is the rewritten and revised text.

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Dec 5, 2018

We did the reading in San Jose.

  1. @dholmes-epcc-ed-ac-uk and I will clarify with word 'defined' to the word 'started' in the text (change bar line 4314)

  2. we will also copy the definition of the total ordering into the section (line 4287-4297)... that will change the overall text by repetition. Then we will streamline to remove
    redundancy.

  3. @abouteiller reminded us to make the examples 'compilable'. So, we will make the examples compilable.

The group agrees that we will take ticket 0 votes in Chattanooga on these changes above, which do not change the syntax or semantics of the info key, and then if it passes, take the first vote in Chattanooga in March, 2019. If ticket 0 votes fails, we will re-read in Chattanooga.

@tonyskjellum tonyskjellum changed the title from Info key(s) for Persistent Collective Communication to Info key(s) for Start Ordering Persistent Collective Communication Dec 5, 2018

@tonyskjellum

This comment has been minimized.

tonyskjellum commented Dec 6, 2018

Latest text, post meeting, thanks to @dholmes-epcc-ed-ac-uk for the work on post-reading edits.
I still have to fix the examples to be compilable.

mpi32-report-ticket83-06dec18.pdf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment