Add example for use of shared memory

[mpiforumbot]

Originally by gropp on 2014-02-15 07:59:20 -0600

---

Add the following advice in MPI-3.0, Sect. 11.7, page 457, after line 3:

-Advice to users.*
In the unified memory model,
in the case where the window is in shared memory,
MPI_WIN_SYNC can be used to order store operations and make
store updates to the window visible to other processes and threads.
Use of this routine is necessary to ensure portable behavior
when point-to-point, collective, or shared memory synchronization
is used in place of an RMA synchronization routine.
MPI_WIN_SYNC should be called by the writer before the
non-RMA synchronization operation and by the reader
after the non-RMA synchronization, as shown in Example 11.13 on page ....
-(End of advice to users.)*

Add the following example at the end of MPI-3.0 Sect. 11.7, i.e., after page 461 line 20:

-Example 11.13*
The following example demonstrates the proper synchronization
in the unified memory model when
a data transfer is implemented with load and store
in the case of windows in shared memory (instead of MPI_PUT or MPI_GET)
and the synchronization between processes is performed using point-to-point
communication. The synchronization between processes must be supplemented
with a memory synchronization through calls to MPI_WIN_SYNC,
which act locally as a processor-memory barrier.
In Fortran, if MPI_ASYNC_PROTECTS_NONBLOCKING is .FALSE. or the variable X
is not declared as ASYNCHRONOUS, reordering of the accesses to the variable X
must be prevented with MPI_F_SYNC_REG operations.
(No equivalent function is needed in C.)

The variable X is contained within a shared memory window and X corresponds
to the same memory location at both processes. The MPI_WIN_SYNC operation
performed by process A ensures completion of the load/store operations
issued by process A. The MPI_WIN_SYNC operation performed by process B
ensures that process A's updates to X are visible to process B.

Process A               Process B

MPI_WIN_LOCK_ALL(       MPI_WIN_LOCK_ALL(
MPI_MODE_NOCHECK,win)   MPI_MODE_NOCHECK,win) 

DO ...                  DO ...
 X=...

 MPI_F_SYNC_REG(X)
 MPI_WIN_SYNC(win)    
 MPI_SEND                MPI_RECV
                         MPI_WIN_SYNC(win)
                         MPI_F_SYNC_REG(X)

                         print X

                         MPI_F_SYNC_REG(X)
 MPI_RECV                MPI_SEND
 MPI_F_SYNC_REG(X)                       
END DO                  END DO

MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win)

No change-log entry.

[mpiforumbot]

Originally by RolfRabenseifner on 2014-02-16 12:03:41 -0600

---

This ticket is the result of a discussion by Jeff Hammond, Pavan Balaji, Brian W Barrett, Dave Goodell, and me. It is implemented in an example and is tested with mpich.
The method with MPI_Win_sync was agreed by this group.

[mpiforumbot]

Originally by RolfRabenseifner on 2014-02-19 11:38:32 -0600

---

• I updated the text according to Jim Dinan's corrections
  • I substituted "reordering of the MPI_WIN_SYNC calls"
    by "reordering of the accesses to the variable X" because this describes
    all MPI_F_SYNC_REG calls
• I removed the two MPI_WIN_SYNC in the part after the "print X"
  (according to Jim Dinan's correct analysis that they are not needed)

[mpiforumbot]

Originally by RolfRabenseifner on 2014-02-20 09:44:39 -0600

---

Added advice as proposed by Jim Dinan.

[mpiforumbot]

Originally by rsthakur on 2014-02-20 15:11:30 -0600

---

I would make it clear that this advice to users is for shared memory.

"Advice to users. In the unified memory model, in the case where the window is in shared memory, MPI_WIN_SYNC can be used to order store operations and make store updates to the window visible to other processes and threads."

I would begin the example in the same format as the other examples:
"Example 16.13 The following example demonstrates..."

I would make it clear in the first sentence itself that is for shared memory windows.

"when a data transfer is implemented with load and store in the case of windows in shared memory (instead of MPI_PUT or MPI_GET)"

Do we want to make it clear that there is no equivalent to MPI_F_SYNC_REG in C. Not everyone reads the whole spec.

"In Fortran, reordering of the accesses to the variable X must be prevented with MPI_F_SYNC_REG operations. (No equivalent function is needed in C.)"

MPI_Send/Recv are not in caps, others are. Do we want to be consistent.

[mpiforumbot]

Originally by RolfRabenseifner on 2014-02-20 16:05:41 -0600

---

Replying to rsthakur:

I would make it clear that this advice to users is for shared memory.

"Advice to users. In the unified memory model, in the case where the window is in shared memory, MPI_WIN_SYNC can be used to order store operations and make store updates to the window visible to other processes and threads."

Done.

I would begin the example in the same format as the other examples:
"Example 16.13 The following example demonstrates..."

Done.

I would make it clear in the first sentence itself that is for shared memory windows.

"when a data transfer is implemented with load and store in the case of windows in shared memory (instead of MPI_PUT or MPI_GET)"

Done.

Do we want to make it clear that there is no equivalent to MPI_F_SYNC_REG in C. Not everyone reads the whole spec.

"In Fortran, reordering of the accesses to the variable X must be prevented with MPI_F_SYNC_REG operations. (No equivalent function is needed in C.)"

Done.
Additionally, I added

"In Fortran, if MPI_ASYNC_PROTECTS_NONBLOCKING is .FALSE. or the
variable X is not declared as ASYNCHRONOUS,"

MPI_Send/Recv are not in caps, others are. Do we want to be consistent.

Done.

[mpiforumbot]

Originally by RolfRabenseifner on 2014-05-19 11:52:41 -0500

---

May 7, 2014, Bill Gropp wrote in an email:

Rolf,

The RMA chapter committee will be adding this example. I'm not convinced that the suggested location is the correct place, and will be looking into it more. As an example to illuminate existing functionality, it isn't covered by the "technical additions" requirement.

Bill

[mpiforumbot]

Originally by jsquyres on 2014-06-05 10:29:29 -0500

---

Ticket passed errata vote: 16 yes, 1 no, 2 abstain

[mpiforumbot]

Originally by gropp on 2015-02-12 17:24:52 -0600

---

Committed as r1965