Explicitly encourage routines for "good" one-sided memory for all memory types

[mpiforumbot]

Originally by schulzm on 2008-10-20 12:47:36 -0500

---

Explicitly encourage routines for "good" one-sided memory for all memory types

Author: Martin Schulz

Description

The following advice to implementors can be found on page 323, line 27-40, Section 11.2:

''Advice to implementors.[[BR]]
In cases where RMA operations use different mechanisms
in different memory areas (e.g., load/store in a shared memory segment, and an asynchronous
handler in private memory), the MPI_WIN_CREATE call needs to figure out
which type of memory is used for the window. To do so, MPI maintains, internally, the
list of memory segments allocated by MPI_ALLOC_MEM, or by other, implementation
specific, mechanisms, together with information on the type of memory segment
allocated. When a call to MPI_WIN_CREATE occurs, then MPI checks which segment
contains each window, and decides, accordingly, which mechanism to use for RMA
operations.[[BR]]
Vendors may provide additional, implementation-specific mechanisms to allow “good”
memory to be used for static variables.[[BR]]
Implementors should document any performance impact of window alignment.[[BR]](End of advice to implementors.)''

Line 36 and 37 explicitly restricts this to static variables, which seems artificial
to me and I propose to remove this restriction. However, the original intent to
point out that this is a special problem for static variables, should be maintained
(although, other memory regions - e.g., heap data not allocated inside a library,
but used by a library for one-sided communication - could also benefit from this
as well).

History

Proposed Solution

Replace the advice to implementors on page 323 with (this including reviewer comments)

''Advice to implementors.[[BR]]
In cases where RMA operations use different mechanisms
in different memory areas (e.g., load/store in a shared memory segment, and an asynchronous
handler in private memory), the MPI_WIN_CREATE call needs to figure out
which type of memory is used for the window. To do so, MPI maintains, internally, the
list of memory segments allocated by MPI_ALLOC_MEM, or by other, implementation
specific, mechanisms, together with information on the type of memory segment
allocated. When a call to MPI_WIN_CREATE occurs, then MPI checks which segment
contains each window, and decides, accordingly, which mechanism to use for RMA
operations.[[BR]]
Vendors may provide additional, implementation-specific mechanisms to allow “good”
memory to be used for static variables.
allocate or to specify memory regions that are preferable for use in one sided communication. In particular, such mechanisms can be used to place static variables into such preferred regions.
Implementors should document any performance impact of window alignment. [[BR]](End of advice to implementors.)''

Impact on Implementations

None / all optional

Impact on Applications / Users

None / all optional

Alternative Solutions

Entry for the Change Log

[mpiforumbot]

Originally by schulzm on 2008-12-15 18:59:12 -0600

---

Needs review by chapter author (Jesper Larsson Traeff)

[mpiforumbot]

Originally by bronis on 2008-12-23 11:05:01 -0600

---

The suggested rewording is confusing and appears overly restrictive. Static variables can be used at the source without requiring they are in a special location. The problem arises with using static variables at a target, where you cannot otherwise ensure they are still in scope. I suggest you reword the new sentence:

In particular, such routines can be used to enable the use of static variables in one sided communication operations.

To read:

In particular, such routines can enable the use of static variables at the targets of one sided communication operations.

[mpiforumbot]

Originally by RolfRabenseifner on 2009-01-18 12:36:14 -0600

---

• You may format the proposal as usual:
  [[BR]]
  MPI-2.1 p323:36-37 reads

  Vendors may provide additional, implementation-specific mechanisms to allow “good” memory~~ to be used for static variables~~.

  but should read

  Vendors may provide additional, implementation-specific mechanisms to allow “good” memory. In particular, such routines can be used to enable the use of static variables in one sided communication operations.

• According to Ticket Change-Log is also used for important clarifications for the MPI users. #99, you may want to add a Change-Log entry.

• You may add additional wishes for review in the CC field.

[mpiforumbot]

Originally by gropp on 2009-02-06 12:19:25 -0600

---

I don't like the text - with the deletion, the good memory is for what? This needs to be re-written to explain for what the "good memory" is being used.

[mpiforumbot]

Originally by schulzm on 2009-02-09 01:22:30 -0600

---

Adding Bronis's and Bill's comments together, here is a new suggestion:

• MPI-2.1 p323:36-37 reads

  Vendors may provide additional, implementation-specific mechanisms to
  allow “good” memory to be used for static variables.

• New suggest text for MPI-2.2:

  Vendors may provide additional, implementation-specific mechanisms to allocate or specify memory regions that be used as targets for one sided communication. In particular, such routines can enable the use of static variables at the targets of one sided communication operations.

-Changelog*: I don't think this change is major enough to need a change log
entry. The semantics of MPI is not changed - this is only a clarifiction about
additional routines not in the MPI standard.

Added Bill to reviewers to get to the necessary 3+1.

[mpiforumbot]

Originally by bronis on 2009-02-09 06:35:38 -0600

---

Please make the change in the proposed solution. When you do, please make it read:

Vendors may provide additional, implementation-specific mechanisms to allocate or to specify memory regions that be used as targets for one sided communication. In particular, such routines can enable the use of static variables as the targets of one sided communication operations.

The above fixes a split infinitive in your new text and a small typo in mine ("at" => "as").

[mpiforumbot]

Originally by schulzm on 2009-02-09 19:09:19 -0600

---

Changed text based on Bronis's comments

[mpiforumbot]

Originally by bronis on 2009-02-09 19:29:11 -0600

---

Looks good to me.

[mpiforumbot]

Originally by traff on 2009-02-09 19:59:53 -0600

---

seems ok to me

[mpiforumbot]

Originally by moody20 on 2009-02-10 12:18:32 -0600

---

So just to clarify my understanding of this issue. I can understand that certain memory regions may be better than others for one-sided communication. For dynamic memory allocation, the implementation can provide segments of this "fast" memory via MPI_ALLOC_MEM over the "slow" memory provided by malloc. I think the original advice about static variables is there to tell vendors that they may provide a way for users to locate static variables in such "fast" memory during the compile, perhaps via some special compiler keyword or #pragma.

The proposed text mentions "such routines" to specify static regions as targets for windows, which sounds like something at runtime to me. I think the intent of the original wording was to enable the compiler to place static variables in "fast" memory during the build. We could keep the new text, but we should also keep the hints about compiler options.

[mpiforumbot]

Originally by schulzm on 2009-02-10 12:29:51 -0600

---

It's not an issue of fast vs. slow access, but whether memory regions are
known to and accessible by remote processes via operations like RMA. This
typically requires locking the memory pages to avoid that their physical
location changes and this is a runtime issue. A compiler pragma would only
work if you have a specific MPI compiler that would call such an implementation
specific routine to lock the corresponding pages.

[mpiforumbot]

Originally by moody20 on 2009-02-10 12:40:33 -0600

---

Hmm, I'm not sure because there is an Advice to Users sections just above the Advice to Implementors that talks about performance regarding memory allocated via MPI_ALLOC_MEM vs something else (e.g., malloc).

I think this sentence in the Advice to Implementors is getting at the same thing:

"Vendors may provide additional, implementation-specific mechanisms to allow "good" memory to be used for static variables."

I still think this is addressing placement of data in "fast" memory for performance.

[mpiforumbot]

Originally by schulzm on 2009-02-10 12:50:17 -0600

---

OK, I see the point - I think the comment refers to the fact that in "bad" memory
one as to fall back to messages in the background and can not use RMA type
operations anymore (hence the lower performance), but that goes way beyond of what
we want to talk about here.

I agree, though, that there is no need to be specific about whether those are
routines. I replaced "routines" with "mechanisms", which opens the possibilities
for any type, including runtime routines and compiler pragmas. Would this be more
acceptable?

[mpiforumbot]

Originally by schulzm on 2009-02-10 13:33:54 -0600

---

included Adam's feedback that this might not only affect targets, but memory
placement in general

[mpiforumbot]

Originally by bronis on 2009-02-10 14:00:26 -0600

---

A grammar error was introduced in the most recent change. It is also wordy. Please change "allocate or to specify “good” memory regions that preferable to be used as sources or targets for one sided communication" to "allocate or to specify “good” memory regions that are preferable for sources or targets of one sided communication"

[mpiforumbot]

Originally by schulzm on 2009-02-10 14:04:36 -0600

---

agreed and changed

[mpiforumbot]

Originally by moody20 on 2009-02-10 14:08:25 -0600

---

How about changing:

allocate or to specify “good” memory regions that preferable to be used as sources or targets for one sided communication.

to (can drop "good" and the text is missing the verb "are" and can drop "sources" and "targets"):

allocate or to specify memory regions that are preferable for use in one sided communication.

[mpiforumbot]

Originally by schulzm on 2009-02-10 14:13:32 -0600

---

OK, changed - I didn't like the "good" in the first place and also Bill had
some reservations against it

[mpiforumbot]

Originally by moody20 on 2009-02-10 14:18:45 -0600

---

Looks good to me. I won't fight too hard if folks would like to add "sources and targets" back, but I don't think they are necessary here (since we mention both).

[mpiforumbot]

Originally by bronis on 2009-02-10 14:24:50 -0600

---

Looks fine to me.

[mpiforumbot]

Originally by schulzm on 2009-02-10 14:33:22 -0600

---

added reviewer

[mpiforumbot]

Originally by traff on 2009-02-10 14:59:04 -0600

---

better, and fine with me

[mpiforumbot]

Originally by koop on 2009-02-10 15:28:27 -0600

---

Looks fine to me.

[mpiforumbot]

Originally by schulzm on 2009-02-10 15:30:43 -0600

---

This proposal had the four required reviews - updating the state

[mpiforumbot]

Originally by traff on 2009-07-13 10:37:52 -0500

---

Chapter author (Jesper) has updated, reviewers please check

Note: did CHANGE "one sided" to "one-sided", in order to be consistent - this should be mentioned to Forum, but ought not to cause problems.

[mpiforumbot]

Originally by RolfRabenseifner on 2009-07-13 12:24:42 -0500

---

Obviously implemented. Therefore priority changed to "Waiting for PDF reviews".

PDF-review: Seems to be not okay:

• The sentence

  Vendors may provide additional, implementation-specific mechanisms to
  allow “good” memory to be used for static variables.

  should be removed.

• The last sentence should not be a new paragraph.

Therefore, prority kept as "Passed".

[mpiforumbot]

Originally by traff on 2009-07-14 08:26:16 -0500

---

Sentence has been deleted (the delete macro was there, but again, I forgot to comment
out the text to be deleted).

Formatting has been kept, new sentences as paragraphs. This is they way the text looks
in the current 2.1 document. Ticket authors should comment! All reviews, please check

[mpiforumbot]

Originally by traff on 2009-07-14 08:26:37 -0500

---

Attachment added: ticket53.pdf (28.9 KiB)

[mpiforumbot]

Originally by RolfRabenseifner on 2009-07-16 03:09:13 -0500

---

PDF-review: Okay.

You kept the two paragraphs "Vendors ..." and Implementors ...".
I expect that it was not the intent of this ticket to combine these two paragraphs although the text of the solution does show such a concatenation.

Therefore, I would say, pdf is okay.

[mpiforumbot]

Originally by gropp on 2009-07-22 10:53:48 -0500

---

Updated the proposed solution to conform to the standard (include the page number)

[mpiforumbot]

Originally by gropp on 2009-07-22 11:41:07 -0500

---

WDG - Reviewed and complete.

[mpiforumbot]

Originally by schulzm on 2009-07-26 17:02:57 -0500

---

As the original ticket author: the PDF looks good to me as well.

[mpiforumbot]

Originally by rsthakur on 2009-08-27 19:01:25 -0500

---

pdf ok. marking as complete

[mpiforumbot]

Originally by jsquyres on 2010-09-18 05:01:45 -0500

---

This ticket is (long-since) complete; marking it resolved/text committed.