From 19f058440b88a1135befc8cc04869987709cf146 Mon Sep 17 00:00:00 2001 From: James Dinan Date: Mon, 23 Jan 2017 10:26:59 -0500 Subject: [PATCH 1/8] Clarify init-finalize pairing Remove passage from API notes that contradicts the following semantic from the API description of shmem_init: "At the end of the OpenSHMEM program which it initialized, the call to shmem_init must be matched with a call to shmem_finalize." Signed-off-by: James Dinan --- content/shmem_init.tex | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/content/shmem_init.tex b/content/shmem_init.tex index 065493f90..1f4f5e32d 100644 --- a/content/shmem_init.tex +++ b/content/shmem_init.tex @@ -36,10 +36,8 @@ As of \openshmem Specification 1.2 the use of \FUNC{start\_pes} has been deprecated and is replaced with \FUNC{shmem\_init}. While support for \FUNC{start\_pes} is still required in \openshmem libraries, users are - encouraged to use \FUNC{shmem\_init}. Replacing \FUNC{start\_pes} with - \FUNC{shmem\_init} in \openshmem programs with no further changes is possible; - there is an implicit \FUNC{shmem\_finalize} at the end of main. However, - \FUNC{shmem\_init} differs slightly from \FUNC{start\_pes}: multiple calls to + encouraged to use \FUNC{shmem\_init}. An important difference between + \FUNC{shmem\_init} and \FUNC{start\_pes} is that multiple calls to \FUNC{shmem\_init} within a program results in undefined behavior, while in the case of \FUNC{start\_pes}, any subsequent calls to \FUNC{start\_pes} after the first one resulted in a no-op. From 09d3b9f7b0c9d728ca67f98df8806bb905d8ff9c Mon Sep 17 00:00:00 2001 From: James Dinan Date: Mon, 23 Jan 2017 10:47:02 -0500 Subject: [PATCH 2/8] Add changelog entry for shmem_init clarification Signed-off-by: James Dinan --- content/backmatter.tex | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/backmatter.tex b/content/backmatter.tex index 62104bc54..b2295aa7a 100644 --- a/content/backmatter.tex +++ b/content/backmatter.tex @@ -440,6 +440,10 @@ \section{Version 1.5} \begin{itemize} % +\item Clarified that \FUNC{shmem\_init} must be matched with a call to + \FUNC{shmem\_finalize}. +\\See Section \ref{subsec:shmem_init}. +% \item The \VAR{SHMEM\_SYNC\_SIZE} constant was added. \\See Section \ref{subsec:library_constants}. % From 17247977ef32b83fd4f00a3d39ce3242a2e2f69e Mon Sep 17 00:00:00 2001 From: James Dinan Date: Wed, 8 Mar 2017 10:48:38 -0500 Subject: [PATCH 3/8] Update changelog entry to ref shmem_finalize Signed-off-by: James Dinan --- content/backmatter.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/backmatter.tex b/content/backmatter.tex index b2295aa7a..e3c82fb91 100644 --- a/content/backmatter.tex +++ b/content/backmatter.tex @@ -442,7 +442,7 @@ \section{Version 1.5} % \item Clarified that \FUNC{shmem\_init} must be matched with a call to \FUNC{shmem\_finalize}. -\\See Section \ref{subsec:shmem_init}. +\\See Sections \ref{subsec:shmem_init} and \ref{subsec:shmem_finalize}. % \item The \VAR{SHMEM\_SYNC\_SIZE} constant was added. \\See Section \ref{subsec:library_constants}. From cbd1a268515a34b856fd5be307b3fcd61c8aeab8 Mon Sep 17 00:00:00 2001 From: James Dinan Date: Wed, 8 Mar 2017 10:51:30 -0500 Subject: [PATCH 4/8] Update Annex A reference to finalize Signed-off-by: James Dinan --- content/backmatter.tex | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/content/backmatter.tex b/content/backmatter.tex index e3c82fb91..6ca8e65e1 100644 --- a/content/backmatter.tex +++ b/content/backmatter.tex @@ -23,11 +23,9 @@ \section*{Incorporating \openshmem{} into Programs}\label{sec:writing_programs} \begin{itemize} \item Add the include file shmem.h (for \Clang) or shmem.fh (for \Fortran). \item Add the initialization call \FUNC{shmem\_init}, (line 9). -\item Use OpenSHMEM calls to query the the total number of PEs (line 10) and PE +\item Use OpenSHMEM calls to query the total number of PEs (line 10) and PE id (line 11). -\item There is no explicit finalize call; either a return from \texttt{main()} - (line 13) or an explicit \texttt{exit()} acts as an implicit \openshmem - finalization. +\item Add the finalization call \FUNC{shmem\_finalize}, (line 13). \item In \openshmem the order in which lines appear in the output is not fixed as \ac{PE}s execute asynchronously in parallel. \end{itemize} From b4590fb73f322b078a432e3632e673cf64068491 Mon Sep 17 00:00:00 2001 From: James Dinan Date: Wed, 8 Mar 2017 10:53:19 -0500 Subject: [PATCH 5/8] Remove implicit finalize text from API notes Signed-off-by: James Dinan --- content/shmem_finalize.tex | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/content/shmem_finalize.tex b/content/shmem_finalize.tex index cf0ab98b1..156ec78ac 100644 --- a/content/shmem_finalize.tex +++ b/content/shmem_finalize.tex @@ -44,11 +44,7 @@ \FUNC{shmem\_ptr}. This collective operation requires all \acp{PE} to participate in the call, not just a subset of the \acp{PE}. The non-\openshmem portion of a program may continue after a call to - \FUNC{shmem\_finalize} by all \acp{PE}. There is an implicit - \FUNC{shmem\_finalize} at the end of main, so that having an explicit call - to \FUNC{shmem\_finalize} is optional. However, an explicit - \FUNC{shmem\_finalize} may be required as an entry point for wrappers used - by profiling or other tools that need to perform their own finalization. + \FUNC{shmem\_finalize} by all \acp{PE}. } \begin{apiexamples} From 854efdedaab756ca8d58dd1f20c5d447c76e1e61 Mon Sep 17 00:00:00 2001 From: James Dinan Date: Mon, 10 Apr 2017 13:32:59 -0400 Subject: [PATCH 6/8] Add implicit finalize to start_pes and dep. annex Signed-off-by: James Dinan --- content/backmatter.tex | 10 ++++++++-- content/execution_model.tex | 12 ++++-------- content/start_pes.tex | 9 +++++++-- 3 files changed, 19 insertions(+), 12 deletions(-) diff --git a/content/backmatter.tex b/content/backmatter.tex index 7aaf64eaf..062a32913 100644 --- a/content/backmatter.tex +++ b/content/backmatter.tex @@ -23,7 +23,7 @@ \section*{Incorporating \openshmem{} into Programs}\label{sec:writing_programs} \begin{itemize} \item Add the include file \HEADER{shmem.h} (for \Cstd) or \HEADER{shmem.fh} (for \Fortran). \item Add the initialization call \FUNC{shmem\_init}, (line 9). -\item Use OpenSHMEM calls to query the total number of PEs (line 10) and PE +\item Use OpenSHMEM calls to query the total number of \acp{PE} (line 10) and \ac{PE} id (line 11). \item Add the finalization call \FUNC{shmem\_finalize}, (line 13). \item In \openshmem the order in which lines appear in the output is not fixed @@ -242,7 +242,7 @@ \section{\ac{MPI} Interoperability} Another important feature of these systems is that the \FUNC{shmem\_pe\_accessible} routine returns \CONST{TRUE} only if the remote -\ac{PE} is a process running from the same executable file as the local PE, +\ac{PE} is a process running from the same executable file as the local \ac{PE}, indicating that full \openshmem support (static memory and symmetric heap) is available. When using \openshmem routines within an \ac{MPI} program, the use of \ac{MPI} memory placement environment variables is required when using @@ -334,6 +334,7 @@ \section{Overview}\label{subsec:dep_overview} \hline \CorCpp: \hyperref[subsec:start_pes]{\FUNC{start\_pes}} & 1.2 & Current & \hyperref[subsec:shmem_init]{\FUNC{shmem\_init}} \\ \hline \Fortran: \hyperref[subsec:start_pes]{\FUNC{START\_PES}} & 1.2 & Current & \hyperref[subsec:shmem_init]{\FUNC{SHMEM\_INIT}} \\ \hline + \hyperref[subsec:start_pes]{Implicit finalization} & 1.2 & Current & \hyperref[subsec:shmem_finalize]{\FUNC{shmem\_finalize}} \\ \hline \CorCpp: \FUNC{\_my\_pe} & 1.2 & Current & \hyperref[subsec:shmem_my_pe]{\FUNC{shmem\_my\_pe}} \\ \hline \CorCpp: \FUNC{\_num\_pes} & 1.2 & Current & \hyperref[subsec:shmem_n_pes]{\FUNC{shmem\_n\_pes}} \\ \hline \Fortran: \FUNC{MY\_PE} & 1.2 & Current & \hyperref[subsec:shmem_my_pe]{\FUNC{SHMEM\_MY\_PE}} \\ \hline @@ -393,6 +394,11 @@ \subsection{\CorCpp: \_my\_pe, \_num\_pes, shmalloc, shfree, shrealloc, shmemali to normalize the \openshmem \ac{API} to use \shmemprefixLC{} as the standard prefix for all routines. +\subsection{Implicit Finalization} +Implicit finalization has been replaced with explicit finalization using the +\FUNC{shmem\_finalize} routine. Explicit finalization improves portability and +also improves interoperability with profiling and debugging tools. + \subsection{\Fortran: START\_PES, MY\_PE, NUM\_PES} The \Fortran routines \FUNC{START\_PES}, \FUNC{MY\_PE}, and \FUNC{NUM\_PES} were deprecated in order to minimize the API differences from the deprecation diff --git a/content/execution_model.tex b/content/execution_model.tex index 60b4a1fc3..ca36bc51b 100644 --- a/content/execution_model.tex +++ b/content/execution_model.tex @@ -6,14 +6,10 @@ start by calling the initialization routine \FUNC{shmem\_init} \footnote{\textbf{start\_pes} has been deprecated as of Specification 1.2} before using any of the other \openshmem library routines. An \openshmem -program finishes execution by returning from the main routine or when any PE -calls \FUNC{shmem\_global\_exit}. When returning from main, \openshmem must -complete all pending communication and release all the resources associated to -the library using an implicit collective synchronization across PEs. The user -has the option to call \FUNC{shmem\_finalize} (before returning from main) to -complete all pending communication and release all the \openshmem library -resources without terminating the program. Calling any \openshmem routine after -\FUNC{shmem\_finalize} leads to undefined behavior. +program concludes its use of the \openshmem library when all \acp{PE} call +\FUNC{shmem\_finalize} or when any \ac{PE} calls \FUNC{shmem\_global\_exit}. +Calling any \openshmem routine after \FUNC{shmem\_finalize} leads to undefined +behavior. The \acp{PE} of the \openshmem program are identified by unique integers. The identifiers are integers assigned in a monotonically increasing manner from zero diff --git a/content/start_pes.tex b/content/start_pes.tex index c8fa2a939..a99c61440 100644 --- a/content/start_pes.tex +++ b/content/start_pes.tex @@ -25,8 +25,13 @@ \apidescription{ The \FUNC{start\_pes} routine initializes the \openshmem execution - environment. An \openshmem program must call \FUNC{start\_pes} before - calling any other \openshmem routine. + environment. An \openshmem program must call either \FUNC{start\_pes} or + \FUNC{shmem\_init} before calling any other \openshmem routine. Unlike + \FUNC{shmem\_init}, \FUNC{start\_pes} does not require a call to + \FUNC{shmem\_finalize}. Instead, the \openshmem library is implicitly + finalized when the program exits. Implicit finalization is collective and + includes a global synchronization to ensure that all pending communication + is completed before resources are released. } \apireturnvalues{ From 53843d41998232c1651c21ac94b1254bc86f43cf Mon Sep 17 00:00:00 2001 From: James Dinan Date: Tue, 9 May 2017 20:41:42 -0400 Subject: [PATCH 7/8] Put back finalize description in exec model sec Signed-off-by: James Dinan --- content/execution_model.tex | 3 +++ 1 file changed, 3 insertions(+) diff --git a/content/execution_model.tex b/content/execution_model.tex index ca36bc51b..ae9a1bbc7 100644 --- a/content/execution_model.tex +++ b/content/execution_model.tex @@ -8,6 +8,9 @@ before using any of the other \openshmem library routines. An \openshmem program concludes its use of the \openshmem library when all \acp{PE} call \FUNC{shmem\_finalize} or when any \ac{PE} calls \FUNC{shmem\_global\_exit}. +During finalization, \openshmem must +complete all pending communication and release all the resources associated to +the library using an implicit collective synchronization across PEs. Calling any \openshmem routine after \FUNC{shmem\_finalize} leads to undefined behavior. From c1dfa16b7573f097cc8364bec3d19db02e6e1f72 Mon Sep 17 00:00:00 2001 From: James Dinan Date: Thu, 25 May 2017 15:41:19 -0400 Subject: [PATCH 8/8] Updates from 5/25/2017 reading Signed-off-by: James Dinan --- content/execution_model.tex | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/execution_model.tex b/content/execution_model.tex index ae9a1bbc7..571551a08 100644 --- a/content/execution_model.tex +++ b/content/execution_model.tex @@ -7,8 +7,9 @@ \footnote{\textbf{start\_pes} has been deprecated as of Specification 1.2} before using any of the other \openshmem library routines. An \openshmem program concludes its use of the \openshmem library when all \acp{PE} call -\FUNC{shmem\_finalize} or when any \ac{PE} calls \FUNC{shmem\_global\_exit}. -During finalization, \openshmem must +\FUNC{shmem\_finalize}. An \openshmem program can also be terminated +when any \ac{PE} calls \FUNC{shmem\_global\_exit}. +During a call to \FUNC{shmem\_finalize}, the \openshmem library must complete all pending communication and release all the resources associated to the library using an implicit collective synchronization across PEs. Calling any \openshmem routine after \FUNC{shmem\_finalize} leads to undefined