Pythonization of LaTeX bindings

[jsquyres]

Problem

The MPI API bindings are currently hard-coded five times in the LaTeX for the MPI standard:

1. Language independent specification (LIS)
2. C
3. C++ (yes, these are still in the LaTeX ⚠️)
4. Fortran 90
5. Fortran 08

We propose to convert this 5x-hard-coding into a single specification of each API in Python, and then automatically render each of the 4 desired bindings in LaTeX (i.e., it's no longer necessary to render the C++ bindings).

Proposal

As stated above, convert all hard-coded LaTeX MPI API bindings to a block of Python in the .tex files that are then rendered to LaTeX during make.

We have a wiki setup for Chapter Authors, including:

1. Rationale / benefits of this scheme
2. Help for those who do not know Python

This scheme was detailed in:

• The 9 Dec 2019 MPI Forum Physical meeting. See the slides
• The 13 Nov 2019 MPI Forum Virtual meeting. See the slides.

Note that while this effort is related to the Embiggenment (a.k.a. "Large Count"), it is a separate effort. Meaning:

1. This effort will create a more-or-less comparible LaTeX to MPI-3.1
   • See the slides for a list of differences compared to MPI-3.1
2. Embiggenment depends on this Pythonization. I.e., Embiggenment will likely tweak the Python rendering engine to also generate new interfaces for "large count" parameters.

Changes to the Text

This touches nearly every chapter in the MPI specification document. Here's an example of one change -- for MPI_Send.

From this (note the extremely-long lines -- scroll to the right):

\begin{funcdef}{MPI\_SEND(buf, count, datatype, dest, tag, comm)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements in send buffer (non-negative integer)}
\funcarg{\IN}{datatype}{datatype of each send buffer element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\end{funcdef}

\cdeclmainindex{MPI\_Comm}%
\mpibind{MPI\_Send(const~void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm)}

\mpifnewbind{MPI\_Send(buf, count, datatype, dest, tag, comm, ierror) \fargs TYPE(*), DIMENSION(..), INTENT(IN) :: buf \\ INTEGER, INTENT(IN) :: count, dest, tag \\ TYPE(MPI\_Datatype), INTENT(IN) :: datatype \\ TYPE(MPI\_Comm), INTENT(IN) :: comm \\ INTEGER, OPTIONAL, INTENT(OUT) :: ierror}
\mpifbind{MPI\_SEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)\fargs <type> BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, IERROR}
\mpicppemptybind{MPI::Comm::Send(const void*~buf, int~count, const MPI::Datatype\&~datatype, int~dest, int~tag) const}{void}

To this:

\cdeclmainindex{MPI\_Comm}
\begin{mpi-binding}
    function_name('MPI_Send')

    parameter('buf', 'BUFFER', desc='initial address of send buffer', constant=True)
    parameter('count', 'POLYXFER_NUM_ELEM', desc='number of elements in send buffer')
    parameter('datatype', 'DATATYPE', desc='datatype of each send buffer element')
    parameter('dest', 'RANK', desc='rank of destination')
    parameter('tag', 'TAG', desc='message tag')
    parameter('comm', 'COMMUNICATOR')
\end{mpi-binding}

Impact on Implementations

None.

Impact on Users

None.

References

• Corresponding PR: https://github.com/mpi-forum/mpi-standard/pull/152
• Wiki for Chapter Authors -- describing how this will affect them, examples to show how to Pythonize, etc.
• Slides from 13 Nov 2019 MPI Forum Virtual meeting
• MPI Forum issue for The Embiggenment
• Pull Request for the Pythonization effort -- on the Large Count WG repo

[jsquyres]

Here is a copy of the "cleandoc" PDF from the Pythonization PR as of 2019-11-26. The Pythonization is not yet complete, and we have not yet verified all the converted bindings. But it gives a good example of what the Pythonization process is doing.

mpi-report-pythonization-as-of-2019-11-26.pdf

Most notably, the following chapters are not yet complete:

• Chapter 5: collectives
• Chapter 7: process topologies (partially completed)
• Chapter 11: one sided
• Chapter 13: io

[omor1]

An interesting note is that this could allow MPI to relatively simply adopt future bindings if necessary by simply adding them as additional backends. Obviously not something to be looked at in this juncture, but some food for thought.

[jsquyres]

An interesting note is that this could allow MPI to relatively simply adopt future bindings if necessary by simply adding them as additional backends.

That is definitely in the plan for the future. 😄

[jsquyres]

Here is a copy of the "cleandoc" PDF from the Pythonization PR as of 2019-12-09. It is all but complete; it needs review by chapter committees.

mpi-report-pythonized-2019-12-09.pdf

[jsquyres]

Here is a copy of the "cleandoc" PDF from the Pythonization PR as of 2020-02-03 for reading at the Feb 2020 Portland MPI Forum meeting.

https://aws.open-mpi.org/~jsquyres/forum-unofficial/mpi-report-2020-02-03-f876294-pythonized-NOT-embiggened.pdf

[jjhursey]

A ticket0 suggestion. Can we change F to Fortran and F08 to Fortran 2008 in the header to the Fortran bindings?

Rationale:

• There is an F programming language link which is related to Fortan, but distinct from it (from what I can tell).
• We should be as precise as practical about the programming language to which the binding applies.
• It also makes Fortran searchable through the document and matching to the interfaces.

[jsquyres]

Done -- new (trivial) commit is now on this PR. I will present at the 2020-02 Portland Forum meeting.

New PDF: https://aws.open-mpi.org/~jsquyres/forum-unofficial/mpi-report-2020-02-19-eba417f-pythonized-NOT-emibggened.pdf

[wesbland]

This proposal passed the first and only vote at the February 2020 meeting:

Yes - 24
No - 0
Abstain - 0

[martinruefenacht]

Newest PDF:
mpi40-report.pdf

This fixes:

• None return in F08 generator for VARARG kinds
• Escaping spaces for uniform spacing behaviour
• Line breaking within LIS parameter names
• Autogenerates "For this routine, an interface within the mpi_f08 module was never defined." for bindings which require this proxy rendering of F08 bindings.

Not yet fixed, because they are Latex changes (Bill):

• Indentation of line breaking F08 descriptors to ensure typing clarity
• "Fortran 2008 Binding" for proxy rendered (above)
  • Need to modify "mpifnewnonebind" latex macro