Skip to content

Latest commit

 

History

History
67 lines (49 loc) · 3.81 KB

mpi.rst

File metadata and controls

67 lines (49 loc) · 3.81 KB
Raw

MPI

Collective Behavior

openPMD-api is designed to support both serial as well as parallel I/O. The latter is implemented through the Message Passing Interface (MPI).

A collective operation needs to be executed by all MPI ranks of the MPI communicator that was passed to openPMD::Series. Contrarily, independent operations can also be called by a subset of these MPI ranks. For more information, please see the MPI standard documents, for example MPI-3.1 in "Section 2.4 - Semantic Terms".

Functionality Behavior Description
Series collective open and close
::flush() collective read and write
Iteration1 independent declare and open
::open()2 collective explicit open
Mesh3 independent declare, open, write
ParticleSpecies4 independent declare, open, write
::setAttribute5 backend-specific declare, write
::getAttribute independent open, reading
RecordComponent6 independent declare, open, write
::resetDataset78 backend-specific declare, write
::makeConstant9 backend-specific declare, write
::storeChunk10 independent write
::loadChunk independent read
::availableChunks11 collective read, immediate result

Alternatively, iterations might be accessed for the first time by immediate operations such as ::availableChunks().

Tip

Just because an operation is independent does not mean it is allowed to be inconsistent. For example, undefined behavior will occur if ranks pass differing values to ::setAttribute or try to use differing names to describe the same mesh.

Efficient Parallel I/O Patterns

Note

This section is a stub. We will improve it in future versions.

Write as large data set chunks as possible in ::storeChunk operations.

Read in large, non-overlapping subsets of the stored data (::loadChunk). Ideally, read the same chunk extents as were written, e.g. through ParticlePatches (example to-do).

See the implemented I/O backends <backends-overview> for individual tuning options.

  1. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  2. We usually open iterations delayed on first access. This first access is usually the flush() call after a storeChunk/loadChunk operation. If the first access is non-collective, an explicit, collective Iteration::open() can be used to have the files already open.↩

  3. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  4. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  5. HDF5 <backends-hdf5> only supports collective attribute definitions/writes; ADIOS1 <backends-adios1> and ADIOS2 <backends-adios2> attributes can be written independently. If you want to support all backends equally, treat as a collective operation. Note that openPMD represents constant record components with attributes, thus inheriting this for ::makeConstant.↩

  6. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  7. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  8. Dataset declarations in parallel HDF5 <backends-hdf5> are only non-collective if chunking <backendconfig-hdf5> is set to none (auto by default). Otherwise these operations are collective.↩

  9. HDF5 <backends-hdf5> only supports collective attribute definitions/writes; ADIOS1 <backends-adios1> and ADIOS2 <backends-adios2> attributes can be written independently. If you want to support all backends equally, treat as a collective operation. Note that openPMD represents constant record components with attributes, thus inheriting this for ::makeConstant.↩

  10. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  11. We usually open iterations delayed on first access. This first access is usually the flush() call after a storeChunk/loadChunk operation. If the first access is non-collective, an explicit, collective Iteration::open() can be used to have the files already open.↩