Skip to content

Latest commit

 

History

History
67 lines (49 loc) · 3.81 KB

mpi.rst

File metadata and controls

67 lines (49 loc) · 3.81 KB
Raw

MPI

Collective Behavior

openPMD-api is designed to support both serial as well as parallel I/O. The latter is implemented through the Message Passing Interface (MPI).

A collective operation needs to be executed by all MPI ranks of the MPI communicator that was passed to openPMD::Series. Contrarily, independent operations can also be called by a subset of these MPI ranks. For more information, please see the MPI standard documents, for example MPI-3.1 in "Section 2.4 - Semantic Terms".

Functionality Behavior Description
Series collective open and close
::flush() collective read and write
Iteration1 independent declare and open
::open()2 collective explicit open
Mesh3 independent declare, open, write
ParticleSpecies4 independent declare, open, write
::setAttribute5 backend-specific declare, write
::getAttribute independent open, reading
RecordComponent6 independent declare, open, write
::resetDataset78 backend-specific declare, write
::makeConstant9 backend-specific declare, write
::storeChunk10 independent write
::loadChunk independent read
::availableChunks11 collective read, immediate result

Tip

Just because an operation is independent does not mean it is allowed to be inconsistent. For example, undefined behavior will occur if ranks pass differing values to ::setAttribute or try to use differing names to describe the same mesh.

Efficient Parallel I/O Patterns

Note

This section is a stub. We will improve it in future versions.

Write as large data set chunks as possible in ::storeChunk operations.

Read in large, non-overlapping subsets of the stored data (::loadChunk). Ideally, read the same chunk extents as were written, e.g. through ParticlePatches (example to-do).

See the implemented I/O backends <backends-overview> for individual tuning options.

  1. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  2. We usually open iterations delayed on first access. This first access is usually the flush() call after a storeChunk/loadChunk operation. If the first access is non-collective, an explicit, collective Iteration::open() can be used to have the files already open. Alternatively, iterations might be accessed for the first time by immediate operations such as ::availableChunks().↩

  3. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  4. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  5. HDF5 <backends-hdf5> only supports collective attribute definitions/writes; ADIOS1 <backends-adios1> and ADIOS2 <backends-adios2> attributes can be written independently. If you want to support all backends equally, treat as a collective operation. Note that openPMD represents constant record components with attributes, thus inheriting this for ::makeConstant.↩

  6. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  7. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  8. Dataset declarations in parallel HDF5 <backends-hdf5> are only non-collective if chunking <backendconfig-hdf5> is set to none (auto by default). Otherwise these operations are collective.↩

  9. HDF5 <backends-hdf5> only supports collective attribute definitions/writes; ADIOS1 <backends-adios1> and ADIOS2 <backends-adios2> attributes can be written independently. If you want to support all backends equally, treat as a collective operation. Note that openPMD represents constant record components with attributes, thus inheriting this for ::makeConstant.↩

  10. Individual backends, i.e. parallel HDF5 <backends-hdf5>, will only support independent operations if the default, non-collective (aka independent) behavior is kept. Otherwise these operations are collective.↩

  11. We usually open iterations delayed on first access. This first access is usually the flush() call after a storeChunk/loadChunk operation. If the first access is non-collective, an explicit, collective Iteration::open() can be used to have the files already open. Alternatively, iterations might be accessed for the first time by immediate operations such as ::availableChunks().↩