Skip to content

Latest commit

 

History

History
142 lines (115 loc) · 5.38 KB

frameworks.rst

File metadata and controls

142 lines (115 loc) · 5.38 KB

Internal frameworks

The Modular Component Architecture (MCA) is the backbone of Open MPI -- most services and functionality are implemented through MCA components.

MPI layer frameworks

Here is a list of all the component frameworks in the MPI layer of Open MPI:

  • bml: BTL management layer
  • coll: MPI collective algorithms
  • fbtl: point to point file byte transfer layer: abstraction for individual read: collective read and write operations for MPI I/O
  • fcoll: collective file system functions for MPI I/O
  • fs: file system functions for MPI I/O
  • hook: Generic hooks into Open MPI
  • io: MPI I/O
  • mtl: Matching transport layer, used for MPI point-to-point messages on some types of networks
  • op: Back end computations for intrinsic MPI_Op operators
  • osc: MPI one-sided communications
  • pml: MPI point-to-point management layer
  • sharedfp: shared file pointer operations for MPI I/O
  • topo: MPI topology routines
  • vprotocol: Protocols for the "v" PML

OpenSHMEM component frameworks

  • atomic: OpenSHMEM atomic operations
  • memheap: OpenSHMEM memory allocators that support the PGAS memory model
  • scoll: OpenSHMEM collective operations
  • spml: OpenSHMEM "pml-like" layer: supports one-sided, point-to-point operations
  • sshmem: OpenSHMEM shared memory backing facility

Miscellaneous frameworks

  • allocator: Memory allocator
  • backtrace: Debugging call stack backtrace support
  • btl: Point-to-point Byte Transfer Layer
  • dl: Dynamic loading library interface
  • hwloc: Hardware locality (hwloc) versioning support
  • if: OS IP interface support
  • installdirs: Installation directory relocation services
  • memchecker: Run-time memory checking
  • memcpy: Memory copy support
  • memory: Memory management hooks
  • mpool: Memory pooling
  • patcher: Symbol patcher hooks
  • pmix: Process management interface (exascale)
  • rcache: Memory registration cache
  • reachable: Network reachability determination
  • shmem: Shared memory support (NOT related to OpenSHMEM)
  • smsc: Shared memory single-copy support
  • threads: OS and userspace thread support
  • timer: High-resolution timers

Framework notes

Each framework typically has one or more components that are used at run-time. For example, the btl framework is used by the MPI layer to send bytes across different types underlying networks. The tcp btl, for example, sends messages across TCP-based networks; the ucx pml sends messages across InfiniBand-based networks.

MCA parameter notes

Each component typically has some tunable parameters that can be changed at run-time. Use the :ref:`ompi_info(1) <man1-ompi_info>` command to check a component to see what its tunable parameters are. For example:

shell$ ompi_info --param btl tcp

shows some of the parameters (and default values) for the tcp btl component (use --all or --level 9 to show all the parameters).

Note that ompi_info (without --all or a specified level) only shows a small number a component's MCA parameters by default. Each MCA parameter has a "level" value from 1 to 9, corresponding to the MPI-3 MPI_T tool interface levels. :ref:`See the LEVELS section in the ompi_info(1) man page <man1-ompi_info-levels>` for an explanation of the levels and how they correspond to Open MPI's code.

Here's rules of thumb to keep in mind when using Open MPI's levels:

  • Levels 1-3:
    • These levels should contain only a few MCA parameters.
    • Generally, only put MCA parameters in these levels that matter to users who just need to run Open MPI applications (and don't know/care anything about MPI). Examples (these are not comprehensive):
      • Selection of which network interfaces to use.
      • Selection of which MCA components to use.
      • Selective disabling of warning messages (e.g., show warning message XYZ unless a specific MCA parameter is set, which disables showing that warning message).
      • Enabling additional stderr logging verbosity. This allows a user to run with this logging enabled, and then use that output to get technical assistance.
  • Levels 4-6:
    • These levels should contain any other MCA parameters that are useful to expose to end users.
    • There is an expectation that "power users" will utilize these MCA parameters |mdash| e.g., those who are trying to tune the system and extract more performance.
    • Here's some examples of MCA parameters suitable for these levels (these are not comprehensive):
      • When you could have hard-coded a constant size of a resource (e.g., a resource pool size or buffer length), make it an MCA parameter instead.
      • When there are multiple different algorithms available for a particular operation, code them all up and provide an MCA parameter to let the user select between them.
  • Levels 7-9:
    • Put any other MCA parameters here.
    • It's ok for these MCA parameters to be esoteric and only relevant to deep magic / the internals of Open MPI.
    • There is little expectation of users using these MCA parameters.

See :ref:`this section <label-running-setting-mca-param-values>` for details on how to set MCA parameters at run time.