Home

This manual describes MANA (MPI-Agnostic Network-Agnostic Checkpointing) — transparent checkpoint-restart for MPI applications, implemented as a plugin on top of the DMTCP checkpointing package. DMTCP itself is bundled as a git submodule under dmtcp/, and the rest of the repository is the MANA plugin, its launcher scripts, and tests.

MANA layers on DMTCP rather than reimplementing it. For topics that are not MPI-specific — the DMTCP coordinator protocol, generic environment variables, the plugin event model, GDB attach mechanics, and so on — this manual refers to the document DMTCP_USER_MANUAL, which can be found at the DMTCP website, instead of duplicating its content. Sections that are MANA-specific (the split-process architecture, the MPI wrappers, the collective and point-to-point drain protocols, SLURM integration, the mana_* launcher scripts) are described in full here.

For a quick start, jump to Build System and then Usage Examples. For the design rationale and the split-process internals, see Appendix B: Architecture.

Project Overview

MANA provides transparent checkpoint-restart for MPI jobs:

MPI-Agnostic. No source-code changes to the user application; no rebuild of the MPI implementation. MANA has been tested against MPICH-3.x, MPICH-4.x, Open MPI-4.x, Open MPI-5.x, and ExaMPI.
Network-Agnostic. The MPI library handles the network; MANA does not reach into the network stack. The same MANA build that checkpoints a job on TCP/IP works on InfiniBand, Slingshot, etc.
Plugin on DMTCP. All process-level state (memory, threads, file descriptors, signals, pids) is checkpointed by DMTCP. MANA's job is to drain MPI-specific in-flight state at checkpoint time and reconstruct MPI-specific state at restart time.

License. MANA is licensed under the GNU Lesser General Public License v3 (LGPL v3), matching DMTCP.

Supported platforms. As of release 1.4.0:

Tested on Linux distributions: CentOS 7, Rocky 8 / 9, SUSE Enterprise Linux, Ubuntu.
Architectures: x86_64, ARM64 (aarch64), RISC-V.
MPI implementations: MPICH-3.x / 4.x, Open MPI-4.x / 5.x, ExaMPI.
Compiler: gcc supporting C++14 or later. The default gcc-4.8 shipped with CentOS 7 is too old; a newer toolchain (e.g. devtoolset-8) is required there.

Where to file bugs. https://github.com/mpickpt/mana/issues.

Source. https://github.com/mpickpt/mana.

Citations. When citing MANA in a publication, please cite the original MANA paper. See Further Reading for the bibliography.

GPU / CUDA support. DMTCP support CUDA checkpointing in 4.2.0, which is included in MANA 1.4.0. DMTCP's CUDA plugin is an optional plugin. Follow DMTCP's manual to compile and enable the CUDA plugin. Currently, it doesn't support NCCL.

Build System

Prerequisites

A working MPI installation (header, library, and mpicc/mpic++/mpifort wrappers), plus standard build tooling:

# Debian / Ubuntu
sudo apt install git gcc g++ make python3 autoconf automake -y

# CentOS 7 (default gcc-4.8 is too old; use devtoolset)
sudo yum install centos-release-scl
sudo yum install devtoolset-8
scl enable devtoolset-8 bash

Compilation requires C++14 or higher. Where Intel icc is available, it must be configured against a sufficiently new gcc (not CentOS 7's default gcc-4.8).

A stock dynamic MPICH or Open MPI install is sufficient; no special configure flags on the MPI side are required.

First-time setup

DMTCP is a git submodule under dmtcp/. After cloning MANA, initialize it:

git clone https://github.com/mpickpt/mana
cd mana
git submodule update --init

Configure and build

The default invocation builds for a development workstation:

./configure
make -j8

For a debug build (debug symbols, no optimization, verbose plugin logs):

./configure --enable-debug
make -j8

Performance caveat. --enable-debug compiles MANA with -g3 -O0 and lowers runtime performance substantially. Do not benchmark a debug build.

The top-level make runs DMTCP's build first, then cd mpi-proxy-split && make install && make -j tests. Final artifacts land in ./bin/ and ./lib/:

lib/dmtcp/libmana.so — the MANA DMTCP plugin (the upper half).
lib/dmtcp/libmpistub.so — stub MPI library, used when linking *.mana.exe test programs so they link without a real MPI library present.
bin/lower-half — the lower-half loader (also exposed as lh_proxy).
bin/mana_coordinator, bin/mana_launch, bin/mana_restart, bin/mana_status, bin/mana_start_coordinator — the user-facing Python launcher scripts.
bin/mpicc_mana — wrapper compiler that produces an executable directly runnable under MANA without --use-shadowlibs.
bin/mana_p2p_update_logs — replay-log post-processor for MANA_P2P_LOG/MANA_P2P_REPLAY.

Configuring for a specific MPI

Plain ./configure autodetects an MPI on $PATH. To target a specific installation, override the autoconf variables. The repository includes configure-mana as a working template; edit it for your MPI and run it instead of ./configure:

# Excerpt of configure-mana (defaults shown):
MPI_ETHERNET_INTERFACE=$(PATH=$PATH:/usr/sbin ip addr | grep -B1 link/ether | head -1 | sed -e 's%[^ ]*: \([^ ]*\): .*%\1%')

./configure --enable-debug \
            CFLAGS=-fno-stack-protector \
            CXXFLAGS=-fno-stack-protector \
            MPI_BIN=$MPI_INSTALL_DIR/bin \
            MPI_INCLUDE=$MPI_INSTALL_DIR/include \
            MPI_LIB=$MPI_INSTALL_DIR/lib \
            MPICC='${MPI_BIN}/mpicc' \
            MPICXX='${MPI_BIN}/mpic++' \
            MPIFORTRAN='${MPI_BIN}/mpifort' \
            MPIRUN='${MPI_BIN}/mpirun -iface '${MPI_ETHERNET_INTERFACE} \
            MPI_LD_FLAG=-lmpich \
            MPI_CFLAGS= \
            MPI_CXXFLAGS= \
            MPI_LDFLAGS=

The variables are:

Variable	Meaning
`MPI_BIN`	Directory containing `mpicc`, `mpic++`, etc.
`MPI_INCLUDE`	Directory containing `mpi.h`.
`MPI_LIB`	Directory containing the MPI shared libraries.
`MPICC`	Full path to the MPI C compiler wrapper.
`MPICXX`	Full path to the MPI C++ compiler wrapper.
`MPIFORTRAN`	Full path to the MPI Fortran compiler wrapper.
`MPIRUN`	Command (with any flags) used to launch test jobs. On SLURM sites, override to `srun`.
`MPI_LD_FLAG`	Linker flag to use the MPI runtime (`-lmpich`, `-lmpi`, ...).
`MPI_CFLAGS`, `MPI_CXXFLAGS`, `MPI_LDFLAGS`	Extra C/C++/linker flags.

The autoconf output lands in mpi-proxy-split/Makefile_config. Every sub-Makefile under mpi-proxy-split/ includes that file, so re-running ./configure is the right way to change MPI versions.

Why `-fno-stack-protector` is mandatory

MANA swaps the FS register on every MPI call (see Appendix B). glibc's stack protector reads its canary through the FS register, so an FS swap from inside a stack-protected function reads a canary from the wrong TLS and aborts as if the stack had been smashed. The repository disables stack protection globally via CFLAGS=-fno-stack-protector and CXXFLAGS=-fno-stack-protector. Do not remove these.

Incremental builds

The repository has several sub-Makefiles you can drive while iterating on one part of the plugin:

cd mpi-proxy-split && make            # rebuild libmana.so + libmpistub.so + tests
cd mpi-proxy-split/lower-half && make # rebuild bin/lower-half only
cd mpi-proxy-split/mpi-wrappers && make libmpiwrappers.a

When iterating on mpi-proxy-split/mpi-wrappers/, always rebuild libmpiwrappers.a before re-linking libmana.so — make in mpi-proxy-split/ enforces this order.

Pre-commit hook

The top-level make installs util/hooks/pre-commit into .git/hooks/ (via make add-git-hooks). The hook runs util/dmtcp-style.py on every staged C/C++/Python file — license header, line length, whitespace, etc. C/C++ formatting is configured in .clang-format (Google style with Linux brace style, BinPackParameters: false, AccessModifierOffset: 0, ContinuationIndentWidth: 2).

Cleaning checkpoint artifacts

make tidy in any subdirectory removes stale ckpt_*.dmtcp files and ckpt_rank_* directories. mana_launch refuses to start if it finds old checkpoints in --ckptdir, so a make tidy is often the right thing between test runs.

Usage Examples

Single host, no SLURM

This is the right workflow on a developer workstation, or on an HPC login node that allows mpirun (some sites do, most don't).

# In one terminal: start the coordinator.
bin/mana_coordinator

# In another terminal: launch the MPI job under MANA.
mpirun -n 4 bin/mana_launch ./my_mpi_app

# Create a checkpoint from any terminal.
bin/mana_status --checkpoint

# Kill the running job (Ctrl+C).

# Restart from the checkpoint.
mpirun -n 4 bin/mana_restart

Periodic checkpointing — checkpoint every 30 seconds — is the same idiom, just with -i:

bin/mana_coordinator -i 30
mpirun -n 4 bin/mana_launch ./my_mpi_app

SLURM cluster (`srun`)

Most HPC sites require SLURM. After salloc/sbatch puts you on a compute allocation:

bin/mana_coordinator
srun -n 4 bin/mana_launch ./my_mpi_app
# In a second terminal on the same node:
bin/mana_status --checkpoint
# Then to restart:
srun -n 4 bin/mana_restart

The coordinator's host and port are written by mana_coordinator to a ~/.mana.rc file (or ~/.mana-slurm-${SLURM_JOB_ID}.rc when SLURM is involved), and mana_launch, mana_restart, and mana_status read it to find the coordinator. See .mana.rc below.

A complete test cycle with a bundled test app

A common smoke test once MANA is built:

# Build all .mana.exe test programs.
cd mpi-proxy-split/test && make

# Set up coordinator and run a test under MANA with periodic checkpoints.
../../bin/mana_coordinator -i 5
mpirun -n 2 ../../bin/mana_launch ./ping_pong.mana.exe

# After a few checkpoints land in ./ckpt_rank_*, ^C the running job,
# then restart.
mpirun -n 2 ../../bin/mana_restart

Site-specific notes

The workflows above cover the common case. This subsection extends them with details for a few specific HPC sites. The upstream documentation at https://mana-doc.readthedocs.io includes the same material as separate pages for Explorer (Northeastern University; Rocky Linux 9) and Perlmutter (NERSC/LBNL; SUSE Enterprise).

Explorer (Rocky Linux 9.7)

Compile on a compute node, not on a login node. Login-node policy on Explorer forbids long-running commands.
Module choices vary by site update; run module avail and pick a recent gcc, python, and MPI (MPICH or Open MPI). Rocky 9 ships gcc 11 by default, so an explicit module load gcc/... is often unnecessary:
```
module avail gcc python mpich openmpi
module load mpich    # or openmpi, depending on your application
```
Interactive allocation (partition names and constraints may change; check Explorer's site documentation for current values):
```
srun --partition=short --nodes=1 --ntasks=8 --cpus-per-task=1 \
     --time=08:00:00 --mem=8GB --constraint=ib --pty /bin/bash
```
--constraint=ib requests an InfiniBand-equipped node; without it you may land on a TCP/IP-only node.
mpirun -n <N> is the standard launcher on Explorer.

Perlmutter (SUSE Enterprise Linux at NERSC)

Compilation can be done on a login node (this is the recommended practice on Perlmutter).

Use salloc for interactive jobs, with --constraint cpu for CPU-only nodes:

salloc --qos interactive --nodes 1 --time 04:00:00 \
       --constraint cpu --account YOUR_ACCOUNT

Use srun -n <N> (or mpirun -n <N> when using Open MPI). The default Cray MPICH stack works with srun.
Coordinator status file lives at ~/.mana-slurm-${SLURM_JOB_ID}.rc on the shared NERSC filesystem; all nodes in the allocation see it.

A site not listed

If your site is materially different from Explorer or Perlmutter, the upstream MANA maintainers are interested in documenting it. Open an issue at https://github.com/mpickpt/mana-doc.

Software Components

MANA ships seven user-facing executables in bin/ plus the plugin shared libraries in lib/dmtcp/. The launchers are a mix of shell scripts (mana_coordinator, mana_status, mpicc_mana) and Python scripts (mana_launch, mana_restart). All of them parse a small set of MANA-specific flags, look up the coordinator via .mana.rc, and exec into the corresponding DMTCP binary with the MANA plugin pre-loaded.

`mana_coordinator`

Starts a DMTCP coordinator configured for MANA: detached as a daemon, quiet by default, with a status file at ~/.mana.rc (or ~/.mana-slurm-${SLURM_JOB_ID}.rc if $SLURM_JOB_ID is set). The coordinator's host and port are written to that status file so the other launchers can find it.

USAGE: mana_coordinator [--verbose] [--help] [-q|--quiet]
                        [-i SECONDS | --interval SECONDS]
                        [DMTCP_OPTIONS ...]

Flag	Meaning
`--verbose`	Print the underlying `dmtcp_coordinator` command before exec'ing.
`--help`	Show DMTCP coordinator options.
`-q`, `--quiet`	Suppress coordinator output.
`-i SEC`, `--interval SEC`	Auto-checkpoint every `SEC` seconds (0 = disabled).

mana_coordinator ultimately runs:

dmtcp_coordinator --exit-on-last -q --daemon --status-file "$MANA_RC" \
                  [your options]

--exit-on-last means the coordinator goes away when the last worker disconnects; together with the daemon flag, this matches the usual HPC-batch lifecycle (the coordinator should not outlive the job).

On NERSC systems (where $NERSC_HOST is cori or gerty) the script requires $SLURM_JOB_ID to be set so the status file goes into the SLURM-specific location.

mana_launch and mana_restart each delete any ~/.mana*.rc file older than 7 days on startup (via find $HOME/.mana*.rc -mtime +7 -delete). Stale .mana.rc files from previous, crashed jobs are otherwise harmless but accumulate.

`mana_start_coordinator`

A symlink to mana_coordinator. Some sites prefer the longer name in batch scripts for clarity; the behavior is identical.

`mana_launch`

Launches an MPI executable under MANA. Equivalent to dmtcp_launch but prepares the kernel-loader / lower-half / libmana.so chain automatically.

USAGE: [srun|mpirun] mana_launch [--verbose] [--timing]
                                 [--ckptdir DIR]
                                 [--use-shadowlibs] [--gdb] [-q|--quiet]
                                 [DMTCP_OPTIONS ...]
                                 MPI_EXECUTABLE [ARGS ...]

Flag	Meaning
`--verbose`	Print the underlying `dmtcp_launch` command before exec'ing.
`--with-plugin`	A colon-separated list of DMTCP plugins. MANA will be loaded first
`--timing`	Set `MANA_TIMING=1`; print INIT/EXIT and checkpoint-event timings to stderr.
`--ckptdir DIR`	Write checkpoint images to `DIR` instead of the current directory.
`-h HOST`, `--coord-host HOST`	Coordinator hostname. Default: read from `.mana.rc`, fallback to localhost.
`-p PORT`, `--coord-port PORT`	Coordinator port. Default: read from `.mana.rc`, fallback to 7779.
`-i SEC`, `--interval SEC`	Auto-checkpoint every `SEC` seconds.
`--ckpt-signal SIG`	Internal checkpoint signal (default: SIGUSR2).
`--with-plugin PATH`	Load an additional DMTCP plugin alongside `libmana.so`.
`--tmpdir PATH`	Temporary directory (default `./tmp`).
`--coord-logfile PATH`	Coordinator log file path.
`--use-shadowlibs`	Use the MANA shadow-library directory to avoid pre-MANA library constructors. See Compiling MPI Applications for MANA.
`--gdb`	EXPERTS ONLY. Launch under gdb.
`-q`, `--quiet`	Set `MANA_QUIET=1`; suppress most DMTCP output.
`--help`	Show DMTCP launch options.

Internally, mana_launch chains into:

dmtcp_launch --kernel-loader \
             -h <host> -p <port> \
             --no-gzip --join-coordinator --disable-dl-plugin \
             --with-plugin <mana>/lib/dmtcp/libmana.so \
             [DMTCP_OPTIONS] \
             <mana>/bin/lower-half MPI_EXECUTABLE [ARGS ...]

The --no-gzip flag is deliberate: gzip compression adds seconds to each checkpoint and is rarely worth it for MPI workloads where the upper-half image is dominated by application data.

The launcher refuses to start if it finds pre-existing ckpt_rank_* directories in --ckptdir. Run make tidy (or rm -rf ckpt_rank_*) to clean up between runs.

`mana_restart`

Restarts a previously checkpointed job from disk. The restart command takes no application binary — the executable name is recorded in the checkpoint image.

USAGE: [srun|mpirun] mana_restart [--verbose] [--timing]
                                  [--restartdir DIR] [--ckptdir DIR]
                                  [--gdb] [-q|--quiet]
                                  [DMTCP_OPTIONS ...]

Flag	Meaning
`--verbose`	Print the underlying `dmtcp_restart` command.
`--timing`	Set `MANA_TIMING=1`.
`--restartdir DIR`	Directory containing the checkpoint images. Default: cwd.
`--ckptdir DIR`	Where to put new checkpoints taken after restart. Default: same as `--restartdir`.
`-h HOST`, `--coord-host HOST`	Coordinator hostname.
`-p PORT`, `--coord-port PORT`	Coordinator port.
`-i SEC`, `--interval SEC`	Auto-checkpoint every `SEC` seconds after restart.
`--ckpt-signal SIG`	Internal checkpoint signal.
`--with-plugin PATH`	Extra DMTCP plugin.
`--tmpdir PATH`	Temporary directory.
`--coord-logfile PATH`	Coordinator log file path.
`--gdb`	EXPERTS ONLY. Launch under gdb.
`-q`, `--quiet`	Set `MANA_QUIET=1`.
`--help`	Show DMTCP restart options.

mana_restart validates the restart directory: it must exist and must not contain incomplete .tmp files (a partial checkpoint that was interrupted mid-write).

The number of MPI processes at restart must match the number at checkpoint. Mixing rank counts across launch/restart is not supported.

`mana_status`

Queries the running coordinator. Wrapper around dmtcp_command with the coordinator host/port pre-filled from .mana.rc.

USAGE: mana_status [--verbose] [DMTCP_COMMAND_OPTIONS ...]

Common idioms:

bin/mana_status --list         # show all ranks and their states
bin/mana_status --checkpoint   # trigger a checkpoint
bin/mana_status -c             # short form of --checkpoint
bin/mana_status --status       # show coordinator summary

mana_status --list is especially useful during checkpoint debugging: it shows which rank is in which state (e.g. SUSPENDED vs RUNNING) so you can tell which rank is hanging.

`mana_p2p_update_logs`

Post-processes the MANA_P2P_LOG files produced by a logging run to trim records past the checkpoint point. Run between launch (with MANA_P2P_LOG=1) and restart (with MANA_P2P_REPLAY=1):

MANA_P2P_LOG=1 mana_launch ./app
# ... checkpoint happens ...
# Continue running for a few minutes so log buffers flush.
bin/mana_p2p_update_logs
MANA_P2P_REPLAY=1 mana_restart

See Deterministic replay.

`mpicc_mana`

Wrapper compiler that builds MPI applications such that the produced binary is directly runnable under MANA (no --use-shadowlibs needed). Compared to a plain mpicc, it adds -L<mana>/lib/dmtcp -lmpistub, so the binary links against MANA's MPI stub library (which provides symbol exports) rather than the real libmpi.so; the real symbols come from libmana.so at runtime.

USAGE: mpicc_mana [-cc=COMPILER] [-show] [-v] [mpicc args ...] FILE.c

Flag	Meaning
`-cc=GCC`	Use a specific C compiler instead of `gcc -std=gnu99`.
`-show`	Print the underlying compile/link command without executing.
`-show-link-info`, `-show-compile-info`	Show just the link or compile half of the command.
`-c`, `-S`, `-E`, `-M`, `-MM`	Compilation-only modes; no linking.
`-v`	Print version.
`-help`, `--help`	Help.

Note: do not combine mpicc_mana-built binaries with mana_launch --use-shadowlibs. Pick one approach. See Compiling MPI Applications for MANA.

`lower-half` / `lh_proxy`

The lower-half executable, normally invoked as a child of dmtcp_launch --kernel-loader. Users do not run it directly, but it is useful in two debugging contexts:

Under gdb, switch the symbol file with (gdb) file bin/lower-half (or bin/lh_proxy) when debugging lower-half code.
The --gdb flag on mana_launch/mana_restart arranges to run the computation under gdb so that breakpoints in the lower half resolve.

`libmana.so` and `libmpistub.so`

lib/dmtcp/libmana.so is the DMTCP plugin loaded into the upper half. Internally it wraps the static library libmpiwrappers.a (built in mpi-proxy-split/mpi-wrappers/) whole-archive. Users do not link against this; mana_launch --with-plugin handles it.

lib/dmtcp/libmpistub.so exists only so that *.mana.exe test binaries (and any binary built with mpicc_mana) can link without a real MPI library available at link time. At runtime, MPI symbols resolve into libmana.so. Users see libmpistub.so only through mpicc_mana.

The `.mana.rc` file {#the-manarc-file}

Created by mana_coordinator and read by mana_launch, mana_restart, and mana_status. The location is:

Environment	Path
No SLURM	`~/.mana.rc`
`$SLURM_JOB_ID` set	`~/.mana-slurm-${SLURM_JOB_ID}.rc`

Contents (plain text, key-value):

Host: cn123.example.com
Port: 7779
SLURM_JOB_ID: 4567890       # only in the SLURM variant
# This is a temporary file for communication between
#   mana_coordinator and mana_launch/mana_restart.

The launchers parse the Host: and Port: lines and use them as the default --coord-host and --coord-port. You can always override with the explicit flags.

mana_launch and mana_restart automatically delete any ~/.mana*.rc file older than 7 days on startup. If a job crashes, its .mana-slurm-*.rc may still be left behind; this is harmless but you can remove it manually.

Compiling MPI Applications for MANA

Default path: native `mpicc`

In MANA 1.2.0 and later, mana_launch can directly execute a binary compiled with the site's normal mpicc. This is what most users should do. Build the application the way you normally would, then launch it under mana_launch with no extra flags:

mpicc -o my_app my_app.c
mana_coordinator
mpirun -n 4 mana_launch ./my_app

This works out of the box for most MPI implementations and applications. If it works for you, stop here — you do not need shadow libraries, you do not need mpicc_mana, and the rest of this chapter is not relevant to your workflow.

When the default path doesn't work

A few MPI installations — notably some MPICH-4.x builds and any MPI stack that pulls in UCX or Intel runtime libraries — link in helper libraries whose global constructors run before main. Those constructors execute before MANA's plugin can take control, and they touch state (network endpoints, thread pools, signal handlers) that MANA later has no way to checkpoint cleanly. The symptom is usually a segfault or hang very early in execution, sometimes before your main is even reached.

When this happens, MANA offers two solutions, both of which keep the real MPI / UCX / Intel libraries out of the upper half so their constructors never run there. Pick whichever option fits your situation:

Stub library (compile time). Recompile the application with mpicc_mana. The resulting binary links against MANA's libmpistub.so at compile time instead of the real MPI library. Best when you have the source.
Shadow library (runtime). Keep the existing binary and launch it with mana_launch --use-shadowlibs. MANA generates a directory of shadow libraries on the fly and interposes them on the upper half's library search path. Best when you can't recompile.

The two mechanisms are different — one is a link-time substitution, the other is a runtime interposition — but they solve the same problem and you only need one of them per binary.

Option 1 — stub library (`mpicc_mana`)

Rebuild the application with the provided mpicc_mana wrapper. The binary then runs under mana_launch with no extra flag:

mpicc_mana -o my_app.mana.exe my_app.c
mana_coordinator
mpirun -n 4 mana_launch ./my_app.mana.exe

mpicc_mana is a thin wrapper around the site's mpicc that adds -L<mana>/lib/dmtcp -lmpistub, so the binary links against MANA's MPI stub library at compile time. The real MPI symbols are resolved into libmana.so at runtime.

The binary produced by mpicc_mana runs only under MANA — it cannot be executed by the bare MPI implementation, because its MPI symbols come from the stub library. By convention MANA's test suite uses the .mana.exe suffix for such binaries.

Use this option when:

you have the application source and can recompile,
you want a small, repeatable launch command (no --use-shadowlibs flag to remember), and
you don't need the binary to also run without MANA.

Option 2 — shadow library (`mana_launch --use-shadowlibs`)

Keep the binary you already built with the site's mpicc, and ask the launcher to interpose shadow libraries at runtime:

mpicc -o my_app my_app.c
mana_coordinator
mpirun -n 4 mana_launch --use-shadowlibs ./my_app

mana_launch creates the shadow-library directory at <mana>/lib/tmp on demand and prepends it to the upper half's library search path. The lower half continues to use the real MPI libraries.

Use this option when:

you have a closed-source binary you cannot recompile, or
you want a single binary that can also run under bare mpirun (without MANA), and only want the interposition when launching under MANA.

Don't combine Options 1 and 2

mpicc_mana-built binaries already include the stub library; passing them through mana_launch --use-shadowlibs adds a second layer of interposition and produces undefined behavior. Pick one approach per binary.

Summary

Situation	What to do
Standard MPI; no early-startup crash	Native `mpicc`, neither stub nor shadow library
Early crash; you can recompile from source	Stub library (Option 1): rebuild with `mpicc_mana`
Early crash; closed-source binary	Shadow library (Option 2): native binary + `mana_launch --use-shadowlibs`
Early crash; want one binary that also runs without MANA	Shadow library (Option 2): native binary + `mana_launch --use-shadowlibs`

Running MANA with SLURM

SLURM is the most common job scheduler on the HPC sites where MANA is deployed. The mechanics differ slightly from cluster to cluster but the structure is always:

Allocate one or more compute nodes (interactively with salloc or in batch with sbatch).
On a compute node, start the coordinator with mana_coordinator.
Launch the MPI job with srun -n <N> mana_launch ... (some sites prefer mpirun -n <N>; see below).
Restart in the same way with mana_restart.

The coordinator's host and port are written to ~/.mana-slurm-${SLURM_JOB_ID}.rc, which mana_launch, mana_restart, and mana_status discover automatically. You can override with -h and -p if needed.

`srun` vs `mpirun`

On most modern SLURM sites, srun replaces mpirun for launching MPI applications. On some sites — notably CentOS 7 and Rocky 8 — the recommended pattern is still mpirun -n <num> ... even inside a SLURM allocation. Check your site's documentation. On Open MPI sites, mpirun is generally preferred over srun.

Process placement and the checkpoint thread

MANA adds a checkpoint thread to every MPI process. When sizing your SLURM allocation (--ntasks-per-node, --cpus-per-task), keep in mind that each MPI process under MANA has one more thread than the same process without MANA. This is usually negligible but matters at extreme oversubscription.

Multi-node coordinator discovery

When the MPI job spans multiple nodes, every rank reads the same ~/.mana-slurm-${SLURM_JOB_ID}.rc from its home directory (which is assumed to be on a shared filesystem) to find the coordinator. The coordinator itself is hosted on whichever node ran mana_coordinator — usually the head/login node of the allocation.

If your home directory is not on a shared filesystem, you must set DMTCP_COORD_HOST and DMTCP_COORD_PORT (or pass -h/-p) explicitly to every mana_launch / mana_restart.

Stale `.mana*.rc` files

mana_launch and mana_restart automatically delete ~/.mana*.rc files older than 7 days on startup. If a job crashes such that its status file is never cleaned up, the next mana_coordinator will overwrite it on the next run. You can also delete stale files manually after a job completes.

Controlling MANA with Environment Variables

MANA reads its own MANA_* variables in addition to the standard DMTCP_* set (see DMTCP_USER_MANUAL.md, Controlling DMTCP with Environment Variables). The MANA-specific variables are listed below.

User-facing

MANA_DEBUG : When set to any non-empty value, MANA prints additional diagnostic messages to stderr. Useful for filing bug reports.

MANA_TIMING : When set to any non-empty value (or by passing --timing), MANA prints elapsed-time measurements for the DMTCP_EVENT_INIT, DMTCP_EVENT_EXIT, and checkpoint/restart events to stderr. Useful for profiling checkpoint overhead.

MANA_QUIET : Set by --quiet/-q. Suppresses most DMTCP output.

Pause-for-gdb

These mirror DMTCP's launch and restart pause variables; see DMTCP_USER_MANUAL.md, Advanced Debugging with GDB attach.

DMTCP_LAUNCH_PAUSE, DMTCP_MANA_PAUSE : Pause during mana_launch to allow gdb attach (gdb must be on the same node).

DMTCP_RESTART_PAUSE : Pause during mana_restart, before resuming, to allow gdb attach. Accepts a level 1–7; see the DMTCP debugging chapter for the per-level pause sites.

MANA_SEGV_DEBUG_LOOP : When set, a SIGSEGV in the plugin enters an infinite loop instead of dumping core, so you can gdb attach to the live, crashed process. Intended for diagnosing FS-register misalignment and similar low-level faults.

Experts only

MPI_COLLECTIVE_P2P (debugging only) : When set, MPI collective calls are translated at runtime into MPI_Send/MPI_Recv sequences. This significantly slows the application and is intended only as a fallback for debugging collectives whose own implementation interferes with MANA's collective drain. Requires rebuilding the relevant files in mpi-proxy-split/mpi-wrappers/ — touch mpi_collective_p2p.c && make after setting the variable.

You can choose which collectives are affected by adjusting the #ifdef MPI_COLLECTIVE_P2P / #ifndef MPI_COLLECTIVE_P2P blocks in mpi_collective_p2p.c and mpi_collective_wrappers.cpp.

MANA_P2P_LOG : Enables deterministic-replay logging for point-to-point calls. Set before mana_launch. See Deterministic P2P and Allreduce replay.

MANA_P2P_REPLAY : Enables replay of a previously-logged P2P trace. Currently must be set before mana_launch as well as mana_restart; this is expected to be relaxed in a future release.

MANA_USE_ALLREDUCE_REPRODUCIBLE : When set, MPI_Allreduce reductions over associative/commutative operations are performed in a deterministic order so that results are bit-identical across launch and restart. Without this, an MPI library is free to use any reduction order it likes, which can produce slightly different floating-point results between the original launch and the replay after restart.

MANA_FILE_REGEX (TO BE REMOVED) : Regex selecting which application file paths to include in the checkpoint. Default: .* (all files opened by the application). Note that this controls only MANA's per-file include filter; DMTCP's own file-checkpoint behavior is governed by --ckpt-open-files.

Internal / developer

These are used by MANA internals; users normally do not set them directly:

MANA_LH_INFO_ADDR — address of the lower-half info struct (lh_info) passed from lower half to upper half at launch. On restart the /tmp/mana_tmp_lh_info_* file is used instead.
MANA_MPICC_RECURSION_CHECK — guard against mpicc_mana invoking itself recursively.
DMTCP_FSGSBASE_ENABLED — internal flag; set automatically based on kernel capability detection.

Inherited DMTCP variables

All standard DMTCP variables apply under MANA. See DMTCP_USER_MANUAL.md, Controlling DMTCP with Environment Variables. The ones most commonly relevant under MANA are:

DMTCP_COORD_HOST, DMTCP_COORD_PORT — coordinator location. mana_coordinator writes these into .mana.rc and the other launchers pick them up automatically; set the env vars only if you need to override.
DMTCP_CHECKPOINT_DIR — checkpoint output directory; equivalent to --ckptdir.
DMTCP_TMPDIR — temporary file directory.
DMTCP_SLEEP_ON_FAILURE, DMTCP_ABORT_ON_FAILURE — what to do on a fatal JASSERT.

Checkpoint and Restart Flow

A user-visible description of what MANA does between "checkpoint requested" and "checkpoint written". For the design rationale, see Appendix B. For the implementation, see Appendix C.

Plugin states

The MANA plugin tracks the rank's progress through a checkpoint with the mana_state_t state machine (defined in mpi-proxy-split/mpi_plugin.h):

State	Meaning
`UNKNOWN_STATE`	Before plugin initialization.
`RUNNING`	Normal execution; MPI calls proceed directly to the lower half.
`CKPT_COLLECTIVE`	Pre-checkpoint phase 1: draining outstanding collectives.
`CKPT_P2P`	Pre-checkpoint phase 2: draining outstanding point-to-point messages.
`RESTART_RESTORE`	Reserved; not actively used in the current code.
`RESTART_REPLAY`	Restart phase: replaying logged MPI calls to reconstruct lower-half state.

mana_status --list shows each rank's DMTCP-level state at checkpoint time; for the per-rank plugin state, attach gdb and inspect current_state.

Checkpoint sequence

When the coordinator triggers a checkpoint (typically because mana_status --checkpoint was called or the periodic interval elapsed), each rank:

DMTCP_EVENT_PRESUSPEND. State moves to CKPT_COLLECTIVE. MANA drains all collective MPI calls by polling the sequence-number algorithm until every rank has caught up to a global, agreed-upon "checkpoint barrier" in the collective stream.
Global barrier. All ranks signal CKPT_COLLECTIVE complete via a DMTCP global barrier.
State moves to CKPT_P2P. MANA drains all outstanding non-blocking sends and receives, probes for unexpected messages, and confirms global_sent == global_recv. Any rank blocked inside MPI_Recv is unblocked by a dummy MPI_Send injected from another rank.
DMTCP_EVENT_PRECHECKPOINT. MANA records the MPI init-time memory map, open file descriptors, the local rank, the MANA header metadata file, and MPI file-handle metadata. Final global barrier.
DMTCP writes the image. Memory mappings, threads, fds, and signal state are serialized to ckpt_rank_*/ckpt_*.dmtcp.
DMTCP_EVENT_RESUME. State returns to RUNNING. The drain counters and sequence-number flags are reset. The dummy P2P messages injected in step 3 are discarded.

If your application observes a long pause at checkpoint, it is almost always step 1 or step 3. The application has a long-running collective with uneven entry/exit times across ranks, or a large backlog of un-received non-blocking sends. Logging the per-event times with MANA_TIMING=1 will pinpoint which phase is slow. For internals, see Appendix C.

Restart sequence

When mana_restart runs:

The lower half (re-)initializes from scratch — fresh libc, fresh libmpi.
The lower half calls MPI_Init(NULL, NULL) and acquires an MPI rank.
The upper-half memory image is mapped from ckpt_rank_<rank>/ckpt_*.dmtcp over the running process.
DMTCP_EVENT_RESTART. MANA reconnects the upper half's MPI wrappers to the new lower-half symbol addresses (the lower-half addresses change at every restart).
State moves to RESTART_REPLAY. MANA replays the per-category record-replay logs to rebuild the virtual-ID → real-handle mapping in the lower half:
- communicators (restoreComms),
- datatypes (restoreTypes),
- groups (restoreGroups),
- operations (restoreOps),
- Cartesian / topology constructors (restoreCarts).
P2P messages that were in flight at checkpoint time are re-injected from the P2P log (if MANA_P2P_REPLAY is set; without it, the application receives whatever the live MPI traffic produces).
MPI file handles are reopened and seek positions restored.
State returns to RUNNING and user code resumes from where the checkpoint was taken.

Deterministic P2P and Allreduce replay

By default, MANA's checkpoint-restart is causally correct — the application observes a valid execution — but is not bit-for-bit reproducible against a launch with no checkpoint. Two optional mechanisms tighten this:

MANA_P2P_LOG=1 at launch and MANA_P2P_REPLAY=1 at restart log and replay the order of every MPI_Send/MPI_Recv family call. Between launch and restart, run mana_p2p_update_logs to trim the log to the checkpoint point. Currently the variable must also be set at launch.
MANA_USE_ALLREDUCE_REPRODUCIBLE=1 makes MPI_Allreduce over associative/commutative reduction operators (where the MPI standard allows the library to reorder) commit to a deterministic order, so floating-point results are bit-identical across launch and replay.

Use these for debugging non-deterministic behavior or for publication-quality reproducibility benchmarks. In normal use they are unnecessary and add overhead.

Debugging

The DMTCP debugging chapter (DMTCP_USER_MANUAL.md, Debugging) covers the topics shared with MANA: configuring a debug build, the gdb-dmtcp-utils.py script, the "while dummy" spin-loop trick, the --debug-restart-pause N mechanism, DMTCP_SLEEP_ON_FAILURE, and the WSL2 gotcha. Read that first; everything in it applies under MANA.

This section covers what is specific to MANA.

Hierarchy of validation

Before reaching for gdb, work through this hierarchy. Bugs are most often introduced at the lowest level on which something changed:

Does the application work under plain mpirun / srun with no MANA?
Does it work with mana_launch on a single MPI process?
Does it work with mana_launch on multiple processes on a single node?
Does it work with mana_launch on two nodes, two processes?
Does it work with mana_launch at full scale?

Find the lowest level where the bug appears, then debug from there.

`mana_status --list`

The first command to run when an MPI job appears stuck during checkpoint. It prints each rank's DMTCP state. A common pattern is "15 ranks SUSPENDED, 1 rank RUNNING" — the running rank is stuck inside an MPI call that has not yet returned, and is therefore not yet ready for checkpoint. Attach gdb to that rank and look at the stack.

Inspecting a checkpoint image

util/readdmtcp.sh ckpt_rank_0/ckpt_*.dmtcp

dumps a human-readable header — memory map, fds, threads — of a checkpoint image. Useful for confirming that the image was written completely.

Attaching gdb at restart

The recommended way to break very early in restart is to set DMTCP_RESTART_PAUSE=3 (a single-threaded restart phase; see DMTCP_USER_MANUAL.md, Advanced Debugging with GDB attach for level selection):

srun ... env DMTCP_RESTART_PAUSE=3 mana_restart ... &
# After ssh'ing into the compute node where the rank is running
# (see "Reaching the compute node" below):
gdb -p <PID-printed-by-mtcp_restart>
(gdb) source util/gdb-dmtcp-utils
(gdb) load-symbols-library
(gdb) set var dmtcp::restartPauseLevel = 0
(gdb) continue

MANA_DEBUG=1 prints the gdb-attach instructions for each rank to stderr.

Reaching the compute node

On a SLURM cluster, your srun shell sits on the login node, but mtcp_restart and the application run on one or more compute nodes in your allocation. gdb -p <PID> only works if it runs on the same node as the target process, so you must first ssh from a second login-node terminal into the compute node that holds the rank you want to debug.

The general pattern is:

# 1. List the compute nodes allocated to your job.
squeue --me

# 2. SSH to one of the listed hostnames.
ssh <compute-node-hostname>

# 3. Run gdb there.
gdb -p <PID-from-mtcp_restart>

The site-specific details differ:

Explorer (Northeastern): compute-node hostnames typically look like c<NNNN> or d<NNNN>. Explorer requires passwordless SSH between nodes; if ssh c1234 prompts for a password, generate an SSH key (ssh-keygen) and append the public key to ~/.ssh/authorized_keys so node-to-node SSH works without a prompt.

Perlmutter (NERSC): compute-node hostnames look like nid<NNNNNN>. Direct SSH from a login node to any compute node in one of your active jobs is allowed without extra setup. If your local configuration requires an intermediate data/gateway node, ask NERSC support for the current recommended path.

Other sites: check the site's documentation. Some sites require ssh through an intermediate "data node" before reaching a compute node, and some restrict direct compute-node SSH entirely.

Debugging the lower half

The lower half binary is bin/lower-half (or, on older builds, bin/lh_proxy). When you attach gdb to a MANA process and the backtrace lands inside the lower half, gdb may show only addresses — the lower half's symbols are not loaded by default. Bring them in with:

(gdb) file bin/lower-half

If the lower half is dynamically linked (default in MANA 1.0+), you may instead need:

(gdb) source util/gdb-dmtcp-utils
(gdb) load-symbols-library lower-half

For low-level restart issues (faults before the upper half is mapped), read restart/plugin/README in the source tree.

SEGV in an MPI wrapper

A crash with MANA_SEGV_DEBUG_LOOP set spins forever instead of dying, so you can attach. This is the right knob when the crash is in a wrapper or a switch-context call and you cannot reproduce it under gdb directly.

Debugging your own plugin alongside MANA

If you have written a DMTCP plugin that you want to load alongside libmana.so, pass it via mana_launch --with-plugin /path/to/your.so. The DMTCP manual's Debugging Your Own Plugin (Appendix C of DMTCP_USER_MANUAL.md) describes how to set breakpoints in your hook function so they survive the launch-vs-restart code path differences.

Known Limitations

These are the limitations users hit most often. The MANA paper and the source tree's mpi-proxy-split/TODO.md track them in more detail.

Checkpoint after MPI_Finalize is forbidden. Once MPI_Finalize returns, the lower half has no MPI rank and there is no consistent point at which a checkpoint can be taken. MANA detects this and skips the checkpoint, rather than producing a corrupt image. This is a deliberate guard, added in 1.3.0.
Rank count must match between launch and restart. Restarting with a different -n is unsupported.
MPI implementation and network must match between launch and restart. While MANA's mechanism is MPI- and network-agnostic (the same MANA build works on any supported combination), an individual checkpoint must be restarted on the same MPI implementation, MPI version, and network fabric that was used at launch. Cross-implementation or cross-network restart (e.g. checkpoint on MPICH, restart on Open MPI; or checkpoint over TCP, restart over InfiniBand) is not supported.
MPI-2-only functions are unsupported. MPI functions that were in MPI-2 but removed from MPI-3 are not wrapped (some MPI implementations have already removed their signatures from mpi.h).
One-sided communication (MPI_Win_*) coverage is limited. Wrappers exist in mpi_win_wrappers.cpp but are less extensively tested than collectives and point-to-point.
MPI_THREAD_MULTIPLE is not supported. MANA's collective and point-to-point drain protocols assume that MPI calls from a given rank are serialized — at most one thread inside MPI at a time. Applications must call MPI_Init_thread with MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, or MPI_THREAD_SERIALIZED. Passing MPI_THREAD_MULTIPLE may appear to work but can deadlock or produce inconsistent state at checkpoint time.
MPI_Spawn and dynamic process creation are not supported.
GPU memory checkpointing. MANA does not snapshot device memory. Applications using CUDA / HIP / SYCL today need an additional plugin (not shipped with MANA) to handle device state. CUDA support is added to the underlying DMTCP in MANA 1.4.0. NCCL support is working in progress.
FSGSBASE on old kernels. On Linux < 5.9, every MPI call pays the cost of two arch_prctl syscalls. Up to ~5% additional runtime overhead has been observed.

See mpi-proxy-split/TODO.md for the maintainers' running list.

Appendix A: Testing

Test trees

MANA has two distinct test trees, both under mpi-proxy-split/:

test/ — end-to-end MPI tests. Each .c file builds two binaries: <name>.exe (linked against the real MPI library) and <name>.mana.exe (linked against libmpistub.so so the binary runs only under MANA).
unit-test/ — C++ unit tests for internal modules; uses gtest and does not need a running MPI.

End-to-end tests

# Build every test app.
cd mpi-proxy-split/test && make

# Run a single test through launch + checkpoint + restart.
make check Bcast_test       # or check-Bcast_test on some Makefiles

# Run the full suite via autotest.py.
cd mpi-proxy-split && python autotest.py

mpi-proxy-split/autotest.py is the Python test harness. It assumes srun -n 4 by default; on a workstation without Slurm, edit the MPIRUN and MPIRUN_FLAGS variables near the top, then re-run.

A representative subset of the ~60 test apps in test/:

Collectives: Allgather_test, Allreduce_test, Alltoall_test, Alltoallv_test, Barrier_test, Bcast_test, Gather_test, Gatherv_test, Reduce_test, Scan_test, Scatter_test, Scatterv_test, Ibarrier_test, Ibcast_test.
Point-to-point: send_recv, Sendrecv_test, sendrecv_replace_test, ping_pong, Isend_test, Irecv_test, Waitall_test, Waitany_test, Testany_test, large_async_p2p.
Datatypes: Type_commit_contiguous, Type_vector_test, Type_hvector_test, Type_create_struct_test, Type_create_resized_test, Type_dup_test, MPI_Double_Int_test.
Communicators / groups / topologies: Comm_dup_test, Comm_split_test, Comm_free_test, Comm_get_attr_test, Comm_compare_test, Group_size_rank, keyval_test, Cart_map_test, Cart_sub_test, Cartdim_get_test.
MPI-IO: File_read_write_test, File_read_write_all_test, File_size_test, File_characteristics_test, File_mixed_fds_test, file_test.
Init / Abort / Finalize: hello_mpi_init_thread, Initialized_test, MANA_MPI_Initialized_test, Abort_test, unsync_Finalize, integrated_dmtcp_test, two-phase-commit-{1,2,3}.

The full list lives under mpi-proxy-split/test/.

Unit tests

cd mpi-proxy-split/unit-test
make
make check

The unit tests exercise the most fragile internal modules:

drain-send-recv-test.cpp — P2P drain and dummy-injection protocol.
record-replay-comm-test.cpp — communicator replay.
record-replay-group-test.cpp — group replay.
record-replay-cart-test.cpp — Cartesian topology replay.
record-replay-types-test.cpp — datatype replay.

Each unit test builds an .exe and runs under mpirun -n 1. No coordinator is required.

Continuous integration

ci/unit-test.sh is the closest thing to a "run everything" command. It runs end-to-end:

# from the repo root
source /opt/rh/devtoolset-8/enable        # only on CentOS 7
git submodule update --init
./configure
make -j8 mana                              # builds libmana.so, lower-half, etc.
cd mpi-proxy-split/unit-test
make
make clean && make check                   # runs every .exe with mpirun -n 1

On a fresh checkout this is the recommended bring-up check.

Appendix B: Architecture (Split-Process Execution)

MANA's defining design is split-process execution. A single MPI rank runs as two cooperating halves inside one address space:

The upper half holds the user program, the DMTCP libraries, MANA's library (libmana.so), and the MANA MPI wrappers. The upper half is the part that gets checkpointed.
The lower half holds the real MPI library (libmpich.so / libmpi.so / ...), the real network library (UCX, libfabric, ...), and the C library these depend on. The lower half is never written to the checkpoint image; at restart time it is discarded, and a fresh lower half is created.

The purpose of this split is to keep the checkpoint mechanism itself independent of any specific MPI implementation or network library. The MANA code that runs in the upper half — the wrappers, the drain protocols, the record-replay logs, the virtual-handle tables — talks only to the lower half through a stable function-pointer interface. It contains no MPICH-specific or Open-MPI-specific assumptions, and no TCP- or InfiniBand-specific code. The same MANA build works under any supported MPI implementation and any supported network fabric. This is what "MPI-Agnostic, Network-Agnostic" actually means.

Note. A given checkpoint must still be restarted on the same MPI implementation and the same network fabric that was used at launch. The portability is of the checkpoint-restart mechanism, not of an individual checkpoint image. Cross-MPI or cross-network restart is not supported.

At restart time, a fresh lower half is started, it calls MPI_Init and acquires a (possibly different) MPI rank, and then the saved upper-half image is mapped over the new process and execution resumes where the checkpoint was taken. The MPI handles the application holds (MPI_Comm, MPI_Datatype, MPI_Request, ...) are virtualized so that they remain stable across the swap, even though the underlying real handles in the new lower half are entirely new.

The rest of this appendix explains how this works. It is foundational reading for the wrapper-level material in Appendix C.

Why split-process

A transparent MPI checkpoint-restart tool faces two problems that non-MPI checkpointing does not:

MPI implementations vary widely, and so do network stacks. A checkpoint-restart tool that links directly against one MPI library inherits dependencies on that library's internal handle layout, its memory map, and the network library it uses. Such a tool would need to be reimplemented (or at least retested) for every MPI / network combination. This is what previous MPI-checkpointing tools had to do.
Live MPI library state. Open connections, queued sends, partially-completed collectives, and similar transient state in the MPI runtime cannot meaningfully be serialized to disk and reloaded, even with the same MPI library.

The split-process design addresses both at once:

The lower half — the only part that depends on the specific MPI and network libraries, and the only part that holds live network state — is excluded from the checkpoint image. Everything written to disk is upper-half memory, which references the lower half only through a stable function-pointer table. The same upper-half code therefore composes with any supported MPI implementation; MANA does not need a separate port per MPI.
The upper half holds only virtual MPI handles. Every "MPI handle" in the upper-half image is an opaque integer index into a mapping table, not a real pointer into MPI library memory. After restart the table is rewritten so each virtual ID maps to a fresh real handle in the new lower half.

The boundary: FS-register context switch

The two halves share one address space but each has its own copy of glibc and its own thread-local storage. Every MPI call from the user application crosses the boundary: the upper-half wrapper function in libmana.so saves the upper-half FS register, loads the lower-half FS register, calls the real MPI function pointer (whose address is in lh_info, the lower-half info struct), and then restores the upper-half FS register.

The FS register on x86_64 holds the base of thread-local storage. Swapping it cleanly is essential because both halves use libc-thread-local globals (errno, malloc arenas, pthread book-keeping, ...) and the wrong FS value causes immediate corruption.

The swap is implemented in mpi-proxy-split/lower-half/switch-context.{h,cpp}:

Fast path (Linux 5.9 or later, with FSGSBASE enabled). Use the rdfsbase / wrfsbase x86 instructions in user space (~20 cycles per swap).
Fallback (older kernels). Issue SYS_arch_prctl(ARCH_GET_FS, ...) / SYS_arch_prctl(ARCH_SET_FS, ...) syscalls (~500 cycles per swap, and visible to glibc's stack protector — see Build System).

On CentOS 7 (kernel < 5.9) the user-space fsgsbase instructions are unavailable and a few percent of additional runtime overhead is the expected cost. ./configure reports whether FSGSBASE is available; look for the line checking if FSGSBASE for x86_64 (setting fs in user space) is available.

Editing switch-context.cpp is high-risk. A mis-matched save/restore corrupts TLS in ways that segfault inside glibc internals (malloc, stdio) rather than at the call site, and debugging is correspondingly painful. Approach this file with care.

The boundary: handle virtualization

Every MPI handle the user application receives is a virtual ID, not the real handle returned by the underlying MPI library. The wrappers translate in both directions:

On every wrapper entry, virtual handles passed in by the application are translated to the real handles the lower half expects.
On every wrapper exit, real handles returned by the lower half are recorded in a virtual-ID table and only the virtual ID is given back to the application.

Predefined handles like MPI_INT, MPI_COMM_WORLD, etc., have fixed identities and do not need a virtual ID. Application-created handles do: MPI_Comm from MPI_Comm_split, MPI_Datatype from MPI_Type_contiguous, MPI_Request from MPI_Isend, MPI_Group, MPI_Op, MPI_File, and MPI_Win.

See Appendix C for the implementation.

The boundary: record-replay

Reconstructing the lower-half MPI state at restart time is done by replaying the calls that created it. The upper-half wrappers, every time they successfully create a non-predefined handle, append a record of the call (function + arguments) to a per-category log:

restoreComms for MPI_Comm_dup, MPI_Comm_create, MPI_Comm_split, ...
restoreTypes for MPI_Type_contiguous, MPI_Type_create_struct, ...
restoreGroups for MPI_Group_incl, MPI_Group_union, ...
restoreOps for user-defined MPI_Op_create.
restoreCarts for MPI_Cart_create, MPI_Cart_sub, ...

At restart, the lower half re-initializes MPI, then MANA replays these logs in order. The replay produces fresh real handles, and the virtual-ID table is rewritten to map each virtual ID to the fresh real handle. The upper half — which only ever held virtual IDs — observes no change.

Anything the upper half creates that needs to survive restart must go through the LOG_CALL macro in mpi-proxy-split/record-replay.h. When adding a new wrapper, see Appendix D.

Checkpoint quiesce: drain in two phases

A transparent MPI checkpoint must not leave in-flight messages between ranks at the moment of the snapshot. At checkpoint time, MANA quiesces the MPI traffic in two ordered phases before allowing DMTCP to write the image:

Collective drain. All ranks agree on a consistent ordering of outstanding collectives (MPI_Allreduce, MPI_Bcast, ...) and ensure each rank has finished entering and exiting collectives up to a shared sequence number. This uses a topological-sort-based sequence-number algorithm; see the Cluster'24 paper (Xu and Cooperman).
Point-to-point drain. Each rank polls outstanding non-blocking receives, probes for unexpected messages, and confirms global_sent_messages == global_recv_messages. If a rank is blocked inside MPI_Recv at the time of the checkpoint request, MANA injects a dummy MPI_Send to unblock it; the dummy is discarded after the checkpoint completes. This implements the "probe-before-recv" strategy introduced in the SuperCheck-SC23 paper.

Application code observes these phases as a brief pause at checkpoint time. The pause length is proportional to the volume of outstanding MPI traffic, not to the size of the application memory.

For internals, see Appendix C; the relevant source files are seq_num.{h,cpp} and p2p_drain_send_recv.{h,cpp}.

The restart problem

Restart faces a chicken-and-egg constraint. We want to do, in order:

Initialize the lower half's libc and libmpi.
Call MPI_Init in the lower half to acquire an MPI rank.
Use that rank to choose which checkpoint image to restore.

Step 1 needs a stack pointer (for glibc startup), and the stack we want is the one in the checkpoint image — but the image is not selected until step 3. MANA's workaround is described in mpi-proxy-split/NOTES. In summary:

The lower half is initialized using mtcp_restart's temporary stack.
MPI_Init is called with NULL, NULL (the MPI standard does not require these to be non-null; all tested implementations tolerate it).
After memory restoration, mremap moves the VDSO/VVAR pages. MANA assumes those pages are at identical addresses across MPI ranks so glibc's cached VDSO pointers remain valid.

This is the single most fragile area of MANA. Symptoms of VDSO mismatch or stack corruption usually appear as segfaults in glibc internals very early in restart, before any user code runs.

Appendix C: Internals of MPI Wrapper functions

This appendix documents the MPI wrapper machinery at the level a plugin developer needs. It mirrors Appendix B: Base Code Internals in DMTCP_USER_MANUAL.md; the latter remains the reference for the DMTCP layer beneath these wrappers.

File layout under `mpi-proxy-split/`

mpi_plugin.{h,cpp} — DMTCP plugin entry points; registers all MANA hooks; owns the mana_state_t state machine.
mpi_files.h — header listing every .cpp that compiles into libmpiwrappers.a.
mpi_plugin.h — public plugin types: mana_state_t and the MANA event names.
mpi-wrappers/ — one .cpp per category of MPI function; see Wrapper categories below.
seq_num.{h,cpp} — sequence-number algorithm for the collective drain.
p2p_drain_send_recv.{h,cpp} — P2P drain (probe-before-recv).
p2p_log_replay.{h,cpp} — MANA_P2P_LOG / MANA_P2P_REPLAY log handling.
record-replay.{h,cpp} — LOG_CALL macro and per-category replay logs.
virtual_id.{h,cpp} — bidirectional virtual-ID to/from real-handle map.
uh_wrappers.{h,cpp} — upper-half-only wrappers (e.g. fork).
lower-half/ — the lower-half loader, switch-context.cpp, and FS-register helpers.
mana_header.h and mana_coord_proto.h — the header metadata file format and the protocol MANA uses to talk to the DMTCP coordinator beyond DMTCP's own messages.
cartesian.h — Cartesian topology helpers used by mpi_cart_wrappers.cpp and the replay code.

The MANA plugin and its DMTCP hooks

mpi_plugin.cpp registers a single DmtcpPluginDescriptor_t named mpi_plugin. The event hook switches on event type and dispatches to the actions listed below. Events are grouped by lifecycle phase.

Process lifecycle

DMTCP_EVENT_INIT : Initialize sequence-number state; set up signal handlers; create the upper-half FS-base tracking map; mmap a guard page just past the heap so glibc's main arena does not grow into the area mtcp_restart later reserves.

DMTCP_EVENT_EXIT : Tear down sequence-number locks.

DMTCP_EVENT_RUNNING : Close checkpoint-time file descriptors that were opened in DMTCP_EVENT_PRECHECKPOINT.

Thread management (needed by SwitchContext)

DMTCP_EVENT_PTHREAD_START : Record the new thread's FS base in the upper-half FS-base map.

DMTCP_EVENT_PTHREAD_EXIT and DMTCP_EVENT_PTHREAD_RETURN : Remove the exiting thread's FS base from the map.

DMTCP_EVENT_THREAD_RESUME : Record the new FS base for a thread that has just been resumed after restart.

File-descriptor tracking

DMTCP_EVENT_OPEN_FD : Log opened paths and flags for the file checkpoint filter (MANA_FILE_REGEX).

Checkpoint quiesce

DMTCP_EVENT_PRESUSPEND : Run the collective drain (sequence-number algorithm), then the P2P drain (probe-before-recv); open checkpoint-time file descriptors.

DMTCP_EVENT_PRECHECKPOINT : Snapshot the MPI init-time memory maps; record open file descriptors; write the MANA header metadata file.

DMTCP_EVENT_RESUME : Reset drain counters and sequence-number flags; return the rank to RUNNING.

Restart

DMTCP_EVENT_RESTART : Rebind the wrapper-known lower-half function addresses to the new lower half; replay the record-replay logs (restoreComms, restoreTypes, restoreGroups, restoreOps, restoreCarts); replay logged P2P messages; reopen MPI files.

For the DMTCP-level mechanics behind these hooks (registration order, forward vs. reverse dispatch, the four internal coordinator barriers), see Appendix C: Plugin Internals in DMTCP_USER_MANUAL.md.

Wrapper categories

Each .cpp under mpi-wrappers/ covers one category of MPI function.

Hand-written wrappers

mpi_wrappers.cpp : MPI_Init, MPI_Finalize, MPI_Comm_rank, MPI_Comm_size, and other init-time MPI calls.

mpi_p2p_wrappers.cpp : Point-to-point — MPI_Send, MPI_Recv, MPI_Isend, MPI_Irecv, MPI_Sendrecv, the MPI_Wait* family, the MPI_Test* family, and so on.

mpi_collective_wrappers.cpp : Collectives — MPI_Allreduce, MPI_Bcast, MPI_Reduce, MPI_Gather*, MPI_Scatter*, MPI_Alltoall*, and so on. The sequence-number drain calls live here.

mpi_collective_p2p.c : Debug only. Emulates each collective as a sequence of MPI_Send / MPI_Recv calls; activated by MPI_COLLECTIVE_P2P.

mpi_comm_wrappers.cpp : Communicator management — MPI_Comm_dup, MPI_Comm_create, MPI_Comm_split, MPI_Comm_free, and so on. Each successful call records a restoreComms log entry.

mpi_group_wrappers.cpp : Group construction — MPI_Group_incl, MPI_Group_excl, the union / intersection / difference operations, etc. Logged under restoreGroups.

mpi_type_wrappers.cpp : Datatype construction — MPI_Type_contiguous, MPI_Type_vector, MPI_Type_create_struct, MPI_Type_commit, MPI_Type_free, and so on. Logged under restoreTypes.

mpi_op_wrappers.cpp : User-defined reduction operators — MPI_Op_create, MPI_Op_free. Logged under restoreOps.

mpi_cart_wrappers.cpp : Cartesian topology — MPI_Cart_create, MPI_Cart_map, MPI_Cart_shift, MPI_Cart_sub, and so on. Logged under restoreCarts.

mpi_request_wrappers.cpp : MPI_Request_* queries and conversions.

mpi_file_wrappers.cpp : MPI-IO — MPI_File_open, the MPI_File_read* family, the MPI_File_write* family, MPI_File_close, and so on.

mpi_win_wrappers.cpp : One-sided communication — the MPI_Win_* family.

mpi_error_wrappers.cpp : MPI_Error_string, MPI_Error_class.

Generated wrappers

Three .cpp files in mpi-wrappers/ are produced from .txt definition files by Python generators that the Makefile runs automatically:

mpi_fortran_wrappers.cpp — Fortran bindings. Generated by generate-mpi-fortran-wrappers.py from mpi_fortran_wrappers.txt.
mpi_unimplemented_wrappers.cpp — NOT_IMPL stubs for MPI functions MANA does not wrap. Generated by generate-mpi-unimplemented-wrappers.py from mpi_unimplemented_wrappers.txt.
libmpistub.so symbol exports — used by mpicc_mana-built binaries. Generated by generate-mpi-stub-wrappers.py from mpi_stub_wrappers.txt.

A fourth helper, p2p-deterministic.c, is hand-written but driven by p2p-deterministic.txt, which lists the P2P functions that MANA_P2P_LOG records for deterministic replay.

All hand-written and generated .cpp files compile into libmpiwrappers.a, which is linked whole-archive into libmana.so. That preserves every MPI_* symbol regardless of whether the upper half actively references it at link time.

Anatomy of a typical wrapper

A wrapper has three jobs: translate virtual handles to real handles on the way in, call the lower-half implementation, translate any new real handles back to virtual IDs on the way out, and (if the call mutates persistent state) log the call so restart can replay it.

// From mpi_type_wrappers.cpp (lightly elided).
int MPI_Type_contiguous(int count, MPI_Datatype oldtype,
                        MPI_Datatype *newtype) {
  int retval;
  oldtype = get_real_datatype(oldtype);          // 1. virtual -> real
  DMTCP_PLUGIN_DISABLE_CKPT();                   // 2. lock out checkpoints
  JUMP_TO_LOWER_HALF(lh_info.fsaddr);             //    swap FS register
  retval = NEXT_FUNC(Type_contiguous)(count, oldtype, newtype);
  RETURN_TO_UPPER_HALF();
  DMTCP_PLUGIN_ENABLE_CKPT();                    //    release lock
  if (retval == MPI_SUCCESS && MPI_LOGGING()) {
    *newtype = new_virt_datatype(*newtype);      // 3. real -> virtual
    LOG_CALL(restoreTypes, Type_contiguous,      // 4. log for replay
             count, oldtype, *newtype);
  }
  return retval;
}

The relevant macros:

JUMP_TO_LOWER_HALF / RETURN_TO_UPPER_HALF — construct/destruct a SwitchContext RAII guard.
NEXT_FUNC(<name>) — the function pointer in lh_info for the real MPI_<name> in the lower half.
DMTCP_PLUGIN_DISABLE_CKPT() / _ENABLE_CKPT() — prevent checkpoint interruption during the lower-half call (the wrapper releases the lock before returning to user code).
LOG_CALL(<bucket>, <fn>, args...) — append a record to the named replay log. Only fires when MPI_LOGGING() is true (during normal execution).
new_virt_* and get_real_* — virtual-ID table accessors.

Handle virtualization

The virtual-ID table (virtual_id.{h,cpp}) is one C++ unordered_map per category, with a fresh, monotonically-increasing integer assigned for every newly-created handle. Predefined handles (MPI_COMM_WORLD, MPI_INT, ...) bypass the table and pass through unchanged.

The categories with virtual IDs:

MPI_Comm → real MPI_Comm.
MPI_Datatype → real MPI_Datatype (predefined types only).
MPI_Group → real MPI_Group.
MPI_Op → real MPI_Op (user-defined only).
MPI_Request → real MPI_Request.
MPI_File → real MPI_File.
MPI_Win → real MPI_Win.

The header virtual_id.h exposes the accessors used by every wrapper:

MPI_Comm     get_real_comm(MPI_Comm virtual_comm);
MPI_Comm     new_virt_comm(MPI_Comm real_comm);
MPI_Datatype get_real_datatype(MPI_Datatype virtual_type);
MPI_Datatype new_virt_datatype(MPI_Datatype real_type);
// ... similarly for Group, Op, Request, File, Win.

Record-replay

record-replay.h defines LOG_CALL(bucket, fn, args...). Each bucket is one of restoreComms, restoreTypes, restoreGroups, restoreOps, restoreCarts. At checkpoint time the bucket is serialized into the upper-half image; at restart time mpi_plugin.cpp:restoreMpiLogState() iterates each bucket and re-issues every recorded call to the lower half, capturing the fresh real handle and writing it back into the existing virtual-ID slot.

Arguments to LOG_CALL are deep-copied — buffer pointers are followed so that the replay does not depend on the application's runtime buffers still being live. The FncArg discriminated union in record-replay.h holds the copy.

Collective drain (sequence numbers)

seq_num.{h,cpp} implements the algorithm described in Enabling Practical Transparent Checkpointing for MPI: A Topological Sort Approach (Cluster'24). The key entry points:

commit_begin() — called from every collective wrapper before the lower-half call. Increments the per-collective counter.
commit_finish() — called after the lower-half call returns.
share_seq_nums() / check_seq_nums() — at checkpoint time, ranks exchange counters and confirm consensus. If a rank's count is behind the global maximum on any collective, it must complete more iterations before the checkpoint can proceed.

The algorithm is local-write, global-readback, polling-based. It does not require an extra synchronization point in non-checkpoint execution; the overhead in the steady state is a small per-call atomic increment.

P2P drain (probe-before-recv)

p2p_drain_send_recv.{h,cpp} implements the probe-before-recv protocol. The wrappers maintain per-rank counters local_sent_messages and local_recv_messages. At checkpoint time:

drainInFlightP2p() polls outstanding non-blocking sends/receives and probes for unexpected messages. It continues until a global reduction shows sum(local_sent) == sum(local_recv).
unblockPendingRecvs() walks the table of ranks currently inside a blocking MPI_Recv and injects a dummy MPI_Send of zero bytes to each. The receiver's wrapper recognizes the dummy by a distinguishing tag (p2p_dummy_phase is set during the drain) and discards it after the checkpoint completes.

The P2P log (p2p_log_replay.{h,cpp}) is independent of the drain; it records the order of MPI_Send/Recv family calls so that MANA_P2P_REPLAY can reproduce them deterministically at restart.

switch-context

lower-half/switch-context.{h,cpp} implements the FS-register swap described in Appendix B. The interface is the RAII class SwitchContext:

class SwitchContext {
  unsigned long upperFs;
public:
  SwitchContext(unsigned long lowerFs);  // save upper, load lower
  ~SwitchContext();                       // restore upper
};

Used through the JUMP_TO_LOWER_HALF / RETURN_TO_UPPER_HALF macros in mpi_files.h. The class also handles a re-entrancy edge case: if the upper-half code calls a wrapper that calls another wrapper (rare but possible through callback paths), the nested SwitchContext must save and restore the correct FS even though it is "already in the lower half".

The lower-half info structure

The lower half publishes lh_info (declared in mpi-proxy-split/lower-half/lower-half-api.h) to the upper half. It contains:

The lower-half text/data ranges (startText, endText, startData, endOfHeap) — used by DMTCP to know which memory to skip when writing the checkpoint.
The lower-half FS base (fsaddr) — used by SwitchContext.
Function pointers for every real MPI function the upper half might call, plus a few from libc (mmap, munmap, sbrk).

At launch, the lower half's main passes the address of lh_info to the upper half via MANA_LH_INFO_ADDR. At restart this is the addresses-change problem: the upper half remembers the addresses from before checkpoint, but the lower half has been re-loaded and the addresses are different. MANA fixes this in the DMTCP_EVENT_RESTART hook by re-reading lh_info and rewriting every wrapper's NEXT_FUNC slot.

Appendix D: Developers: Adding or Modifying MPI Wrappers

When you need to add a wrapper for an MPI function MANA does not yet cover, or modify an existing one, here is the checklist.

1. Add the wrapper

In the appropriate mpi-proxy-split/mpi-wrappers/mpi_<category>_wrappers.cpp. Use the existing wrappers in the same file as templates. The skeleton:

int MPI_NewFunc(... args ...) {
  int retval;
  // Translate any virtual handles in the args to real handles.
  comm = get_real_comm(comm);
  datatype = get_real_datatype(datatype);
  ...
  DMTCP_PLUGIN_DISABLE_CKPT();
  JUMP_TO_LOWER_HALF(lh_info.fsaddr);
  retval = NEXT_FUNC(NewFunc)(... args ...);
  RETURN_TO_UPPER_HALF();
  DMTCP_PLUGIN_ENABLE_CKPT();
  if (retval == MPI_SUCCESS && MPI_LOGGING()) {
    // If the call returns a new handle, virtualize it.
    *newhandle = new_virt_<kind>(*newhandle);
    // If the call mutates state that must survive restart, log it.
    LOG_CALL(restore<Bucket>, NewFunc, ... args ...);
  }
  return retval;
}

Rules of thumb:

Translate every input handle. Forgetting one leads to the lower half receiving a virtual ID and crashing or producing nonsense.
Virtualize every output handle. If the application sees a real handle, the next call passing it back in will fail virtual-ID lookup.
Wrap with DMTCP_PLUGIN_DISABLE_CKPT() ... _ENABLE_CKPT(). Without this, a checkpoint that arrives mid-wrapper (between the FS swap and the lower-half call) can leave the rank in CKPT_COLLECTIVE with the FS register pointing at the wrong half.
LOG_CALL only when the call mutates persistent MPI state. Persistent state means anything that has to be re-issued at restart to bring the lower half back to the same configuration: communicator creation, datatype construction, group operations, op creation, Cartesian topology setup, attribute updates. MPI_Send, MPI_Recv, collectives, and queries do not need LOG_CALL.

2. If the function returns a new handle, extend `virtual_id.cpp`

Most categories already have new_virt_* and get_real_* helpers; adding a new MPI function that produces an MPI_Comm (or Group, Type, Op, Request, File, Win) generally needs no changes to virtual_id.cpp.

For a new category of handle, add the table and accessors. This is unusual; the categories above cover essentially all MPI types.

3. Update the replay routine

If you added a LOG_CALL(restore<Bucket>, NewFunc, ...), ensure the restore routine in record-replay.cpp handles the NewFunc opcode. The restore routines are dispatched on the function name; an unrecognized opcode aborts at restart.

4. Generate Fortran bindings (if needed)

If the function needs a Fortran entry point, add it to mpi-wrappers/mpi_fortran_wrappers.txt:

MPI_NewFunc

make will re-run generate-mpi-fortran-wrappers.py and emit the lowercase Fortran wrapper that defers to the C wrapper.

5. Don't leave undefined MPI functions

Any MPI function MANA does not wrap and is referenced by an application binary will fail at link or load time. If you cannot write a real wrapper, add the function name to mpi-wrappers/mpi_unimplemented_wrappers.txt. The generator emits a NOT_IMPL stub that aborts cleanly with a clear error message if the application ever calls it.

6. Rebuild

make clean
make           # rebuilds libmpiwrappers.a then libmana.so

make enforces the ordering: libmpiwrappers.a first, then libmana.so is linked against it whole-archive.

7. Add a test

Drop a .c file into mpi-proxy-split/test/ named NewFunc_test.c. The Makefile auto-builds both NewFunc_test.exe and NewFunc_test.mana.exe. Add the test name to autotest.py's list to run it in CI.

If the wrapper touches the replay path, add a unit test under mpi-proxy-split/unit-test/ following the existing record-replay-*-test.cpp pattern.

8. Watch the linter

util/hooks/pre-commit runs the DMTCP / MANA style checker on every staged file. The most common gotchas:

License header at the top of every new .cpp / .h.
80-column line limit.
.clang-format enforcement: Google base style with Linux brace style and BinPackParameters: false.

License

MANA is licensed under the GNU Lesser General Public License v3 (LGPL v3), matching DMTCP. The public header for the DMTCP-side API (dmtcp.h) is in the public domain to allow unrestricted static linking. See LICENSE and dmtcp/LICENSE in the source tree.

Uh oh!