Pilgrim is a scalable and near lossless MPI Tracing library. It is lossless for MPI functionality but lossy for non-MPI metadata. Pilgrim captures the complete set of MPI calls and all their parameters. To ensure completeness, the interception codes are generated automatically from the MPI standard.
Chen Wang, Pavan Balaji, and Marc Snir. "Pilgrim: Scalable and (near) Lossless MPI Tracing", SC, 2021
Chen Wang, Yanfei Guo, Pavan Balaji, and Marc Snir. "Near-Lossless MPI Tracing and Proxy Application Autogeneration", IEEE TPDS, 2022
cd /PATH/TO/pilgrim ./autogen.sh ./configure make make install prefix=`pwd` // install to the current directory
(1) Storing parameters of CPU memory pointers
Pilgrim tracks memory management operations such as malloc, calloc, free, etc. The memory pointers are encoded in symbolic representations, which allows post-processings such as memory reuse analysis. To store parameters of memory pointers, i.e., void*, use
(2) Storing parameters of CUDA memory pointers
Pilgrim can also track CUDA memory operations such as cudaMalloc, cudaMallocPitch, cudaFree, etc. In addition, the CUDA device where the memory was allocated is also preserved. To enable this feature and store parameters of CUDA memory pointers, use
(3) Storing thread id of each call
For multithreaded MPI programs, where MPI calls can be made from different threads, Pilgrim can preseve the thread id of each MPI call. To enable this feature, use
Pilgrim supports all thread levels defined by the MPI standard.
(4) Specifying lossy timing compression library
Pilgrim provides three modes to store the timing information.
- Aggregated: Only store statistics of MPI functions, i.e., average duration and interval of each function.
- Lossless: Store each call's duration and interval in a lossless mode. Lossless compression algorithm ZSTD can be used in this mode.
- Lossy: Store each call's duration and interval in a lossy mode. Several lossy algorithms are provided.
The last two algorithms require external libraries SZ and ZFP to be installed.
- CFG: Use the same context-free-grammar to store the timestamps.
- HIST: Use the histogram-based compression algorithm.
- SZ: Use the lossy compressor SZ. Require
- ZFP: Use the lossy compressor ZFP. Require
mpirun -np N -env LD_PRELOAD /PATH/TO/pilgrim/lib/libpilgrim.so ./your_app
mpirun can be replaced with your job scheduler, e.g., srun.
PILGRIM_TIMING_MODE can be set to one of the options below:
- AGGREGATED: Store only statistics of each MPI function.
- LOSSLESS: Store lossless timestamps without any compression.
- ZSTD: Store lossless timestamps but with ZSTD compression.
- CFG: Store lossy timestamps using the CFG compression algorithm.
- HIST: Store lossy timestamps using the HIST compression algorithm.
- SZ: Store lossy timestamps using the SZ lossy compressor.
- ZFP: Store lossy timestamps using the ZFP lossy compressor.
- DEFAULT: Tracing is enabled by default. Call
MPI_Info_set(info, "PILGRIM_TRACING", "OFF")to disable tracing and
MPI_Info_set(info, "PILGRIM_TRACING", "ON")to enable tracing.
- DYNAMIC: Tracing is disabled by default. Call
MPI_Info_set(info, "PILGRIM_TRACING", "ON")to enable tracing and
MPI_Info_set(info, "PILGRIM_TRACING", "OFF")to disable tracing.
PILGRIM_DEBUG: set to 1 to allow debug output.