MPI Profiler

MPI Profiler (alias mprof) is a tool to profile MPI functions. It uses an interposition library, named libmprof.so. A wrapper is also provided, named mprof.

Disclaimer

This tool profiles only MPI_COMM_WORLD communication (others communication will work but not profiled) and it is NOT currently thread safe.

Therefore, we recommend you to run this tool only with full MPI applications and with only MPI_COMM_WORLD communicator.

Also, we assume that your code following the few MPI rules:

• you don’t call 2 times or more MPI_Init or MPI_Finalize
• MPI_Init and MPI_Finalize should wrap almost all your code

Dependencies

Please ensure that following dependencies are installed in your system:

• gcc
• make
• MPI library (any vesrion)
• coreutils (install)

Example of installation on Arch Linux

$ sudo pacman -S --needed gcc make openmpi coreutils

Installation

$ git clone https://github.com/Sholde/mprof.git
$ cd mprof
$ make
$ sudo make install

Usage

It is easy to use. You just need to put mprof word ahead your MPI execution line. For example:

$ mprof mpiexec -np 4 ./a.out
or
$ mprof mpirun -np 4 ./a.out

List of options

mprof provides different command line options:

• -v, --version print the current version
• -h, --help print a friendly help message
• -l, --list print the list of environment variable options
• --options=... allow you to put environment variable options directly in the command line. Note that you must remove hyphen --.
• --clean clean all mprof files.
• --summary=PROCESS_ID print the file that contain the summary of process PROCESS_ID

Environment variable

mprof allows you to specify different options in an environment variable, named MPROF_OPTIONS.

NOTE: the number of file created correspond to the number of MPI process. We advise you to run mprof with a reduce number of process when you use an environment variable option.

List of options

• --verbose print in a file named mprof_ProcessId.out, all call of MPI
• --barrier print in a file named mprof_ProcessId.out, all call of MPI_Barrier
• --init print in stderr all call of MPI_Init with all argument
• --finalize print in stderr all call of MPI_Finalize
• --profile print in a file named mprof_ProcessId.profile, all information of profiling, like contiguous sends

Example of usage with deadlock

Here it is a code where only the process 0 call MPI_Barrier and others finished.

$ MPROF_OPTIONS="--finalize" mprof mpirun -np 4 ./lbm
=================== CONFIG ===================
iterations           = 16
width                = 800
height               = 160
obstacle_r           = 17.000000
obstacle_x           = 161.000000
obstacle_y           = 83.000000
reynolds             = 100.000000
reynolds             = 100.000000
inflow_max_velocity  = 0.100000
output_filename      = result.raw
write_interval       = 50
------------ Derived parameters --------------
kinetic_viscosity    = 0.034000
relax_parameter      = 1.661130
==============================================
 RANK 0 ( LEFT -1 RIGHT -1 TOP -1 BOTTOM 1 CORNER -1, -1, -1, -1 ) ( POSITION 0 0 ) (WH 802 42 ) 
 RANK 2 ( LEFT -1 RIGHT -1 TOP 1 BOTTOM 3 CORNER -1, -1, -1, -1 ) ( POSITION 0 80 ) (WH 802 42 ) 
 RANK 1 ( LEFT -1 RIGHT -1 TOP 0 BOTTOM 2 CORNER -1, -1, -1, -1 ) ( POSITION 0 40 ) (WH 802 42 ) 
 RANK 3 ( LEFT -1 RIGHT -1 TOP 2 BOTTOM -1 CORNER -1, -1, -1, -1 ) ( POSITION 0 120 ) (WH 802 42 ) 
Progress [    1 /    16]
Progress [    2 /    16]
Progress [    3 /    16]
Progress [    4 /    16]
Progress [    5 /    16]
Progress [    6 /    16]
Progress [    7 /    16]
Progress [    8 /    16]
Progress [    9 /    16]
Progress [   10 /    16]
Progress [   11 /    16]
Progress [   12 /    16]
Progress [   13 /    16]
Progress [   14 /    16]
Progress [   15 /    16]
==mprof== Process 3 enter in MPI_Finalize
==mprof== Process 1 enter in MPI_Finalize
==mprof== Process 2 enter in MPI_Finalize
^C$

Example of usage with contigous send

A little benchmark where Process 0 send to Process 1 an array of 2 integer but one by one.

$ MPROF_OPTIONS="--profile" mprof mpirun -np 2 ./a.out
==mprof== PROFILE: Process 0 send independently elements which are contiguous to Process 1
==mprof==             - Sending 1 element(s) of MPI_INT
===============================================================================
================================= MPI PROFILER ================================
===============================================================================
==mprof== GLOBAL SUMMARY:
==mprof==               running: 74 us
==mprof==          message sent: 2 msg take 8 bytes - waiting 26 us in total
==mprof==          message recv: 2 msg take 8 bytes - waiting 44 us in total
==mprof==     barrier(s) passed: 0 - waiting 0 ns in total
==mprof==            warning(s): 1 - 1 contiguous send
==mprof== 
==mprof== LOCAL SUMMARY (Process 0):
==mprof==               running: 46 us
==mprof==          message sent: 2 msg take 8 bytes - waiting 26 us (max: 25 us)
==mprof==          message recv: 0 msg take 0 bytes - waiting 0 ns (max: 0 ns)
==mprof==     barrier(s) passed: 0 - waiting 0 ns (max: 0 ns)
==mprof==       list(s) sent to: 1
==mprof==     list(s) recv from:
==mprof==            warning(s): 1 - 1 contiguous send
==mprof== 
==mprof== LOCAL SUMMARY (Process 1):
==mprof==               running: 74 us
==mprof==          message sent: 0 msg take 0 bytes - waiting 0 ns (max: 0 ns)
==mprof==          message recv: 2 msg take 8 bytes - waiting 44 us (max: 35 us)
==mprof==     barrier(s) passed: 0 - waiting 0 ns (max: 0 ns)
==mprof==       list(s) sent to:
==mprof==     list(s) recv from: 0
==mprof== 
==mprof== ERROR SUMMARY:
==mprof==          No errors