PGMPI - Performance Guideline Verification Tool for MPI Collectives

Introduction

The PGMPI framework is designed to verify self-consistent performance guidelines of MPI libraries. It enables users to configure an experiment on their local machine and then execute it on a (possibly) remote parallel machine (target machine).

The main steps of a typical PGMPI execution are the following:

• NREP Prediction step
• Experiment Execution step
• Data Analysis step.

A detailed description of each of these steps can be found in our paper:

• S. Hunold, A. Carpen-Amarie, F.D. Lübbe and J.L. Träff, “Automatic Verification of Self-Consistent MPI Performance Guidelines”, EuroPar (2016)

Installation

On the local machine

• Python 2.7
• R
• PGMPI

Note that PGMPI is only required on the local machine where the experiment is configured.

This tutorial assumes that the pgmpi/bin directory is exported to the current PATH, as well as the python and R binary paths.

On the target machine

• MPI library
• the ReproMPI Benchmark (version >= 1.0.0) compiled for the target MPI library

Configuration Files

Three configuration files are required to define the set of interesting performance guidelines, the guideline measurement setup, and finally, the experiment configuration.

After the experiment initialization, default versions of the configuration files will be created in the config subdirectory.

exp_dir
    └── config
        ├── experiment_conf.py
        ├── guideline_catalog.json
        └── guideline_conf.json

Note that the PGMPI tool needs to access these files on this predefined location. Their contents need to be modified to fit the experiment purpose.

Evaluated Performance Guidelines (guideline_catalog.json)

The PGMPI tool is designed to verify three types of performance guidelines for MPI collectives:

• Pattern guidelines
• Monotony guidelines
• Split-robustness guidelines.

The full list of supported performance guidelines can be found in our paper.

The input file specifying the performance guidelines to be tested is a json file with the following format:

[   
      {         
            "orig": mpi_func1,
            "mock": comparison_func1,
            "msizes":  list of message sizes to be tested
       },
      {         
            "orig": mpi_func2,
            "mock": comparison_func2,
            "msizes":  list of message sizes to be tested
       },
      {         
            "orig": mpi_func1,
            "msizes":  list of message sizes to be tested
       },
.....
]

Pattern guidelines are defined by an MPI collective call (named orig) and a mock-up version (mock). The guidelines that only define the orig function will be evaluated for both monotony and split-robustness violations.

Each of the collectives specified in this file is benchmarked only once, for each the message sizes that are needed for the guidelines in which the collective function is involved.

Example

[   
      {         
            "orig": "MPI_Gather",
            "mock": "MPI_Gather_with_MPI_Allgather",
            "msizes":  [100, 1024, 8192]
       },
       {
            "orig": "MPI_Gather",
            "mock": "MPI_Gather_with_MPI_Reduce",
            "msizes":  [2, 3, 4, 100, 1024, 8192]
       },
       {
            "orig": "MPI_Bcast",
            "mock": "MPI_Bcast_with_MPI_Scatter_MPI_Allgather",
            "msizes":  [100, 1024, 8192, 10000]
       },
       {
            "orig": "MPI_Bcast",
            "msizes":  [1, 2, 4, 8, 16, 32, 64, 100, 1024, 8192, 10000]
       }
]

Guideline Configuration (guideline_conf.json)

This input file defines the parameters of the experiment execution for both the prediction step and the measurement step.

{
"procs": 8,
"nodes": 1,
"nnp": 8,

"nmpiruns": 10,

"prediction": {
  "min": 10,
  "max": 1000,
  "step": 5,
  "methods": ["rse","cov_mean"],   
  "thresholds": [0.01, 0.02],
  "windows": [1,30],
  "nmpiruns": 3
  }
}

It contains two types of parameters:

• parameters that are relevant for both steps
  • procs - total number of processes
  • nodes - number of nodes
  • nnp - number of processes per node
• specific parameter for the measurement step
  • nmpiruns - the number of repetitions of the measurement task.
• parameters that configure the prediction step
  • min/max - limits of the number of repetitions for each pair MPI functions/message size
  • step - the increment applied to the previous number of repetitions before checking whether the prediction conditions are fulfilled; the step value doubles at each iteration. Thus, at each iteration i, the number of measurements nrep is

    nrep(0)=min
    nrep(i) = nrep(i-1) + step * 2^(i-1)
            

  • methods - list of evaluation methods used to stop the prediction step (rse, cov_mean, cov_median)
  • thresholds/windows - list of thresholds and (non-zero) measurement windows corresponding to the specified prediction methods. Rse does not rely on a measurement window, however a dummy window value is required in the windows list when multiple methods are used.
  • nmpiruns - number of repetitions of the prediction task (among which the number of repetitions is selected)

Experiment Configuration (experiment_conf.py)

The experiment configuration file is a Python class that specifies the execution environment and setup of the entire experiment.

An example can be found in pgmpi/examples/slurm_cluster1/experiment_conf.py.

from pgmpi.glexp_desc import abs_exp_desc
from pgmpi.experiment import glexp
from pgmpi.benchmark import reproMPIbench
from pgmpi.machsetup import slurm_cluster


class ExpDescription(abs_exp_desc.AbstractExpDescription):

    # Path to the ReproMPI benchmark binaries on the target machine 
    # (more info on how to install ReproMPI can be found here: 
    # https://github.com/hunsa/reprompi)
    benchmark_path_remote = "/home/carpenamarie/code/mpibenchmark-1.0.0/bin"

    def setup_exp(self):
        
        bench    = reproMPIbench.GLReproMPIBench(self.benchmark_path_remote)
        machinfo = slurm_cluster.PGMPIMachineConfiguratorSlurm()
   
        exp = glexp.GLExperimentWriter(bench, machinfo)

        return exp    

The ExpDescription extends the abs_exp_desc.AbstractExpDescription abstract class, which requires derived classes to implement only one method.

def setup_exp(self):
    .....

This method generates an experiment object that is later used by PGMPI to perform the guideline violations detection.

The type of the experiment object is glexp.GLExperimentWriter, and it has the following constructor interface:

def __init__(self, benchmark, machine_configurator)

The parameters have to be defined by the user as follows:

• benchmark: indicates which benchmark will be used for conducting the experiment. Currently, PGMPI only supports the ReproMPI benchmark, but other tools can potentially be used, as long as the user provides a python class implementing the interface that PGMPI requires to produce results (which can be found in pgmpi/lib/benchmark/abs_benchmark.py).

  bench    = reproMPIbench.GLReproMPIBench(self.__benchmark_path_remote)
      

• machine_configurator: object that holds information regarding the MPI library installed on the target machine and the format of job files for that particular machine. For instance, in the previous example, the target machine is a cluster that uses SLURM to submit jobs and to execute MPI code.

  PGMPI provides two predefined machine configurator classes:

  • PGMPIMachineConfiguratorLocal it can be used for local experiments (where mpirun is available in the PATH and no batch job submission is required)

    pgmpi/lib/machsetup/machine_setup_local.py
            

  • PGMPIMachineConfiguratorSlurm it is designed to generate jobs for a machine that relies on SLURM for the batch submission of MPI jobs.

    pgmpi/lib/machsetup/slurm_cluster.py
            

  In other cases, the users will have to define their own machine configurator classes suitable to their clusters.

Using the PGMPI Tool for a Local Experiment

This section describes how to verify a set of MPI performance guidelines on the local machine.

It requires that the following conditions are met:

• an MPI library is installed and the mpirun command is available in the PATH
• the ReproMPI benchmark is installed
• PGMPI is installed and pgmpi/bin is in the current PATH

Local Step 1: Experiment Initialization

Create a directory on the local machine, e.g., exp1.

$ mkdir exp1
$ cd exp1
$ pgmpirun.py init ./

The script will create the experiment directory tree in the current directory and copy a set of default configuration files into the config subdirectory.

Now the exp1 directory will have the following subdirectory structure:

exp1
    ├── 01-nrep_prediction_exp
    │   ├── jobs
    │   │   ├── input_files
    │   │   └── raw_data
    │   └── results
    │       └── summary
    ├── 02-experiment_exec
    │   ├── jobs
    │   │   ├── input_files
    │   │   └── raw_data
    │   └── results
    │       ├── alldata
    │       ├── plots
    │       └── summary
    └── config
        ├── experiment_conf.py
        ├── guideline_catalog.json
        └── guideline_conf.json

The config directory will contain the default version of the three configuration files:

• guideline_catalog.json the complete list of guidelines that can be verified with the PGMPI tool, each of them having a predefined set of message sizes to be tested.
• guideline_conf.json the configuration of the guideline-verification experiment.
• experiment_conf.py the experiment description.

Important note: The configuration files name or location cannot be changed.

To proceed, it is necessary to modify the benchmark_path_remote in the experiment_conf.py file to match the path to the ReproMPI binaries on your machine.

Local Step 2: Set up the Prediction Step

All following steps can be executed from the exp1 directory, without the need to specify the experiment directory at each step.

$ pgmpirun setup pred

This script will create the job file for the NREP prediction step for each of the MPI collectives specified in the guidelines catalog file. It will populate the 01-nrep_prediction_exp/jobs directory with a job file and an input file.

The NREP Prediction Step

All job and input files have been created in the exp1/01-nrep_prediction_exp/jobs directory. As in this case there is no remote machine, we can directly proceed to executing the job.

Execution of the Prediction Step

Important note: A job should only be executed from its parent directory.

$ cd 01-nrep_prediction_exp/jobs
$ sh job.sh 

This will execute the ReproMPI benchmark and generate output files in the raw_data subdirectory.

Local Step 3: Process Prediction Results

Continue the experiment from the exp1 directory on the local machine.

$ cd ../../
$ pgmpirun process pred

The obtained nrep values will be computed and written in the 01-nrep_prediction_exp/results directory.

Local Step 4: Set up the Measurement Step

$ pgmpirun setup verif

The script generates an experiment directory in 02_experiment_exec directory. The calls to the benchmark are configured to use the nrep parameter computed from the previously obtained prediction results.

The Measurement Step

The guideline verification job can now be directly executed.

Execution of the Measurement Step

$ cd 02_experiment_exec/jobs
$ sh job.sh 

This will execute the ReproMPI benchmark and generate output files in the raw_data subdirectory.

Local Step 5: Process the Measurement Step Data

The measurement data is collected in a single data file which can later be reused to compute guideline violations and a summary of the measured data is computed to facilitate the re-execution of the detection of guideline violations step.

The processing step has to be executed again from the root directory of the experiment.

$ cd ../../
$ pgmpirun process verif

Local Step 6: Analyze Step - Detection of Guideline Violations

The detection script will print the guideline violations that were found for the specified experiment.

$ pgmpirun check

Running the PGMPI Tool - Advanced Scenario

This tutorial describes the steps to use PGMPI for an experiment that targets a cluster where jobs can be submitted using the SLURM job scheduler.

We assume the experiment configuration is done on the user’s local machine, while the jobs have to be executed on a remote target machine.

Local Step 1: Experiment Initialization

Create a directory on the local machine, e.g., exp1.

$ mkdir exp1
$ cd exp1
$ pgmpirun.py init ./

The script will create the experiment directory tree in the current directory and copy the set of default configuration files into the config subdirectory.

Now the exp1 directory will have the following subdirectory structure:

exp1
    ├── 01-nrep_prediction_exp
    │   ├── jobs
    │   │   ├── input_files
    │   │   └── raw_data
    │   └── results
    │       └── summary
    ├── 02-experiment_exec
    │   ├── jobs
    │   │   ├── input_files
    │   │   └── raw_data
    │   └── results
    │       ├── alldata
    │       ├── plots
    │       └── summary
    └── config
        ├── experiment_conf.py
        ├── guideline_catalog.json
        └── guideline_conf.json

The config directory will contain the following files:

• guideline_catalog.json the complete list of guidelines that can be verified with the PGMPI tool, each of them having a predefined set of message sizes to be tested.
• guideline_conf.json the configuration of the guideline-verification experiment.
• experiment_conf.py the experiment description.

To proceed, it is necessary to modify the experiment_conf.py file to generate files that are suitable for the target SLURM cluster.

Several cluster configuration files are provided in pgmpi/examples, and they cover some common usage scenarios for a SLURM cluster.

For the purpose of this tutorial, we make the following assumptions:

• the SLURM job only requires the configuration of (some of) these parameters: partition, qos, account, walltime
• an MPI job can be executed within the SLURM job using srun

Experiment Configuration

A basic configuration file that enables the experiment to generate SLURM jobs can be found in:

pgmpi/examples/slurm_cluster1/experiment_conf.py   

Replace the exp1/config/experiment_conf.py file with the SLURM experiment configuration. It has the following contents:

from pgmpi.glexp_desc import abs_exp_desc
from pgmpi.experiment import glexp
from pgmpi.benchmark import reproMPIbench
from pgmpi.machsetup import slurm_cluster


class ExpDescription(abs_exp_desc.AbstractExpDescription):

    # Path to the ReproMPI benchmark binaries on the target machine 
    # (more info on how to install ReproMPI can be found here: 
    # https://github.com/hunsa/reprompi)
    benchmark_path_remote = "/home/carpenamarie/code/mpibenchmark-1.0.0/bin"

    def setup_exp(self):
   
        bench    = reproMPIbench.GLReproMPIBench(self.benchmark_path_remote)
        machinfo = slurm_cluster.PGMPIMachineConfiguratorSlurm(account = None, 
                                                               partition = None, 
                                                               qos = None, 
                                                               walltime = None)
        exp = glexp.GLExperimentWriter(bench, machinfo)

        return exp

To proceed to the next step of the experiment, the exp1/config/experiment_conf.py has to be updated to match the configuration of the target cluster. Thus, the following changes are necessary:

• benchmark_path_remote should point to the location of the ReproMPI binaries on the target machine
• the partition, qos, account, walltime variables should have suitable values for the target machine. Setting them to None means that no corresponding SBATCH variable will be set in the generated job files.

Optionally, the other two configuration files can be modified for each experiment. In particular, the number of nodes and nnp in the guideline_conf.json file need to be adapted for each experiment.

Local Step 2: Set up the Prediction Step

All following steps can be executed from the exp1 directory, without the need to specify the experiment directory at each step.

$ pgmpirun setup pred

This script will create the job file for the NREP prediction step for each of the MPI collectives specified in the guidelines catalog file. It will populate the 01-nrep_prediction_exp/jobs directory with a job file and an input file.

The NREP Prediction Step

As all job and input files have been created on the user local machine, the jobs directory in the exp1/01-nrep_prediction_exp/ has to be copied to the target machine, as instructed by the pgmpirun script output.

$ scp -r 01-nrep_prediction_exp/jobs user@remote_machine:$REMOTE_PRED_DIR

Execution of the Prediction Step (on the remote machine)

Now the prediction job can be executed on the remote machine. In the case of our SLURM cluster, the job can be submitted using the sbatch command.

Important note: The job file has to be executed from its parent directory.

$ cd $REMOTE_PRED_DIR/jobs
$ sbatch job.sh 

The job will execute the ReproMPI benchmark and generate output files in the raw_data subdirectory (which can be then copied back to the original machine).

$ scp -r  user@remote_machine:$REMOTE_PRED_DIR/jobs/raw_data 01-nrep_prediction_exp/jobs/

Local Step 3: Process Prediction Results

Continue the experiment from the exp1 directory on the local machine.

$ pgmpirun process pred

The obtained nrep values will be computed and written in the 01-nrep_prediction_exp/results directory.

Local Step 4: Set up the Measurement Step

$ pgmpirun setup verif

The script generates an experiment directory in 02_experiment_exec directory. The calls to the benchmark are configured to use the nrep parameter computed from the previously obtained prediction results.

The Measurement Step

The experiment files have to be copied to the remote machine again.

$ scp -r 02_experiment_exec/jobs user@remote_machine:$REMOTE_DIR

Execution of the Measurement Step (on the remote machine)

The experiment can now be executed on the remote machine.

$ cd $REMOTE_DIR/jobs
$ sbatch job.sh 

This will execute the ReproMPI benchmark and generate output files in the raw_data subdirectory (which can be then copied back to the original machine).

$ scp -r  user@remote_machine:$REMOTE_DIR/jobs/raw_data  02_experiment_exec/jobs/

Local Step 5: Process Measurement Data

The measurement data is collected in a single data file which can later be reused to compute guideline violation and a summary of the measured data is computed to facilitate the re-execution of the detection of guideline violations step.

$ pgmpirun process verif

Local Step 6: Analyze Step - Detection of Guideline Violations

The detection script will print guideline violations that were found for the specified experiment.

$ pgmpirun check