Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
A command line tool that uses MPI to load balance a set of serial commands.
C++
branch: master

This branch is 1 commit behind mlunacek:master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
src
CMakeLists.txt
README

README

Load Balancing Serial Jobs

It is often desirable to use an HPC system to run several serial jobs
(e.g.  parameter scans). The load_balance tool allows you to specify
your jobs in a text file that is then read and executed across the
resource you request. This works on multiple nodes. You can generate
the text file using any scripting language you like. There are two
load balancing schedules: static (default), where all the jobs are
divided as evenly as possible among the processors available, and
dynamic, where the jobs are scheduled by a master on an as needed
basis. The dynamic schedule is best used when there is a large
variance in job run times.


Install
-------------------------------------------------------------------------------

Dependencies
1. MPI
2. Boost (built with mpi, program_options, and serialization )
3. CMake

Source

git clone https://github.com/mlunacek/load_balance.git
cd load_balance
mkdir build
cd build
cmake ../
make

Getting Started
-------------------------------------------------------------------------------

The following example uses the bash echo command as a serial job. You can
exchage this line with your own executable.  For a short help message, type:

load_balance --help

Simple Example The following bash loop with create a file called cmd_lines that
calls the echo command 100 times.

for i in {1..100}; do echo "echo simulator $i" >> cmd_lines; done

The file should look like the following.

echo 'simulator' 1
echo 'simulator' 2
echo 'simulator' 3
...
echo 'simulator' 100

Running Interactively
-------------------------------------------------------------------------------

You can create an interactive session by typing the following:

qsub -I -q janus-debug -l nodes=2:ppn=12

Once the session starts, navigate to the directory containing the
cmd_lines file. Then load the dotkit and run the load_balance
tool. The -f argument specifies the name of the file containing the
command lines. The load_balance tool will distribute the 100 jobs
across 24 processors in a one-at-a-time, load-balanced way. It will
use all the resources you request.  To use a dynamic schedule, set `-s
dynamic`.

use LoadBalance
mpirun load_balance -f cmd_lines

Running
-------------------------------------------------------------------------------

The following PBS script shows how to use the load_balance tool.

#!/bin/bash
#PBS -N example_1
#PBS -q janus-debug
#PBS -l walltime=00:05:00
#PBS -l nodes=2:ppn=12

cd $PBS_O_WORKDIR

. /curc/tools/utils/dkinit
reuse LoadBalance

mpirun load_balance -f cmd_lines

Notes
-------------------------------------------------------------------------------

load_balance has been tested on the carver machine at NERSC with the following dependencies:

- cmake/2.8.2
- openmpi-gnu/1.4.5
- boost/1.51


Something went wrong with that request. Please try again.