Performance benchmarks and regression tests for the ExCALIBUR project.
These benchmarks are based on a similar project by StackHPC.
Note: at the moment the ExCALIBUR benchmarks are a work-in-progress.
We require Python version 3.7 or later. Install the excalibur-tests package with pip
by
pip install -e .
The -e/--editable
flag is recommended for two reasons.
- Spack installs packages in a
opt
directory under the spack environment. With-e
the spack environment remains in your local directory andpip
creates symlinks to it. Without-e
spack will install packages inside your python environment. - For development,
the
-e
flag topip
links the installed package to the files in the local directory, instead of copying, to allow making changes to the installed package.
Note that to use -e
with a project configured with a pyproject.toml
you need pip
version 22 or later.
On most systems, it is recommended to install the package in a virtual environment.
For example, using the python3 built-in virtual environment tool venv
,
create an environment called my_environment
with
python3 -m venv ./my_environment
and activate it with
source ./my_environment/bin/activate
The pip install
command will install a compatible version of ReFrame from
PyPi. However, you will have to
manually provide an installation of Spack.
Spack is a package manager specifically designed for HPC facilities. In some HPC facilities there may be already a central Spack installation available. However, the version installed is most likely too old to support all the features used by this package. Therefore we recommend you install the latest version locally, following the instructions below.
Note: if you have already installed spack locally and you want to upgrade to
a newer version, you might first have to clear the cache to avoid conflicts:
spack clean -m
Follow the official instructions to install the latest version of Spack (summarised here for convenience, but not guaranteed to be up-to-date):
- git clone spack:
git clone -c feature.manyFiles=true https://github.com/spack/spack.git
- run spack setup script:
source ./spack/share/spack/setup-env.sh
- check spack is in
$PATH
, for examplespack --version
In order to use Spack in ReFrame, the framework we use to run the benchmarks
(see below), the directory where the spack
program is installed needs to be in
the PATH
environment variable. This is taken care of by the setup-env.sh
script as above, and you can have your shell init script (e.g. .bashrc
)
do that automatically in every session, by adding the following lines to it:
export SPACK_ROOT="/path/to/spack"
if [ -f "${SPACK_ROOT}/share/spack/setup-env.sh" ]; then
source "${SPACK_ROOT}/share/spack/setup-env.sh"
fi
replacing /path/to/spack
with the actual path to your Spack installation.
ReFrame also requires a Spack
Environment. We
provide Spack environments for some of the systems that are part of the
ExCALIBUR and DiRAC projects in
https://github.com/ukri-excalibur/excalibur-tests/tree/main/benchmarks/spack/.
If you want to use a different Spack environment,
set the environment variable EXCALIBUR_SPACK_ENV
to the path of the directory
where the environment is. If this is not set, ReFrame will try to use the
environment for the current system if known, otherwise it will automatically
create a very basic environment (see "Usage on unsupported systems" section below).
ReFrame is a high-level framework for writing regression tests for HPC systems. For our tests we require ReFrame v4.1.3.
We provide a ReFrame configuration file with the settings of some systems that
are part of the ExCALIBUR or DiRAC projects. You can point ReFrame to this file by
setting the
RFM_CONFIG_FILES
environment variable:
export RFM_CONFIG_FILES="${PWD}/benchmarks/reframe_config.py"
If you want to use a different ReFrame configuration file, for example because you use a different system, you can set this environment variable to the path of that file.
Note: in order to use the Spack build system in ReFrame, the spack
executable must be in the PATH
also on the compute nodes of a cluster, if
you want to run your benchmarks on them. This is taken care of by adding it
to your init file (see spack section above).
However, you will also need to set the
RFM_USE_LOGIN_SHELL
environment variable (export RFM_USE_LOGIN_SHELL="true"
) in order to make ReFrame use
!#/bin/bash -l
as shebang line, which would load the user's init script.
Once you have set up Spack and ReFrame, you can execute a benchmark with
reframe -c benchmarks/apps/BENCH_NAME -r --performance-report
where benchmarks/apps/BENCH_NAME
is the directory where the benchmark is. The command
above assumes you have the program reframe
in your PATH. If you have followed the instructions
to install using pip
into the default directory, it should have been automatically added.
If it is not the case, call reframe
with its relative or absolute path.
For example, to run the Sombrero benchmark in the benchmarks/apps/sombrero
directory you can
use
reframe -c benchmarks/apps/sombrero -r --performance-report
For benchmarks that use the Spack build system, the tests define a default Spack specification
to be installed in the environment, but users can change it when invoking ReFrame on the
command line with the
-S
option to set
the spack_spec
variable:
reframe -c benchmarks/apps/sombrero -r --performance-report -S spack_spec='sombrero@2021-08-16%intel'
All the built-in fields of ReFrame regression classes can be set on a per-job basis using the
-S
command-line option. One useful such field is
env_vars
,
which controls the environment variables used in a job.
The syntax to set dictionary items, like for env_vars
, is a comma-separated list of key:value
pairs: -S dict=key_1:value_1,key_2:value_2
.
For example
reframe -c benchmarks/apps/sombrero -r --performance-report -S env_vars=OMP_PLACES:threads
runs the benchmarks/apps/sombrero
benchmark setting the environment variable OMP_PLACES
to threads
.
The provided ReFrame configuration file contains the settings for multiple systems. If you
use it, the automatic detection of the system may fail, as some systems may use clashing
hostnames. To avoid this, you can use the flag --system NAME:PARTITION
to specify the system (and optionally the partition) to use.
Additionally, if submitting jobs to the compute nodes requires additional options, like for
example the resource group you belong to (for example --account=...
for Slurm), you have
to pass the command line flag
--job-option=...
to reframe
(e.g., --job-option='--account=...'
).
The configuration provided in reframe_config.py
lets you run the
benchmarks on pre-configured HPC systems. However you
can use this framework on any system by choosing the "default" system with --system default
, or by using your own ReFrame configuration. You can use the "default" system to run
benchmarks in ReFrame without using a queue manager or an MPI launcher (e.g. on a personal workstation).
If you choose the "default" system and a benchmark using the Spack build system,
a new empty Spack environment will be automatically created in
benchmarks/spack/default
when ReFrame is launched for the first time.
You should populate the environment with the packages already installed on your system
before running Spack to avoid excessively rebuilding system packages. See the
Spack configuration section of CONTRIBUTING.md
for instructions on how
to set up a Spack environment.
In particular, make sure that at least a compiler and an MPI library are added into the environment.
After the Spack environment is set up, tell ReFrame to use it by setting the environment
variable EXCALIBUR_SPACK_ENV
, as described above.
While the aim is to automate as much system-specific configuration as possible, there are some options that have to be provided by the user, such as accounting details, and unfortunately the syntax can vary.
The file SYSTEMS.md
contains information about the use of this framework on specific systems.
Feel free to add new benchmark apps or support new systems that are part of the
ExCALIBUR benchmarking collaboration. Read CONTRIBUTING.md
for more details.