Using the madgraph4gpu benchmarking (BMK) containers

The HEP-benchmarks project provides docker and singularity containers that fully encapsulate typical software workloads of the LHC experiments. A test container based on madgraph4gpu, using the standalone tests with cudacpp matrix elements, has recently been added:

• https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/tree/master/mg5amc/madgraph4gpu-2022

The current version of the container is v0.6, it is available from the following locations:

• docker: https://gitlab.cern.ch/hep-benchmarks/hep-workloads/container_registry/14469
• singularity: https://registry.cern.ch/harbor/projects/892/repositories/mg5amc-madgraph4gpu-2022-bmk/artifacts-tab

The following is an example, where the singularity cache dir and tmp dir are also redirected:

  export SINGULARITY_TMPDIR=/scratch/SINGULARITY_TMPDIR
  export SINGULARITY_CACHEDIR=/scratch/SINGULARITY_CACHEDIR
  singularity run -B /scratch/TMP_RESULTS:/results oras://registry.cern.ch/hep-workloads/mg5amc-madgraph4gpu-2022-bmk:v0.6 -h

The containers are configurable. Using -h will print out a list of options. These are still UNDER TEST: please report any issues to AndreaV. Both CPU and GPU tests are available.

• For CPU tests, you may use -c to change the number of simultaneous copies that run on your node as separate (single threaded) processes. You should typically use $(nproc) copies to fill the CPU, and you can also try overcommitting the node.
• For GPU tests, it is recommended that you use -c1 to have a single copy running. The GPU is able to also share amongst different CPU processes, but the overhead reduces the overall throughput.

Scripts and results in madgraph4gpu

A few preliminary results have been obtained using some simple scripts to run CPU tests and then analyse the results and produce some plots:

• run some CPU tests and produce json results: https://github.com/madgraph5/madgraph4gpu/blob/master/tools/benchmarking/driver.sh
• produce some plots from the json results: https://github.com/madgraph5/madgraph4gpu/blob/master/tools/benchmarking/bmkplots.py

Several png plots are available from two nodes

• Haswell with nproc=32 and AVX2: https://github.com/madgraph5/madgraph4gpu/tree/master/tools/benchmarking/BMK-pmpe04
• Silver with nproc=4 and AVX512 (one FMA unit only): https://github.com/madgraph5/madgraph4gpu/tree/master/tools/benchmarking/BMK-itscrd70

Some example results:

• https://github.com/madgraph5/madgraph4gpu/blob/master/tools/benchmarking/BMK-pmpe04/pmpe04-e001-all-sa-cpp-d-inl0.png : maximum throughput on pmpe04 is a factor ~64 higher than 1-core no-SIMD, in double precision - a factor x16 from the 16 physical cores (only reached by using HT), a factor x4 from AVX2 for doubles
• https://github.com/madgraph5/madgraph4gpu/blob/master/tools/benchmarking/BMK-pmpe04/pmpe04-e001-all-sa-cpp-f-inl0.png : maximum throughput on pmpe04 is a factor ~128 higher than 1-core no-SIMD, in single precision - a factor x16 from the 16 physical cores (only reached by using HT), a factor x8 from AVX2 for floats
• https://github.com/madgraph5/madgraph4gpu/blob/master/tools/benchmarking/BMK-itscrd70/itscrd70-e001-all-sa-cpp-d-inl0.png : maximum throughput on itsrcrd70 is a factor ~16 higher than 1-core no-SIMD, in double precision - a factor x4 from the 4 physical cores (NB the plot is wrong, HT is disabled, there are 4 cores), a factor more than x4 from AVX512/ymm for doubles... note that AVX512/zmm is lower (one FMA unit only)
• https://github.com/madgraph5/madgraph4gpu/blob/master/tools/benchmarking/BMK-itscrd70/itscrd70-e001-all-sa-cpp-f-inl0.png : maximum throughput on itsrcrd70 is a factor ~32 higher than 1-core no-SIMD, in double precision - a factor x4 from the 4 physical cores (NB the plot is wrong, HT is disabled, there are 4 cores), a factor more than x8 from AVX512/ymm for floats... note that AVX512/zmm is lower (one FMA unit only)