Skip to content

Building madgraph4gpu and measuring throughput

Jump to bottom
Andrea Valassi edited this page Oct 11, 2021 · 11 revisions

Building madgraph4gpu and measuring throughput

This page describes how to build the madraph4gpu code and measure its throughput. It is meant to address issue #249.

For the moment, this is a single page focusing on the latest implementation for CUDA and vectorized C++, which is only available for the eemumu process (epoch1/cuda/eemumu). Eventually, other implementations including later CUDA/C++ versions (epochX/cudacpp) and alternative Kokkos, Alpaka and Sycl implementations will also be described.

Download the code

In a chosen directory (here /data/valassi) and using your preferred authentication mechanism (here https) download the latest master

  cd /data/valassi
  git clone https://github.com/madgraph5/madgraph4gpu.git
  cd madgraph4gpu
  git checkout master

For convenience, the download directory will be referred to as MADGRAPH4GPU_HOME in the following (but this environment variable is not used anywhere inside the code or Makefiles).

  export MADGRAPH4GPU_HOME=/data/valassi/madgraph4gpu

Set the runtime environment for the build (compilers, ccache etc)

To build and run the code you need

  • O/S installation using the tsc clocksource (baseline is CentOS8), see issue https://github.com/madgraph5/madgraph4gpu/issues/116 for details
  • C++ compiler and runtime libraries (baseline is gcc9): the CXX environment variable must be set
  • optionally, CUDA compiler and runtime libraries (baseline is nvcc 11.4): CUDA_HOME must be set (or nvcc must be in PATH); if CUDA_HOME points to an invalid path, a C++-only build is performed (using C++ random numbers instead of curand)

In addition:

  • recommended: set up python 3.8 or later
  • recommended: set up ccache
  • recommended: set up perf

C++ compiler

The following C++ compilers are supported

Build the code

Go to the appropriate P1_Sigma subdirectory for the chosen epoch and process. The build is done here.

  cd $MADGRAPH4GPU_HOME
  cd epoch1/cuda/ee_mumu/SubProcesses/P1_Sigma_sm_epem_mupmum

The following make variables (which can be set also via environment variables) control how the build is performed

  • AVX=[none|sse4|avx2|512y|512z]
    • AVX=none: disable C++ vectorization
    • AVX=sse4: enable C++ vectorization with SSE4.2 (128 bit registers, i.e. 2 doubles or 4 floats per vector)
    • AVX=avx2: enable C++ vectorization with AVX2 (256 bit registers, i.e. 4 doubles or 8 floats per vector)
    • AVX=512y (default): enable C++ vectorization with AVX512, limited to 256 bit ymm vector instructions (i.e. 4 doubles or 8 floats per vector)
    • AVX=512y: enable C++ vectorization with AVX512, including 512 bit zmm vector instructions (i.e. 8 doubles or 16 floats per vector)
  • FPTYPE=[d|f]
    • FPTYPE=d (default): use double precision floating-point variables (double)
    • FPTYPE=f: use single precision floating-point variables (float)
  • HELINL=[0|1]
    • HELINL=0 (default): do not use aggressive inlining
    • HELINL=1: use aggressive inlining (emulate LTO optimizations)
  • USEBUILDDIR=[0|1]
    • USEBUILDDIR=0 (default): place binaries (.o, .exe etc) in the P1_Sigma directory itself; if you attempt to recompile using different AVX, FPTYPE or HELINL settings, you will get an error
    • USEBUILDDIR=1 (recommended): place binaries (.o, .exe etc) in a subdirectory of P1_Sigma directory specific to the chosen AVX, FPTYPE or HELINL settings; you may perform several builds in parallel for different AVX, FPTYPE or HELINL settings using different build directories

For detailed performance comparisons, USEBUILDDIR=1 is recommended to allow simultaneous builds with different FPTYPE's (see PR https://github.com/madgraph5/madgraph4gpu/pull/213).

The AVX settings refer to Intel CPUs, but the code builds and runs with C++ vectorizations on AMD CPUs too (see PR https://github.com/madgraph5/madgraph4gpu/pull/238).

Aggressive inlining is found to give almost a factor 4 speedup with no vectorization, and almost a factor 2 speedup with the best vectorization (see issue https://github.com/madgraph5/madgraph4gpu/issues/229). This still neds to be better understood. In particular, note that AVX=none:sse4:avx2 throughputs are more or less in ratios 1:2:4 for double builds (as one would naively expect) when inlining is disabled, but not when inlining is enabled.

Clone this wiki locally