Skip to content

Building KHARMA

Jump to bottom
Ben Prather edited this page Nov 17, 2021 · 45 revisions

Prerequisites

First, be sure to check out all of KHARMA's dependencies by running

$ git submodule update --init --recursive

This will grab KHARMA's two main dependencies (as well as some incidental things):

  1. The Parthenon AMR framework from LANL (accompanying documentation). Note KHARMA actually uses a fork of Parthenon, see here.

  2. The Kokkos performance-portability library, originally from SNL. If they are not present here, many common questions and problems can be answered by the Kokkos wiki and tutorials. Parthenon includes a list of the Parthenon-specific wrappers for Kokkos functions in their developer guide.

The dependencies KHARMA needs from the system are ~exactly the same as Parthenon and Kokkos:

  1. A C++14 compliant compiler with OpenMP (tested on several of GCC >= 8, Intel >= 19, nvc++ (formerly PGI) >= 21.9, clang++ and derivatives >= 12)
  2. MPI of some sort
  3. Parallel HDF5 compiled against said MPI

And optionally

  1. CUDA >= 10.2 and a CUDA-supported C++ compiler

OR

  1. Intel oneAPI < 21.4 and a compatible OpenCL compiler for Intel GPUs

All of these should come packaged, come with installers, or be available as modules on larger systems -- except parallel HDF5. Luckily it is quite easy to compile manually (a script is planned to be included with KHARMA soon). Once compiled, the installation location can be specified with the PREFIX_PATH variable when building KHARMA, as described below.

Build system

KHARMA uses cmake for building, and has a small set of bash scripts to handle loading the correct modules and giving the correct arguments to cmake on specific systems. Contributions with additional machine-specific code are welcome, see the examples in machines/.

Generally, on systems with a parallel HDF5 module, one can then run the following to compile for CPU with OpenMP:

./make.sh clean

You may be able to use the following to compile for GPU with CUDA:

./make.sh clean cuda

In many other cases, (when you have manually compiled parallel HDF5) you should be able to get away with:

PREFIX_PATH=/absolute/path/to/phdf5 HOST_ARCH=CPUVER ./make.sh clean

Optimization

The build script make.sh tries to guess an architecture when compiling, defaulting to code which will be reasonably fast on modern machines. However, you can manually specify a device architecture. For example, when compiling for CUDA:

PREFIX_PATH=/absolute/path/to/phdf5 HOST_ARCH=CPUVER DEVICE_ARCH=GPUVER ./make.sh clean cuda

Where CPUVER and GPUVER are the strings used by Kokkos to denote a particular architecture & set of compile flags, e.g. "SKX" for Skylake-X, "HSW" for Haswell, or "AMDAVX" for Ryzen/EPYC processors, and VOLTA70, TURING75, or AMPERE80 for Nvidia GPUs. A list of a few common architecture strings is provided in make.sh, and a full (usually) up-to-date list is kept in the Kokkos documentation. (Note make.sh needs only the portion of the flag after Kokkos_ARCH_).

If you need to specify multiple custom-installed dependencies (e.g. CUDA), you can set PREFIX_PATH="/path/to/one;/path/to/two". PREFIX_PATH does not support spaces in paths, because shell escapes are hard.

Clone this wiki locally