Cori (NERSC)

The Cori cluster is located at NERSC.

Introduction

If you are new to this system, please see the following resources:

GPU nodes
Cori user guide
Batch system: Slurm
Jupyter service
Production directories:
- $SCRATCH: per-user production directory, purged every 30 days (20TB)
- /global/cscratch1/sd/m3239: shared production directory for users in the project m3239, purged every 30 days (50TB)
- /global/cfs/cdirs/m3239/: community file system for users in the project m3239 (100TB)

Installation

Use the following commands to download the WarpX source code and switch to the correct branch:

git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx

KNL

We use the following modules and environments on the system ($HOME/knl_warpx.profile).

../../../../Tools/machines/cori-nersc/knl_warpx.profile.example

And install ADIOS2, BLAS++ and LAPACK++:

source $HOME/knl_warpx.profile

# c-blosc (I/O compression)
git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git src/c-blosc
rm -rf src/c-blosc-knl-build
cmake -S src/c-blosc -B src/c-blosc-knl-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/knl/c-blosc-1.12.1-install
cmake --build src/c-blosc-knl-build --target install --parallel 16

# ADIOS2
git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git src/adios2
rm -rf src/adios2-knl-build
cmake -S src/adios2 -B src/adios2-knl-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/knl/adios2-2.7.1-install
cmake --build src/adios2-knl-build --target install --parallel 16

# BLAS++ (for PSATD+RZ)
git clone https://github.com/icl-utk-edu/blaspp.git src/blaspp
rm -rf src/blaspp-knl-build
cmake -S src/blaspp -B src/blaspp-knl-build -Duse_openmp=ON -Duse_cmake_find_blas=ON -DBLAS_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=$HOME/sw/knl/blaspp-master-install
cmake --build src/blaspp-knl-build --target install --parallel 16

# LAPACK++ (for PSATD+RZ)
git clone https://github.com/icl-utk-edu/lapackpp.git src/lapackpp
rm -rf src/lapackpp-knl-build
CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S src/lapackpp -B src/lapackpp-knl-build -Duse_cmake_find_lapack=ON -DBLAS_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DLAPACK_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=$HOME/sw/knl/lapackpp-master-install
cmake --build src/lapackpp-knl-build --target install --parallel 16

For PICMI and Python workflows, also install a virtual environment:

# establish Python dependencies
python3 -m pip install --user --upgrade pip
python3 -m pip install --user virtualenv

python3 -m venv $HOME/sw/knl/venvs/knl_warpx
source $HOME/sw/knl/venvs/knl_warpx/bin/activate

python3 -m pip install --upgrade pip
python3 -m pip install --upgrade wheel
python3 -m pip install --upgrade cython
python3 -m pip install --upgrade numpy
python3 -m pip install --upgrade pandas
python3 -m pip install --upgrade scipy
MPICC="cc -shared" python3 -m pip install -U --no-cache-dir -v mpi4py
python3 -m pip install --upgrade openpmd-api
python3 -m pip install --upgrade matplotlib
python3 -m pip install --upgrade yt
# optional: for libEnsemble
#python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt

Haswell

We use the following modules and environments on the system ($HOME/haswell_warpx.profile).

../../../../Tools/machines/cori-nersc/haswell_warpx.profile.example

And install ADIOS2, BLAS++ and LAPACK++:

source $HOME/haswell_warpx.profile

# c-blosc (I/O compression)
git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git src/c-blosc
rm -rf src/c-blosc-haswell-build
cmake -S src/c-blosc -B src/c-blosc-haswell-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/haswell/c-blosc-1.12.1-install
cmake --build src/c-blosc-haswell-build --target install --parallel 16

# ADIOS2
git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git src/adios2
rm -rf src/adios2-haswell-build
cmake -S src/adios2 -B src/adios2-haswell-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/haswell/adios2-2.7.1-install
cmake --build src/adios2-haswell-build --target install --parallel 16

# BLAS++ (for PSATD+RZ)
git clone https://github.com/icl-utk-edu/blaspp.git src/blaspp
rm -rf src/blaspp-haswell-build
cmake -S src/blaspp -B src/blaspp-haswell-build -Duse_openmp=ON -Duse_cmake_find_blas=ON -DBLAS_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=$HOME/sw/blaspp-master-haswell-install
cmake --build src/blaspp-haswell-build --target install --parallel 16

# LAPACK++ (for PSATD+RZ)
git clone https://github.com/icl-utk-edu/blaspp.git src/lapackpp
rm -rf src/lapackpp-haswell-build
CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S src/lapackpp -B src/lapackpp-haswell-build -Duse_cmake_find_lapack=ON -DBLAS_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DLAPACK_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=$HOME/sw/haswell/lapackpp-master-install
cmake --build src/lapackpp-haswell-build --target install --parallel 16

For PICMI and Python workflows, also install a virtual environment:

# establish Python dependencies
python3 -m pip install --user --upgrade pip
python3 -m pip install --user virtualenv

python3 -m venv $HOME/sw/haswell/venvs/haswell_warpx
source $HOME/sw/haswell/venvs/haswell_warpx/bin/activate

python3 -m pip install --upgrade pip
python3 -m pip install --upgrade wheel
python3 -m pip install --upgrade cython
python3 -m pip install --upgrade numpy
python3 -m pip install --upgrade pandas
python3 -m pip install --upgrade scipy
MPICC="cc -shared" python3 -m pip install -U --no-cache-dir -v mpi4py
python3 -m pip install --upgrade openpmd-api
python3 -m pip install --upgrade matplotlib
python3 -m pip install --upgrade yt
# optional: for libEnsemble
#python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt

GPU (V100)

Cori provides a partition with 18 nodes that include V100 (16 GB) GPUs. We use the following modules and environments on the system ($HOME/gpu_warpx.profile). You can copy this file from Tools/machines/cori-nersc/gpu_warpx.profile.example:

../../../../Tools/machines/cori-nersc/gpu_warpx.profile.example

And install ADIOS2:

source $HOME/gpu_warpx.profile

# c-blosc (I/O compression)
git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git src/c-blosc
rm -rf src/c-blosc-gpu-build
cmake -S src/c-blosc -B src/c-blosc-gpu-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/cori_gpu/c-blosc-1.12.1-install
cmake --build src/c-blosc-gpu-build --target install --parallel 16

git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git src/adios2
rm -rf src/adios2-gpu-build
cmake -S src/adios2 -B src/adios2-gpu-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/cori_gpu/adios2-2.7.1-install
cmake --build src/adios2-gpu-build --target install --parallel 16

For PICMI and Python workflows, also install a virtual environment:

# establish Python dependencies
python3 -m pip install --user --upgrade pip
python3 -m pip install --user virtualenv

python3 -m venv $HOME/sw/cori_gpu/venvs/gpu_warpx
source $HOME/sw/cori_gpu/venvs/gpu_warpx/bin/activate

python3 -m pip install --upgrade pip
python3 -m pip install --upgrade wheel
python3 -m pip install --upgrade cython
python3 -m pip install --upgrade numpy
python3 -m pip install --upgrade pandas
python3 -m pip install --upgrade scipy
python3 -m pip install -U --no-cache-dir -v mpi4py
python3 -m pip install --upgrade openpmd-api
python3 -m pip install --upgrade matplotlib
python3 -m pip install --upgrade yt
# optional: for libEnsemble
#python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt

Building WarpX

We recommend to store the above lines in individual warpx.profile files, as suggested above. If you want to run on either of the three partitions of Cori, open a new terminal, log into Cori and source the environment you want to work with:

# KNL:
source $HOME/knl_warpx.profile

# Haswell:
#source $HOME/haswell_warpx.profile

# GPU:
#source $HOME/gpu_warpx.profile

Warning

Consider that all three Cori partitions are incompatible.

Do not source multiple ...warpx.profile files in the same terminal session. Open a new terminal and log into Cori again, if you want to switch the targeted Cori partition.

If you re-submit an already compiled simulation that you ran on another day or in another session, make sure to source the corresponding ...warpx.profile again after login!

Then, cd into the directory $HOME/src/warpx and use the following commands to compile:

cd $HOME/src/warpx
rm -rf build

#                       append if you target GPUs:    -DWarpX_COMPUTE=CUDA
cmake -S . -B build -DWarpX_DIMS=3
cmake --build build -j 16

The general cmake compile-time options and instructions for Python (PICMI) bindings <building-cmake-python> apply as usual:

# PICMI build
cd $HOME/src/warpx

# install or update dependencies
python3 -m pip install -r requirements.txt

# compile parallel PICMI interfaces with openPMD support and 3D, 2D and RZ
WARPX_MPI=ON BUILD_PARALLEL=16 python3 -m pip install --force-reinstall --no-deps -v .

Testing

To run all tests (here on KNL), do:

change in Regressions/WarpX-tests.ini from mpiexec to srun: MPIcommand = srun -n @nprocs@ @command@

# set test directory to a shared directory available on all nodes
#   note: the tests will create the directory automatically
export WARPX_CI_TMP="$HOME/warpx-regression-tests"

# compile with more cores
export WARPX_CI_NUM_MAKE_JOBS=16

# run all integration tests
#   note: we set MPICC as a build-setting for mpi4py on KNL/Haswell
MPICC="cc -shared" ./run_test.sh

Running

Navigate (i.e. cd) into one of the production directories (e.g. $SCRATCH) before executing the instructions below.

KNL

The batch script below can be used to run a WarpX simulation on 2 KNL nodes on the supercomputer Cori at NERSC. Replace descriptions between chevrons <> by relevant values, for instance <job name> could be laserWakefield.

Do not forget to first source $HOME/knl_warpx.profile if you have not done so already for this terminal session.

For PICMI Python runs, the <path/to/executable> has to read python3 and the <input file> is the path to your PICMI input script.

../../../../Tools/machines/cori-nersc/cori_knl.sbatch

To run a simulation, copy the lines above to a file cori_knl.sbatch and run

sbatch cori_knl.sbatch

to submit the job.

For a 3D simulation with a few (1-4) particles per cell using FDTD Maxwell solver on Cori KNL for a well load-balanced problem (in our case laser wakefield acceleration simulation in a boosted frame in the quasi-linear regime), the following set of parameters provided good performance:

amr.max_grid_size=64 and amr.blocking_factor=64 so that the size of each grid is fixed to 64**3 (we are not using load-balancing here).
8 MPI ranks per KNL node, with OMP_NUM_THREADS=8 (that is 64 threads per KNL node, i.e. 1 thread per physical core, and 4 cores left to the system).
2 grids per MPI, i.e., 16 grids per KNL node.

Haswell

The batch script below can be used to run a WarpX simulation on 1 Haswell node on the supercomputer Cori at NERSC.

Do not forget to first source $HOME/haswell_warpx.profile if you have not done so already for this terminal session.

../../../../Tools/machines/cori-nersc/cori_haswell.sbatch

To run a simulation, copy the lines above to a file cori_haswell.sbatch and run

sbatch cori_haswell.sbatch

to submit the job.

For a 3D simulation with a few (1-4) particles per cell using FDTD Maxwell solver on Cori Haswell for a well load-balanced problem (in our case laser wakefield acceleration simulation in a boosted frame in the quasi-linear regime), the following set of parameters provided good performance:

4 MPI ranks per Haswell node (2 MPI ranks per Intel Xeon E5-2698 v3), with OMP_NUM_THREADS=16 (which uses 2x hyperthreading)

GPU (V100)

Do not forget to first source $HOME/gpu_warpx.profile if you have not done so already for this terminal session.

Due to the limited amount of GPU development nodes, just request a single node with the above defined getNode function. For single-node runs, try to run one grid per GPU.

A multi-node batch script template can be found below:

../../../../Tools/machines/cori-nersc/cori_gpu.sbatch

Post-Processing

For post-processing, most users use Python via NERSC's Jupyter service (Docs).

As a one-time preparatory setup, create your own Conda environment as described in NERSC docs. In this manual, we often use this conda create line over the officially documented one:

conda create -n myenv -c conda-forge python mamba ipykernel ipympl==0.8.6 matplotlib numpy pandas yt openpmd-viewer openpmd-api h5py fast-histogram dask dask-jobqueue pyarrow

We then follow the Customizing Kernels with a Helper Shell Script section to finalize the setup of using this conda-environment as a custom Jupyter kernel.

When opening a Jupyter notebook, just select the name you picked for your custom kernel on the top right of the notebook.

Additional software can be installed later on, e.g., in a Jupyter cell using !mamba install -c conda-forge .... Software that is not available via conda can be installed via !python -m pip install ....

Warning

Jan 6th, 2022 (NERSC-INC0179165 and ipympl #416): Above, we fixated the ipympl version to not take the latest release of Matplotlib Jupyter Widgets. This is an intentional work-around; the ipympl version needs to exactly fit the version pre-installed on the Jupyter base system.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cori.rst

cori.rst

Cori (NERSC)

Introduction

Installation

KNL

Haswell

GPU (V100)

Building WarpX

Testing

Running

KNL

Haswell

GPU (V100)

Post-Processing

Files

cori.rst

Latest commit

History

cori.rst

File metadata and controls

Cori (NERSC)

Introduction

Installation

KNL

Haswell

GPU (V100)

Building WarpX

Testing

Running

KNL

Haswell

GPU (V100)

Post-Processing