Skip to content

Latest commit

 

History

History
306 lines (185 loc) · 11.2 KB

perlmutter.rst

File metadata and controls

306 lines (185 loc) · 11.2 KB
Raw

Perlmutter (NERSC)

The Perlmutter cluster is located at NERSC.

Introduction

If you are new to this system, please see the following resources:

Preparation

Use the following commands to download the ImpactX source code:

git clone https://github.com/ECP-WarpX/impactx.git $HOME/src/impactx

On Perlmutter, you can run either on GPU nodes with fast A100 GPUs (recommended) or CPU nodes.

A100 GPUs

We use system software modules, add environment hints and further dependencies via the file $HOME/perlmutter_gpu_impactx.profile. Create it now:

cp $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu_impactx.profile.example $HOME/perlmutter_gpu_impactx.profile

Script Details

perlmutter-nersc/perlmutter_gpu_impactx.profile.example

Edit the 2nd line of this script, which sets the export proj="" variable. Perlmutter GPU projects must end in ..._g. For example, if you are member of the project m3239, then run nano $HOME/perlmutter_gpu_impactx.profile and edit line 2 to read:

export proj="m3239_g"

Exit the nano editor with Ctrl + O (save) and then Ctrl + X (exit).

Important

Now, and as the first step on future logins to Perlmutter, activate these environment settings:

source $HOME/perlmutter_gpu_impactx.profile

Finally, since Perlmutter does not yet provide software modules for some of our dependencies, install them once:

bash $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/install_gpu_dependencies.sh
source ${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/venvs/impactx/bin/activate

Script Details

perlmutter-nersc/install_gpu_dependencies.sh

CPU Nodes

We use system software modules, add environment hints and further dependencies via the file $HOME/perlmutter_cpu_impactx.profile. Create it now:

cp $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu_impactx.profile.example $HOME/perlmutter_cpu_impactx.profile

Script Details

perlmutter-nersc/perlmutter_cpu_impactx.profile.example

Edit the 2nd line of this script, which sets the export proj="" variable. For example, if you are member of the project m3239, then run nano $HOME/perlmutter_cpu_impactx.profile and edit line 2 to read:

export proj="m3239"

Exit the nano editor with Ctrl + O (save) and then Ctrl + X (exit).

Important

Now, and as the first step on future logins to Perlmutter, activate these environment settings:

source $HOME/perlmutter_cpu_impactx.profile

Finally, since Perlmutter does not yet provide software modules for some of our dependencies, install them once:

bash $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/install_cpu_dependencies.sh
source ${CFS}/${proj}/${USER}/sw/perlmutter/cpu/venvs/impactx/bin/activate

Script Details

perlmutter-nersc/install_cpu_dependencies.sh

Compilation

Use the following cmake commands <building-cmake> to compile the application executable:

A100 GPUs

cd $HOME/src/impactx
rm -rf build_pm_gpu

cmake -S . -B build_pm_gpu -DImpactX_COMPUTE=CUDA
cmake --build build_pm_gpu -j 16

The ImpactX application executables are now in $HOME/src/impactx/build_pm_gpu/bin/. Additionally, the following commands will install ImpactX as a Python module:

cd $HOME/src/impactx
rm -rf build_pm_gpu_py

cmake -S . -B build_pm_gpu_py -DImpactX_COMPUTE=CUDA -DImpactX_APP=OFF -DImpactX_PYTHON=ON
cmake --build build_pm_gpu_py -j 16 --target pip_install

CPU Nodes

cd $HOME/src/impactx
rm -rf build_pm_cpu

cmake -S . -B build_pm_cpu -DImpactX_COMPUTE=OMP
cmake --build build_pm_cpu -j 16

The ImpactX application executables are now in $HOME/src/impactx/build_pm_cpu/bin/. Additionally, the following commands will install ImpactX as a Python module:

rm -rf build_pm_cpu_py

cmake -S . -B build_pm_cpu_py -DImpactX_COMPUTE=OMP -DImpactX_APP=OFF -DImpactX_PYTHON=ON
cmake --build build_pm_cpu_py -j 16 --target pip_install

Now, you can submit Perlmutter compute jobs <running-cpp-perlmutter> for ImpactX Python (PICMI) scripts <usage-picmi> (example scripts <usage-examples>). Or, you can use the ImpactX executables to submit Perlmutter jobs (example inputs <usage-examples>). For executables, you can reference their location in your job script <running-cpp-perlmutter> or copy them to a location in $PSCRATCH.

Update ImpactX & Dependencies

If you already installed ImpactX in the past and want to update it, start by getting the latest source code:

cd $HOME/src/impactx

# read the output of this command - does it look ok?
git status

# get the latest ImpactX source code
git fetch
git pull

# read the output of these commands - do they look ok?
git status
git log # press q to exit

And, if needed,

  • update the perlmutter_gpu_impactx.profile or perlmutter_cpu_impactx files <building-perlmutter-preparation>,
  • log out and into the system, activate the now updated environment profile as usual,
  • execute the dependency install scripts <building-perlmutter-preparation>.

As a last step, clean the build directory rm -rf $HOME/src/impactx/build_pm_* and rebuild ImpactX.

Running

A100 (40GB) GPUs

The batch script below can be used to run a ImpactX simulation on multiple nodes (change -N accordingly) on the supercomputer Perlmutter at NERSC. This partition as up to 1536 nodes.

Replace descriptions between chevrons <> by relevant values, for instance <input file> could be plasma_mirror_inputs. Note that we run one MPI rank per GPU.

perlmutter-nersc/perlmutter_gpu.sbatch

To run a simulation, copy the lines above to a file perlmutter_gpu.sbatch and run

sbatch perlmutter_gpu.sbatch

to submit the job.

A100 (80GB) GPUs

Perlmutter has 256 nodes that provide 80 GB HBM per A100 GPU. In the A100 (40GB) batch script, replace -C gpu with -C gpu&hbm80g to use these large-memory GPUs.

CPU Nodes

The Perlmutter CPU partition as up to 3072 nodes, each with 2x AMD EPYC 7763 CPUs.

perlmutter-nersc/perlmutter_cpu.sbatch

Post-Processing

For post-processing, most users use Python via NERSC's Jupyter service (documentation).

As a one-time preparatory setup, log into Perlmutter via SSH and do not source the ImpactX profile script above. Create your own Conda environment and Jupyter kernel for post-processing:

module load python

conda config --set auto_activate_base false

# create conda environment
rm -rf $HOME/.conda/envs/impactx-pm-postproc
conda create --yes -n impactx-pm-postproc -c conda-forge mamba conda-libmamba-solver
conda activate impactx-pm-postproc
conda config --set solver libmamba
mamba install --yes -c conda-forge python ipykernel ipympl matplotlib numpy pandas yt openpmd-viewer openpmd-api h5py fast-histogram dask dask-jobqueue pyarrow

# create Jupyter kernel
rm -rf $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/
python -m ipykernel install --user --name impactx-pm-postproc --display-name ImpactX-PM-PostProcessing
echo -e '#!/bin/bash\nmodule load python\nsource activate impactx-pm-postproc\nexec "$@"' > $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel-helper.sh
chmod a+rx $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel-helper.sh
KERNEL_STR=$(jq '.argv |= ["{resource_dir}/kernel-helper.sh"] + .' $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel.json | jq '.argv[1] = "python"')
echo ${KERNEL_STR} | jq > $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel.json

exit

When opening a Jupyter notebook on https://jupyter.nersc.gov, just select ImpactX-PM-PostProcessing from the list of available kernels on the top right of the notebook.

Additional software can be installed later on, e.g., in a Jupyter cell using !mamba install -y -c conda-forge .... Software that is not available via conda can be installed via !python -m pip install ....