The Perlmutter cluster is located at NERSC.
If you are new to this system, please see the following resources:
- NERSC user guide
- Batch system: Slurm
- Jupyter service (documentation)
- Filesystems:
$HOME
: per-user directory, use only for inputs, source and scripts; backed up (40GB)${CFS}/m3239/
: community file system for users in the projectm3239
(or equivalent); moderate performance (20TB default)$PSCRATCH
: per-user production directory; very fast for parallel jobs; purged every 8 weeks (20TB default)
Use the following commands to download the ImpactX source code:
git clone https://github.com/ECP-WarpX/impactx.git $HOME/src/impactx
On Perlmutter, you can run either on GPU nodes with fast A100 GPUs (recommended) or CPU nodes.
.. tab-set:: .. tab-item:: A100 GPUs We use system software modules, add environment hints and further dependencies via the file ``$HOME/perlmutter_gpu_impactx.profile``. Create it now: .. code-block:: bash cp $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu_impactx.profile.example $HOME/perlmutter_gpu_impactx.profile .. dropdown:: Script Details :color: light :icon: info :animate: fade-in-slide-down .. literalinclude:: perlmutter-nersc/perlmutter_gpu_impactx.profile.example :language: bash Edit the 2nd line of this script, which sets the ``export proj=""`` variable. Perlmutter GPU projects must end in ``..._g``. For example, if you are member of the project ``m3239``, then run ``nano $HOME/perlmutter_gpu_impactx.profile`` and edit line 2 to read: .. code-block:: bash export proj="m3239_g" Exit the ``nano`` editor with ``Ctrl`` + ``O`` (save) and then ``Ctrl`` + ``X`` (exit). .. important:: Now, and as the first step on future logins to Perlmutter, activate these environment settings: .. code-block:: bash source $HOME/perlmutter_gpu_impactx.profile Finally, since Perlmutter does not yet provide software modules for some of our dependencies, install them once: .. code-block:: bash bash $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/install_gpu_dependencies.sh source ${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/venvs/impactx/bin/activate .. dropdown:: Script Details :color: light :icon: info :animate: fade-in-slide-down .. literalinclude:: perlmutter-nersc/install_gpu_dependencies.sh :language: bash .. tab-item:: CPU Nodes We use system software modules, add environment hints and further dependencies via the file ``$HOME/perlmutter_cpu_impactx.profile``. Create it now: .. code-block:: bash cp $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu_impactx.profile.example $HOME/perlmutter_cpu_impactx.profile .. dropdown:: Script Details :color: light :icon: info :animate: fade-in-slide-down .. literalinclude:: perlmutter-nersc/perlmutter_cpu_impactx.profile.example :language: bash Edit the 2nd line of this script, which sets the ``export proj=""`` variable. For example, if you are member of the project ``m3239``, then run ``nano $HOME/perlmutter_cpu_impactx.profile`` and edit line 2 to read: .. code-block:: bash export proj="m3239" Exit the ``nano`` editor with ``Ctrl`` + ``O`` (save) and then ``Ctrl`` + ``X`` (exit). .. important:: Now, and as the first step on future logins to Perlmutter, activate these environment settings: .. code-block:: bash source $HOME/perlmutter_cpu_impactx.profile Finally, since Perlmutter does not yet provide software modules for some of our dependencies, install them once: .. code-block:: bash bash $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/install_cpu_dependencies.sh source ${CFS}/${proj}/${USER}/sw/perlmutter/cpu/venvs/impactx/bin/activate .. dropdown:: Script Details :color: light :icon: info :animate: fade-in-slide-down .. literalinclude:: perlmutter-nersc/install_cpu_dependencies.sh :language: bash
Use the following :ref:`cmake commands <building-cmake>` to compile the application executable:
.. tab-set:: .. tab-item:: A100 GPUs .. code-block:: bash cd $HOME/src/impactx rm -rf build_pm_gpu cmake -S . -B build_pm_gpu -DImpactX_COMPUTE=CUDA -DImpactX_PSATD=ON -DImpactX_QED_TABLE_GEN=ON -DImpactX_DIMS="1;2;RZ;3" cmake --build build_pm_gpu -j 16 The ImpactX application executables are now in ``$HOME/src/impactx/build_pm_gpu/bin/``. Additionally, the following commands will install ImpactX as a Python module: .. code-block:: bash cd $HOME/src/impactx rm -rf build_pm_gpu_py cmake -S . -B build_pm_gpu_py -DImpactX_COMPUTE=CUDA -DImpactX_PSATD=ON -DImpactX_QED_TABLE_GEN=ON -DImpactX_APP=OFF -DImpactX_PYTHON=ON -DImpactX_DIMS="1;2;RZ;3" cmake --build build_pm_gpu_py -j 16 --target pip_install .. tab-item:: CPU Nodes .. code-block:: bash cd $HOME/src/impactx rm -rf build_pm_cpu cmake -S . -B build_pm_cpu -DImpactX_COMPUTE=OMP -DImpactX_PSATD=ON -DImpactX_QED_TABLE_GEN=ON -DImpactX_DIMS="1;2;RZ;3" cmake --build build_pm_cpu -j 16 The ImpactX application executables are now in ``$HOME/src/impactx/build_pm_cpu/bin/``. Additionally, the following commands will install ImpactX as a Python module: .. code-block:: bash rm -rf build_pm_cpu_py cmake -S . -B build_pm_cpu_py -DImpactX_COMPUTE=OMP -DImpactX_PSATD=ON -DImpactX_QED_TABLE_GEN=ON -DImpactX_APP=OFF -DImpactX_PYTHON=ON -DImpactX_DIMS="1;2;RZ;3" cmake --build build_pm_cpu_py -j 16 --target pip_install
Now, you can :ref:`submit Perlmutter compute jobs <running-cpp-perlmutter>` for ImpactX :ref:`Python (PICMI) scripts <usage-picmi>` (:ref:`example scripts <usage-examples>`).
Or, you can use the ImpactX executables to submit Perlmutter jobs (:ref:`example inputs <usage-examples>`).
For executables, you can reference their location in your :ref:`job script <running-cpp-perlmutter>` or copy them to a location in $PSCRATCH
.
If you already installed ImpactX in the past and want to update it, start by getting the latest source code:
cd $HOME/src/impactx
# read the output of this command - does it look ok?
git status
# get the latest ImpactX source code
git fetch
git pull
# read the output of these commands - do they look ok?
git status
git log # press q to exit
And, if needed,
- :ref:`update the perlmutter_gpu_impactx.profile or perlmutter_cpu_impactx files <building-perlmutter-preparation>`,
- log out and into the system, activate the now updated environment profile as usual,
- :ref:`execute the dependency install scripts <building-perlmutter-preparation>`.
As a last step, clean the build directory rm -rf $HOME/src/impactx/build_pm_*
and rebuild ImpactX.
.. tab-set:: .. tab-item:: A100 (40GB) GPUs The batch script below can be used to run a ImpactX simulation on multiple nodes (change ``-N`` accordingly) on the supercomputer Perlmutter at NERSC. This partition as up to `1536 nodes <https://docs.nersc.gov/systems/perlmutter/architecture/>`__. Replace descriptions between chevrons ``<>`` by relevant values, for instance ``<input file>`` could be ``plasma_mirror_inputs``. Note that we run one MPI rank per GPU. .. literalinclude:: perlmutter-nersc/perlmutter_gpu.sbatch :language: bash :caption: You can copy this file from ``$HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu.sbatch``. To run a simulation, copy the lines above to a file ``perlmutter_gpu.sbatch`` and run .. code-block:: bash sbatch perlmutter_gpu.sbatch to submit the job. .. tab-item:: A100 (80GB) GPUs Perlmutter has `256 nodes <https://docs.nersc.gov/systems/perlmutter/architecture/>`__ that provide 80 GB HBM per A100 GPU. In the A100 (40GB) batch script, replace ``-C gpu`` with ``-C gpu&hbm80g`` to use these large-memory GPUs. .. tab-item:: CPU Nodes The Perlmutter CPU partition as up to `3072 nodes <https://docs.nersc.gov/systems/perlmutter/architecture/>`__, each with 2x AMD EPYC 7763 CPUs. .. literalinclude:: perlmutter-nersc/perlmutter_cpu.sbatch :language: bash :caption: You can copy this file from ``$HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu.sbatch``.
For post-processing, most users use Python via NERSC's Jupyter service (documentation).
As a one-time preparatory setup, log into Perlmutter via SSH and do not source the ImpactX profile script above. Create your own Conda environment and Jupyter kernel for post-processing:
module load python
conda config --set auto_activate_base false
# create conda environment
rm -rf $HOME/.conda/envs/impactx-pm-postproc
conda create --yes -n impactx-pm-postproc -c conda-forge mamba conda-libmamba-solver
conda activate impactx-pm-postproc
conda config --set solver libmamba
mamba install --yes -c conda-forge python ipykernel ipympl matplotlib numpy pandas yt openpmd-viewer openpmd-api h5py fast-histogram dask dask-jobqueue pyarrow
# create Jupyter kernel
rm -rf $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/
python -m ipykernel install --user --name impactx-pm-postproc --display-name ImpactX-PM-PostProcessing
echo -e '#!/bin/bash\nmodule load python\nsource activate impactx-pm-postproc\nexec "$@"' > $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel-helper.sh
chmod a+rx $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel-helper.sh
KERNEL_STR=$(jq '.argv |= ["{resource_dir}/kernel-helper.sh"] + .' $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel.json | jq '.argv[1] = "python"')
echo ${KERNEL_STR} | jq > $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel.json
exit
When opening a Jupyter notebook on https://jupyter.nersc.gov, just select ImpactX-PM-PostProcessing
from the list of available kernels on the top right of the notebook.
Additional software can be installed later on, e.g., in a Jupyter cell using !mamba install -y -c conda-forge ...
.
Software that is not available via conda can be installed via !python -m pip install ...
.