The Quartz Intel CPU cluster is located at LLNL.
If you are new to this system, please see the following resources:
- LLNL user account (login required)
- Quartz user guide
- Batch system: Slurm
- Jupyter service (documentation, login required)
- Production directories:
/p/lustre1/$(whoami)
and/p/lustre2/$(whoami)
: personal directory on the parallel filesystem- Note that the
$HOME
directory and the/usr/workspace/$(whoami)
space are NFS mounted and not suitable for production quality data generation.
Use the following commands to download the WarpX source code:
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
We use system software modules, add environment hints and further dependencies via the file $HOME/quartz_warpx.profile
.
Create it now:
cp $HOME/src/warpx/Tools/machines/quartz-llnl/quartz/quartz_warpx.profile.example $HOME/quartz_warpx.profile
.. dropdown:: Script Details :color: light :icon: info :animate: fade-in-slide-down .. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz/quartz_warpx.profile.example :language: bash
Edit the 2nd line of this script, which sets the export proj=""
variable.
For example, if you are member of the project tps
, then run vi $HOME/quartz_warpx.profile
.
Enter the edit mode by typing i
and edit line 2 to read:
export proj="tps"
Exit the vi
editor with Esc
and then type :wq
(write & quit).
Important
Now, and as the first step on future logins to Quartz, activate these environment settings:
source $HOME/quartz_warpx.profile
Finally, since Quartz does not yet provide software modules for some of our dependencies, install them once:
bash $HOME/src/warpx/Tools/machines/quartz-llnl/install_dependencies.sh
source /usr/workspace/${USER}/quartz/venvs/warpx-quartz/bin/activate
.. dropdown:: Script Details :color: light :icon: info :animate: fade-in-slide-down .. literalinclude:: ../../../../Tools/machines/quartz-llnl/install_dependencies.sh :language: bash
Use the following :ref:`cmake commands <building-cmake>` to compile the application executable:
cd $HOME/src/warpx
rm -rf build_quartz
cmake -S . -B build_quartz -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_quartz -j 6
The WarpX application executables are now in $HOME/src/warpx/build_quartz/bin/
.
Additionally, the following commands will install WarpX as a Python module:
rm -rf build_quartz_py
cmake -S . -B build_quartz_py -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_quartz_py -j 6 --target pip_install
Now, you can :ref:`submit Quartz compute jobs <running-cpp-quartz>` for WarpX :ref:`Python (PICMI) scripts <usage-picmi>` (:ref:`example scripts <usage-examples>`).
Or, you can use the WarpX executables to submit Quartz jobs (:ref:`example inputs <usage-examples>`).
For executables, you can reference their location in your :ref:`job script <running-cpp-quartz>` or copy them to a location in $PROJWORK/$proj/
.
If you already installed WarpX in the past and want to update it, start by getting the latest source code:
cd $HOME/src/warpx
# read the output of this command - does it look ok?
git status
# get the latest WarpX source code
git fetch
git pull
# read the output of these commands - do they look ok?
git status
git log # press q to exit
And, if needed,
- :ref:`update the quartz_warpx.profile file <building-quartz-preparation>`,
- log out and into the system, activate the now updated environment profile as usual,
- :ref:`execute the dependency install scripts <building-quartz-preparation>`.
As a last step, clean the build directory rm -rf $HOME/src/warpx/build_quartz
and rebuild WarpX.
The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Quartz at LLNL.
Replace descriptions between chevrons <>
by relevant values, for instance <input file>
could be plasma_mirror_inputs
.
.. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz.sbatch :language: bash :caption: You can copy this file from ``Tools/machines/quartz-llnl/quartz.sbatch``.
To run a simulation, copy the lines above to a file quartz.sbatch
and run
sbatch quartz.sbatch
to submit the job.