Skip to content

Commit

Permalink
update heisenbug doc (#651)
Browse files Browse the repository at this point in the history
  • Loading branch information
Thomas-Ulrich committed Aug 29, 2022
1 parent f73498c commit eadae61
Showing 1 changed file with 18 additions and 17 deletions.
35 changes: 18 additions & 17 deletions Documentation/gpus.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@ does not use a workload manager but is equipped with multiple GPUs per node.
Supported SeisSol Features
~~~~~~~~~~~~~~~~~~~~~~~~~~

- elastic wave propagation model with kinematic point sources
- elastic wave propagation model
- kinematic point sources
- dynamic source (only for the linear slip weakening friction law)
- off-fault plasticity model


Expand Down Expand Up @@ -106,44 +108,43 @@ It also has 2 GPGPUs (NVIDIA GeForce RTX 3090), that can be used to run the GPU
The RTX 3090 belongs to a consumer kind of graphics cards and thus does not perform well with double precision.
Therefore, it is preferable to compile SeisSol with single precision.


A CUDA-Aware MPI on heisenbug is loaded with:
A module integrating all libraries relevant for SeisSol-GPU is preinstalled at ``/export/dump/ravil/modulefiles``.
It can be discovered at startup after adding the following to ``~/.bashrc``:

.. code-block:: bash
module load gcc/10.2.0
module load mpi.ompi/4.1.0
module_hpcsdk=/export/dump/ravil/modulefiles
export MODULEPATH=$MODULEPATH:${module_hpcsdk}
GemmForge and ChainForge (for GPU) can be installed with:
It is then loaded with:

.. code-block:: bash
pip3 install git+https://github.com/ravil-mobile/gemmforge.git
pip3 install git+https://github.com/ravil-mobile/chainforge.git
module load seissol-env-gcc-11.1.0
All dependencies required for the CPU version of SeisSol should also be installed.
A folder containing all precompiled requirements is available at:
Then clone the dr/sycl branch of SeisSol with:

.. code-block:: bash
/export/dump/ulrich/SeisSol-GPU-build
git clone --branch dr/sycl https://github.com/SeisSol/SeisSol.git seissol-sycl
cd seissol-sycl
git submodule update --init --recursive
To compile the GPU version of SeisSol on heisenbug, use the following cmake options ``-DDEVICE_ARCH=sm_86 -DHOST_ARCH=hsw``.
To compile the GPU version of SeisSol on heisenbug, use the following cmake options ``-DDEVICE_ARCH=sm_86 -DHOST_ARCH=hsw -DDEVICE_BACKEND=cuda -DPRECISION=single``.
Use ``-DCOMMTHREAD=ON`` for multiple GPUs, and ``-DCOMMTHREAD=OFF`` for one GPU.

To run on one GPU (here with order 4, elastic), use simply:

.. code-block:: bash
export OMPI_COMM_WORLD_LOCAL_RANK=0
export OMPI_COMM_WORLD_LOCAL_SIZE=1
export OMPI_MCA_btl=vader,self
SeisSol_Release_ssm_86_cuda_4_elastic parameters.par
./launch ./SeisSol_Release_ssm_86_cuda_4_elastic ./parameters.par
`launch` is a simple bash helper script. It is generated by CMake, in the build directory).

On 2 ranks, use:

.. code-block:: bash
export OMPI_MCA_btl=vader,self
mpirun -n 2 SeisSol_Release_ssm_86_cuda_4_elastic parameters.par

0 comments on commit eadae61

Please sign in to comment.