From 61cb1238d8d434641abedd08510ea9b7ac32ad64 Mon Sep 17 00:00:00 2001 From: Simon Pintarelli Date: Mon, 31 Mar 2025 15:51:51 +0200 Subject: [PATCH 1/8] update qe docs --- docs/software/sciapps/quantumespresso.md | 66 +++++++++++++++++++++++- 1 file changed, 64 insertions(+), 2 deletions(-) diff --git a/docs/software/sciapps/quantumespresso.md b/docs/software/sciapps/quantumespresso.md index 3068c682..665562d9 100644 --- a/docs/software/sciapps/quantumespresso.md +++ b/docs/software/sciapps/quantumespresso.md @@ -1,4 +1,66 @@ [](){#ref-uenv-quantumespresso} # Quantum ESPRESSO -!!! todo - complete docs + +Quantum ESPRESSO is an integrated suite of Open-Source computer codes for electronic-structure calculations and materials modeling at the nanoscale. It is based on density-functional theory, plane waves, and pseudopotentials: + +PWscf (Plane-Wave Self-Consistent Field) +FPMD (First Principles Molecular Dynamics) +CP (Car-Parrinello) + +## ALPS (GH200) +### Setup +Download the uenv image for QuantumESPRESSO: + + +### List available images + +```bash +uenv image find quantumespresso +``` + +### Pull the image of interest + +``` +uenv image pull quantumespresso/v7.4:v2 +``` + + +QuantumESPRESSO can be compiled from source using the above uenv. The procedure is described in https://eth-cscs.github.io/alps-uenv/uenv-qe/#building-a-custom-version. + +### How to run + +=== "GH200" +Either run uenv start quantumespresso/v7.4 and then submit the job or set --uenv sbatch option accordingly. The following sbatch script can be used as a template. + +```bash +#SBATCH -N 1 +#SBATCH --ntasks-per-node=4 +#SBATCH --cpus-per-task=71 +#SBATCH --gpus-per-task=1 +#SBATCH -A +#SBATCH --uenv=quantumespresso/v7.4:v2 +#SBATCH --view=default + +export OMP_NUM_THREADS=20 +export MPICH_GPU_SUPPORT_ENABLED=1 +export OMP_PLACES=cores + +srun -u --cpu-bind=socket /user-environment/env/default/bin/pw.x < pw.in +``` + +=== "Eiger" +Either run uenv start quantumespresso/v7.4 and then submit the job or set --uenv sbatch option accordingly. The following sbatch script can be used as a template. + +```bash +#SBATCH -N 1 +#SBATCH --ntasks-per-node=128 +#SBATCH -A +#SBATCH --uenv=quantumespresso/v7.4:v2 +#SBATCH --view=default +#SBATCH --hint=nomultithread + +export OMP_NUM_THREADS=1 + +srun -u /user-environment/env/default/bin/pw.x < pw.in +``` + From 86e7ed25935a70f33751f0c5289e5f75479871c6 Mon Sep 17 00:00:00 2001 From: Simon Pintarelli Date: Mon, 31 Mar 2025 16:00:22 +0200 Subject: [PATCH 2/8] cleanup --- docs/software/sciapps/quantumespresso.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/software/sciapps/quantumespresso.md b/docs/software/sciapps/quantumespresso.md index 665562d9..c0e0e45f 100644 --- a/docs/software/sciapps/quantumespresso.md +++ b/docs/software/sciapps/quantumespresso.md @@ -29,8 +29,9 @@ QuantumESPRESSO can be compiled from source using the above uenv. The procedure ### How to run +The following sbatch script can be used as a template. + === "GH200" -Either run uenv start quantumespresso/v7.4 and then submit the job or set --uenv sbatch option accordingly. The following sbatch script can be used as a template. ```bash #SBATCH -N 1 @@ -49,7 +50,6 @@ srun -u --cpu-bind=socket /user-environment/env/default/bin/pw.x < pw.in ``` === "Eiger" -Either run uenv start quantumespresso/v7.4 and then submit the job or set --uenv sbatch option accordingly. The following sbatch script can be used as a template. ```bash #SBATCH -N 1 From aca13e88f63faf75d2a6acedd43b2ea37fca2eb0 Mon Sep 17 00:00:00 2001 From: Simon Pintarelli Date: Mon, 31 Mar 2025 16:05:02 +0200 Subject: [PATCH 3/8] fix tabbing --- docs/software/sciapps/quantumespresso.md | 50 ++++++++++++------------ 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/docs/software/sciapps/quantumespresso.md b/docs/software/sciapps/quantumespresso.md index c0e0e45f..7bc82d84 100644 --- a/docs/software/sciapps/quantumespresso.md +++ b/docs/software/sciapps/quantumespresso.md @@ -33,34 +33,34 @@ The following sbatch script can be used as a template. === "GH200" -```bash -#SBATCH -N 1 -#SBATCH --ntasks-per-node=4 -#SBATCH --cpus-per-task=71 -#SBATCH --gpus-per-task=1 -#SBATCH -A -#SBATCH --uenv=quantumespresso/v7.4:v2 -#SBATCH --view=default - -export OMP_NUM_THREADS=20 -export MPICH_GPU_SUPPORT_ENABLED=1 -export OMP_PLACES=cores - -srun -u --cpu-bind=socket /user-environment/env/default/bin/pw.x < pw.in -``` + ```bash + #SBATCH -N 1 + #SBATCH --ntasks-per-node=4 + #SBATCH --cpus-per-task=71 + #SBATCH --gpus-per-task=1 + #SBATCH -A + #SBATCH --uenv=quantumespresso/v7.4:v2 + #SBATCH --view=default + + export OMP_NUM_THREADS=20 + export MPICH_GPU_SUPPORT_ENABLED=1 + export OMP_PLACES=cores + + srun -u --cpu-bind=socket /user-environment/env/default/bin/pw.x < pw.in + ``` === "Eiger" -```bash -#SBATCH -N 1 -#SBATCH --ntasks-per-node=128 -#SBATCH -A -#SBATCH --uenv=quantumespresso/v7.4:v2 -#SBATCH --view=default -#SBATCH --hint=nomultithread + ```bash + #SBATCH -N 1 + #SBATCH --ntasks-per-node=128 + #SBATCH -A + #SBATCH --uenv=quantumespresso/v7.4:v2 + #SBATCH --view=default + #SBATCH --hint=nomultithread -export OMP_NUM_THREADS=1 + export OMP_NUM_THREADS=1 -srun -u /user-environment/env/default/bin/pw.x < pw.in -``` + srun -u /user-environment/env/default/bin/pw.x < pw.in + ``` From c6c1a620e5acbd7a554c04f2acfc4318627c9aca Mon Sep 17 00:00:00 2001 From: Simon Pintarelli <1237199+simonpintarelli@users.noreply.github.com> Date: Tue, 1 Apr 2025 22:21:07 +0200 Subject: [PATCH 4/8] Update docs/software/sciapps/quantumespresso.md Co-authored-by: Mikael Simberg --- docs/software/sciapps/quantumespresso.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/software/sciapps/quantumespresso.md b/docs/software/sciapps/quantumespresso.md index 7bc82d84..79cc84ea 100644 --- a/docs/software/sciapps/quantumespresso.md +++ b/docs/software/sciapps/quantumespresso.md @@ -1,7 +1,7 @@ [](){#ref-uenv-quantumespresso} # Quantum ESPRESSO -Quantum ESPRESSO is an integrated suite of Open-Source computer codes for electronic-structure calculations and materials modeling at the nanoscale. It is based on density-functional theory, plane waves, and pseudopotentials: +[Quantum ESPRESSO](https://www.quantum-espresso.org/) is an integrated suite of Open-Source computer codes for electronic-structure calculations and materials modeling at the nanoscale. It is based on density-functional theory, plane waves, and pseudopotentials: PWscf (Plane-Wave Self-Consistent Field) FPMD (First Principles Molecular Dynamics) From 5ba5ac4bec9e4b43a0c8bee016323e60040832f6 Mon Sep 17 00:00:00 2001 From: Simon Pintarelli Date: Tue, 1 Apr 2025 22:38:10 +0200 Subject: [PATCH 5/8] updates --- docs/software/sciapps/quantumespresso.md | 128 +++++++++++++++++++---- 1 file changed, 108 insertions(+), 20 deletions(-) diff --git a/docs/software/sciapps/quantumespresso.md b/docs/software/sciapps/quantumespresso.md index 79cc84ea..e65341e7 100644 --- a/docs/software/sciapps/quantumespresso.md +++ b/docs/software/sciapps/quantumespresso.md @@ -3,31 +3,18 @@ [Quantum ESPRESSO](https://www.quantum-espresso.org/) is an integrated suite of Open-Source computer codes for electronic-structure calculations and materials modeling at the nanoscale. It is based on density-functional theory, plane waves, and pseudopotentials: -PWscf (Plane-Wave Self-Consistent Field) -FPMD (First Principles Molecular Dynamics) -CP (Car-Parrinello) +* `pw.x`: Plane-Wave Self-Consistent Field (PWscf) +* `pw.x` First Principles Molecular Dynamics (FPMD) +* `cp.x` Car-Parrinello (CP) -## ALPS (GH200) -### Setup -Download the uenv image for QuantumESPRESSO: +!!! note "uenvs" -### List available images + [Quantum ESPRESSO] is provided on [ALPS][platforms-on-alps] via [uenv][ref-uenv]. + Please have a look at the [uenv documentation][ref-uenv] for more information about uenvs and how to use them. -```bash -uenv image find quantumespresso -``` - -### Pull the image of interest - -``` -uenv image pull quantumespresso/v7.4:v2 -``` - - -QuantumESPRESSO can be compiled from source using the above uenv. The procedure is described in https://eth-cscs.github.io/alps-uenv/uenv-qe/#building-a-custom-version. -### How to run +## How to run The following sbatch script can be used as a template. @@ -48,6 +35,7 @@ The following sbatch script can be used as a template. srun -u --cpu-bind=socket /user-environment/env/default/bin/pw.x < pw.in ``` + Current observation is that best perfomance is achieved using [one MPI rank per GPU][ref-slurm-gh200-single-rank-per-gpu]. How to run multiple ranks per GPU is describe [here][ref-slurm-gh200-multi-rank-per-gpu]. === "Eiger" @@ -64,3 +52,103 @@ The following sbatch script can be used as a template. srun -u /user-environment/env/default/bin/pw.x < pw.in ``` + +## Building QE from Source + +### Using modules + +=== "GH200" + + ```bash + uenv start --view=modules quantumespresso/v7.4:v2 + module load cmake \ + fftw \ + nvhpc \ + nvpl-lapack \ + nvpl-blas \ + cray-mpich \ + netlib-scalapack \ + libxc + + mkdir build && cd build + FC=mpif90 CXX=mpic++ CC=mpicc cmake .. \ + -DQE_ENABLE_MPI=ON \ + -DQE_ENABLE_OPENMP=ON \ + -DQE_ENABLE_SCALAPACK:BOOL=OFF \ + -DQE_ENABLE_LIBXC=ON \ + -DQE_ENABLE_CUDA=ON \ + -DQE_ENABLE_PROFILE_NVTX=ON \ + -DQE_CLOCK_SECONDS:BOOL=OFF \ + -DQE_ENABLE_MPI_GPU_AWARE:BOOL=OFF \ + -DQE_ENABLE_OPENACC=ON + make -j20 + ``` + +=== "A100" + + ```bash + uenv start --view=modules quantumespresso/v7.3.1:v2 + module load cmake \ + cray-mpich + cuda \ + fftw \ + gcc \ + libxc \ + nvhpc \ + openblas + mkdir build && cd build + FC=mpif90 CXX=mpic++ CC=mpicc cmake .. \ + -DQE_ENABLE_MPI=ON \ + -DQE_ENABLE_OPENMP=ON \ + -DQE_ENABLE_SCALAPACK:BOOL=OFF \ + -DQE_ENABLE_LIBXC=ON \ + -DQE_ENABLE_CUDA=ON \ + -DQE_CLOCK_SECONDS:BOOL=OFF \ + -DQE_ENABLE_MPI_GPU_AWARE:BOOL=OFF \ + -DQE_ENABLE_OPENACC=ON + make -j20 + ``` + +### Using spack + +1. Clone spack using the same version that has been used to build the uenv. +```bash +uenv start quantumespresso/v7.3.1 +# clone the same spack version as has been used to build the uenv +git clone -b $(jq -r .spack.commit /user-environment/meta/configure.json) $(jq -r .spack.repo /user-environment/meta/configure.json) $SCRATCH/spack +``` + +2. Activate spack with the uenv configured as upstream +```bash +# ensure spack is using the uenv as upstream repository (always required) +export SPACK_SYSTEM_CONFIG_PATH=/user-environment/config +# active spack (always required) +. $SCRATCH/spack/share/spack/setup-env.sh +``` + +3. Create an anonymous environment for QE +```bash +spack env create -d $SCRATCH/qe-env +spack -e $SCRATCH/qe-env add quantum-espresso%nvhpc +cuda +spack -e $SCRATCH/qe-env config add packages:all:prefer:cuda_arch=90 +spack -e $SCRATCH/qe-env develop -p /path/to/your/QE-src quantum-espresso@=develop +spack -e $SCRATCH/qe-env concretize -f +``` +Check the output of `spack concretize -f`. All dependencies should have been picked up from spack upstream, marked eiter by a green `[^]` or `[e]`. +Next we create a local filesystem view, this instructs spack to create symlinks for binaries and libraries in a local directory `view`. +```bash +spack -e $SCRATCH/qe-env env view enable view +spack -e $SCRATCH/qe-env install +``` +To recompile QE after editing the source code re-run `spack -e $SCRATCH/qe-env install`. + +4. Run `pw.x` using the filesystem view generated in 3. +```bash +uenv start quantumespresso/v7.3.1 +MPICH_GPU_SUPPORT_ENABLED=1 srun [...] $SCRATCH/qe-env/view/bin/pw.x < pw.in +``` +Note: The `pw.x` is linked to the uenv, it won't work without activating the uenv, also it will only work with the exact same version of the uenv. The physical installation path is in `$SCRATCH/spack`, deleting this directory will leave the anonymous spack environment created in 3. with dangling symlinks. + + + + From beb63fb85a925269a045545c731ed9cfd3b52345 Mon Sep 17 00:00:00 2001 From: Simon Pintarelli Date: Tue, 1 Apr 2025 22:38:37 +0200 Subject: [PATCH 6/8] typo --- docs/software/sciapps/quantumespresso.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/software/sciapps/quantumespresso.md b/docs/software/sciapps/quantumespresso.md index e65341e7..408c3adf 100644 --- a/docs/software/sciapps/quantumespresso.md +++ b/docs/software/sciapps/quantumespresso.md @@ -35,7 +35,7 @@ The following sbatch script can be used as a template. srun -u --cpu-bind=socket /user-environment/env/default/bin/pw.x < pw.in ``` - Current observation is that best perfomance is achieved using [one MPI rank per GPU][ref-slurm-gh200-single-rank-per-gpu]. How to run multiple ranks per GPU is describe [here][ref-slurm-gh200-multi-rank-per-gpu]. + Current observation is that best perfomance is achieved using [one MPI rank per GPU][ref-slurm-gh200-single-rank-per-gpu]. How to run multiple ranks per GPU is described [here][ref-slurm-gh200-multi-rank-per-gpu]. === "Eiger" From 677fad8dd59bad42dccb2543fccc222323b8d8c9 Mon Sep 17 00:00:00 2001 From: Simon Pintarelli Date: Tue, 1 Apr 2025 22:41:52 +0200 Subject: [PATCH 7/8] change version on eiger --- docs/software/sciapps/quantumespresso.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/software/sciapps/quantumespresso.md b/docs/software/sciapps/quantumespresso.md index 408c3adf..bbe173ff 100644 --- a/docs/software/sciapps/quantumespresso.md +++ b/docs/software/sciapps/quantumespresso.md @@ -43,7 +43,7 @@ The following sbatch script can be used as a template. #SBATCH -N 1 #SBATCH --ntasks-per-node=128 #SBATCH -A - #SBATCH --uenv=quantumespresso/v7.4:v2 + #SBATCH --uenv=quantumespresso/v7.3.1 #SBATCH --view=default #SBATCH --hint=nomultithread From 06a20408241c7d6324d3399bf74aa43a6e5e56af Mon Sep 17 00:00:00 2001 From: Simon Pintarelli Date: Tue, 1 Apr 2025 22:43:22 +0200 Subject: [PATCH 8/8] missing link --- docs/software/sciapps/quantumespresso.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/software/sciapps/quantumespresso.md b/docs/software/sciapps/quantumespresso.md index bbe173ff..ed93dab6 100644 --- a/docs/software/sciapps/quantumespresso.md +++ b/docs/software/sciapps/quantumespresso.md @@ -10,7 +10,7 @@ !!! note "uenvs" - [Quantum ESPRESSO] is provided on [ALPS][platforms-on-alps] via [uenv][ref-uenv]. + [Quantum ESPRESSO](https://www.quantum-espresso.org/) is provided on [ALPS][platforms-on-alps] via [uenv][ref-uenv]. Please have a look at the [uenv documentation][ref-uenv] for more information about uenvs and how to use them.