Repository navigation
Piz Daint's programming environment
To get your password for the account, please contact Andreas Jocksch on Slack.
If you already have an account on Piz Daint, you can use this throughout the hackathon week.
If you do not have an account, you will be given a temporary account with a
username of the form hckXX (for some number XX) and a password. The account
is valid until 4 October, 2021.
In order to gain access to CSCS systems you need to first access our front-end
machine Ela which is accessible as ela.cscs.ch. Access Ela by means of ssh as
follows:
ssh -Y hckXX@ela.cscs.ch
The password can be changed on Ela using the kpasswd command.
The machine that we will use is called Piz Daint. Piz Daint consists of the main supercomputer and a set of front-end nodes which you would typically access for compilation, batch job submission, file transfers and use of performance tools. The front end nodes are accessible as daint.cscs.ch and you can access these as:
ssh -Y daint
Having logged in to Daint, you will have a default basic environment and
directories in two file systems which you can access as $HOME and $SCRATCH.
$HOME is a GPFS filesystem and your quota is set at 10G.
$SCRATCH is the high-performance filesystem (Lustre) on Daint. There is no
size quota there, but you are allowed up to 1M i-nodes (file and directory
entries). The storage there is volatile and files will be automatically deleted
if not accessed within a month. It is advised to run your experiments and tests
from $SCRATCH. More information for the filesystems can be found
here.
CSCS uses the module command to change your programming environment. If you
issue the command module list you will see your currently loaded modules. If
you issue the command module avail you see all of the available modules that
you can load. If you want to load a module then issue the command module load <modulename>.
CSCS provides a specially built HPC software stack optimized for Daint. To make available this software stack, you should first do the following:
module load daint-gpu
If your code has external dependencies, please check if there is a module that provides it (either from Cray or from CSCS).
For a simple description of what a module provides use module help <modulename>.
In order to compile codes you will need to select a programming environment for a specific compiler. Available compilers on Daint are the Cray compiler, Intel, GNU and PGI. These are loaded using the module names as shown in the following table.
- Cray:
PrgEnv-cray - PGI:
PrgEnv-pgi - Intel:
PrgEnv-intel - GNU:
PrgEnv-gnu - NVIDIA:
PrgEnv-nvidia
For OpenACC programming we suggest using the PGI compiler. The Cray CCE supports only up to OpenACC 2.0 and for Fortran only. The GCC compiler that is provided does not have support for OpenACC. If you want to use OpenMP 4.5 for accelerators, you should consider using the Cray compiler.
To work with the PGI compiler, one would first have to change the default environment:
module switch PrgEnv-cray PrgEnv-pgi
Finally, code will only be compiled for GPUs if the following module is loaded:
module load craype-accel-nvidia60
The compiler programming environments on the Cray XC40/50 provides convenient wrappers that you can use for compilation, and these wrappers ensure that any libraries and header files that you have loaded through the module command are included or linked automatically.
The wrapper command for Fortran codes is ftn. The wrapper command for C++
codes is CC. The wrapper command for C codes is cc.
You just need to use these wrappers and they will take care of adding the
include paths and linking the libraries. You may need to load additional
modules, and then the wrappers will again take care of adding the correct
paths.
NOTE: If you are compiling an MPI code, the use of wrappers is mandatory; there is no
mpiccand the likes.
You will just need to compile an executable from a single source file as in this example for a Fortran code:
ftn -O2 mpiexercise1.f90 -o myexe1
As mentioned, for GPU compilation and execution you need to the load the following module:
module load craype-accel-nvidia60
The PrgEnv-nvidia provides the NVIDIA HPC SDK, whereas the PrgEnv-pgi
provides just the former PGI compiler. To use the NVIDIA HPC SDK you need to
load a later than the default Cray Development Toolkit (cdt/21.05):
module switch PrgEnv-cray PrgEnv-nvidia
module load cdt/21.05
export CPATH=`echo $CPATH |sed "s@$CRAY_NVIDIA_PREFIX/compilers/include@@"`
export CPATH=$CRAY_NVIDIA_PREFIX/cuda/11.2/targets/x86_64-linux/include:$CPATH
export CPATH=`echo $CPATH |sed "s@$CRAY_NVIDIA_PREFIX/compilers/include@@"`
export CPATH=$CRAY_NVIDIA_PREFIX/cuda/11.2/targets/x86_64-linux/include:$CPATH
The last 2 lines are amending the CPATH variable. This is mandatory for building c/c++ codes with PrgEnv-nvidia, if not the compilation will fail with:
"/usr/include/c++/7/bits/atomic_base.h", line 210: error: identifier
"__GCC_ATOMIC_TEST_AND_SET_TRUEVAL" is undefined
The list of loaded modulefiles will be:
Currently Loaded Modulefiles:
1) modules/3.2.11.4(default) 12) gni-headers/5.0.12.0-7.0.2.1_2.34__g3b1768f.ari
2) craype-network-aries 13) xpmem/2.2.20-7.0.2.1_2.72__g87eb960.ari
3) nvidia/21.3(default) <------- 14) job/2.2.4-7.0.2.1_2.86__g36b56f4.ari
4) craype/2.7.7(default) 15) dvs/2.12_2.2.177-7.0.2.1_13.5__g0b75e43d
5) cray-mpich/7.7.17(default) <------- 16) alps/6.6.59-7.0.2.1_3.85__g872a8d62.ari
6) slurm/20.11.7-7 17) rca/2.2.20-7.0.2.1_2.93__g8e3fb5b.ari
7) craype-haswell 18) atp/3.11.7(default)
8) udreg/2.3.2-7.0.2.1_2.52__g8175d3d.ari 19) perftools-base/21.05.0(default)
9) ugni/6.0.14.0-7.0.2.1_3.77__ge78e5b0.ari 20) PrgEnv-nvidia/6.0.10 <-------
10) pmi/5.0.17(default) 21) cdt-cuda/21.05 <-------
11) dmapp/7.1.1-7.0.2.1_2.98__g38cf134.ari
and the compilers version will be:
| Piz Daint | PrgEnv-nvidia nvidia/21.3 |
|---|---|
| CC --version | nvc++ 21.3-0 |
| cc --version | nvc 21.3-0 |
| ftn --version | nvfortran 21.3-0 |
| nvcc --version | nvcc cuda/11.2.142 |
We also provide nvidia/21.7 on Piz Daint but please note that this version is not officially supported. You need to load additional modulefiles:
module switch PrgEnv-cray PrgEnv-nvidia
module load cdt-cuda/21.05
module use /apps/daint/UES/eurohack/modules/all
module swap nvidia/21.3 nvidia/21.7
# no need to update the CPATH variable with nvidia/21.7
The compilers version will be:
| Piz Daint | PrgEnv-nvidia nvidia/21.7 |
|---|---|
| CC --version | nvc++ 21.7-0 |
| cc --version | nvc 21.7-0 |
| ftn --version | nvfortran 21.7-0 |
| nvcc --version | nvcc cuda/11.4.48 |
We also provide nvidia/21.9 on Piz Daint but please note that this version is not officially supported. You need to load additional modulefiles:
module switch PrgEnv-cray PrgEnv-nvidia
module load cdt-cuda/21.05
module use /apps/daint/UES/eurohack/modules/all
module swap nvidia/21.3 nvidia/21.9
# no need to update the CPATH variable with nvidia/21.9
The compilers version will be:
| Piz Daint | PrgEnv-nvidia nvidia/21.9 |
|---|---|
| CC --version | nvc++ 21.9-0 |
| cc --version | nvc 21.9-0 |
| ftn --version | nvfortran 21.9-0 |
| nvcc --version | nvcc cuda/11.4.100 |
It is possible to use either EasyBuild or Spack to build your code and its dependencies. CSCS provides special installations of EasyBuild and Spack configured for Piz Daint. You can follow the instructions on how to use them in the following locations:
- https://user.cscs.ch/computing/compilation/easybuild/
- https://user.cscs.ch/computing/compilation/spack/
In order to run your code you will need to get an allocation of processors from
the batch system. The mentors will help you generate batch submission scripts.
For basic development an interactive session can be started on the
login nodes of Piz Daint using salloc.
When you have been granted a set of processors, you then use the srun command
to launch jobs on the compute nodes, and the flags that you pass to srun
differ depending upon whether you are running MPI or OpenMP parallel
applications.
When you have finished your practical you should exit the salloc session by
typing exit so that your processors are returned back to the pool.
Before the hackathon, you will have to compete with other jobs running on the system in the normal queue:
salloc -Ahck -Cgpu -N 1 --time=01:00:00
You can also use the debug queue for quicker response times, but your jobs
must be limited to a single node only and have a time limit of 30 minutes.
salloc -Ahck -Cgpu -N 1 --partition=debug
During the hackathon we will have a special reservation eurohack21 on the
machine which is only available to the the hckXX accounts. For development you
should use only few nodes, depending on your needs. The following command
will assign you 4 nodes for up to 1 hour:
salloc -Ahck -Cgpu --reservation=eurohack21 -N 4 --time=01:00:00
The reservation is valid for the duration of the hackathon starting from September 20 @ 9am to September 29 @ 6pm.
If you have a CSCS account, we encourage you to use it to avoid losing potentially valuable work at the end of EuroHack.
You will then have your prompt returned after a message such as the following:
salloc: Granted job allocation XXXX
You are now able to launch your MPI jobs on the compute nodes.
For MPI jobs that are to be launched on the compute nodes you need to use the
-n flag to specify how many processes you wish to launch. For example, to
launch 4 processes of the myexe1 executable you would issue the following
command:
srun -n 4 ./myexe1
srun offers the flexibility to run multi-threaded distributed jobs, e.g., using
both MPI and OpenMP. A common configuration (but by no means always the most
efficient one!) is to run one MPI process per node, and 8 threads on the 12
cores of the Intel Haswell socket. E.g., for 4 processes each with 12 threads,
you would use:
salloc -Ahck -Cgpu --reservation=eurohack21 --nodes=4 --time=01:00:00
and you will then be given back your prompt.
To launch 12 threads of an OpenMP job you need to specify the number of threads
using the OMP_NUM_THREADS variable and then you need to tell srun that you
want 4 processes using the -n flag with 12 threads using the -c flag as
follows:
export OMP_NUM_THREADS=12
srun -n 4 -c 12 ./ompexe1If you have multiple MPI ranks running on the same node and you want them to share the GPU, make sure to export the following variable in your run script:
export CRAY_CUDA_MPS=1Cray's MPICH implementation supports GPU Direct, which eliminates the need to copy the data of the GPU before sending it over. You can pass GPU pointers directly to the MPI calls. To achieve that, you need to set the following variable in your run script:
export MPICH_RDMA_ENABLED_CUDA=1- It is strongly advisable to have a look at our User Portal,
which has more details and up to date information on how to "use" Piz Daint.
You can also interact with us directly via the
#cluster_support_pizdaintchannel in the EuroHack21 Slack.