Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 77 additions & 63 deletions docs/build-install/cpe.md
Original file line number Diff line number Diff line change
@@ -1,97 +1,111 @@
# Cray Programming Environment (CPE)

!!! warning
you don't want to use this
The Cray Programming Environment (CPE) is a suite of software: programming environments, compilers, libraries and tools.

!!! mlp
The CPE is not provided on the machine learning platform.
The CPE is provided to users on Alps as containers.

!!! cwp
The CPE is not provided on the climage and weather platform
!!! info "CPE is the cray modules"
The familiar modules that provide the `Prgenv-gnu` and `Prgenv-cray` programming environments, and packages like `cray-python` and `cray-fftw`, that will be familiar from the old Piz Daint system.

!!! warning "CPE is not supported on Alps"
The Cray modules were provided on the old Daint system, and CSCS supported their use and provided software built on top of them.

**Alps is a big change - the CPE modules are not provide as officially supported software.**
They are provided _as is_ to users who still need to use them, however CSCS will not be able to provide detailed support for issues that arise when using them.

The recommended method for building and running software is to use [uenv][ref-uenv] or [containers][ref-container-engine].

## CPE in a container
!!! info
Currently it is mandatory to export the EDF path manually
```

[](){#ref-cpe-versions}
### Available versions

The `PrgEnv-gnu` and `PrgEnv-cray` programming environments are provided as separate containers, instead of having both in one container, named `gnu-$version` and `cray-$version` respectively.
The `version` is the CPE version in the container.

| | `zen2` | `gh200` |
|-----------------|----------|---------|
| `cpe-gnu-24.7` | ❌ | ✅ |
| `cpe-cray-24.7` | ❌ | ✅ |

!!! warning "only available on gh200"
The CPE container is only provided on systems with the "[container engine][ref-container-engine]" container runtime, which is currently the Grace-Hopper systems [Daint][ref-cluster-daint], [Clariden][ref-cluster-clariden] and [Santis][ref-cluster-santis].

### How to use

!!! info "Before you start"
To use the CPE containers as documented below, you need to set the `EDF_PATH` environment variable.
```console
export EDF_PATH=/capstor/scratch/cscs/anfink/shared/cpe/edf:$EDF_PATH
```
Note that the current location of the EDF files is temporary, to work around a file system bug.
The environment variable __will not be required__ once this issue is fixed.

To start a container with the cray programming environment, there are pre-defined EDF toml files. Currently they are stored in `/capstor/scratch/cscs/anfink/shared/cpe/edf`, but this will change in the future to a different location.
Every toml file in this edf path is a programming environment that can be started
```
[daint][anfink@daint-ln001 ~]$ ls -lh /capstor/scratch/cscs/anfink/shared/cpe/edf
total 8.0K
-rw-r--r--+ 1 anfink csstaff 175 Apr 23 11:48 cpe-cray-24.07.toml
-rw-r--r--+ 1 anfink csstaff 174 Apr 23 11:32 cpe-gnu-24.07.toml
```
Following the naming scheme from the output, you can spawn a container with
```
srun --environment=cpe-cray-24.07 --pty bash
To start a session with the CPE (see [available-versions][ref-cpe-versions] above):
```console
$ srun --environment=cpe-cray-24.07 --pty bash
```
Once the container starts up you can directly use the programming environment, because there will be modules loaded by default at startup.
```
[daint][anfink@daint-ln001 ~]$ srun -p debug --environment=cpe-gnu-24.07 --pty bash
[daint][anfink@nid005417 /]$ module list

Currently Loaded Modules:
1) craype-arm-grace 2) craype-network-ofi 3) xpmem/2.9.6 4) gcc-native/13.2 5) craype/2.7.32 6) PrgEnv-gnu/8.5.0 7) cray-mpich/8.1.30 8) cuda/12.6 9) craype-accel-nvidia90
```console
$ srun -p debug --environment=cpe-cray-24.07 --pty bash
$ module list

Currently Loaded Modules:
1) craype-arm-grace 4) cce/18.0.0 7) cray-mpich/8.1.30
2) craype-network-ofi 5) craype/2.7.32 8) cuda/12.6
3) xpmem/2.9.6 6) PrgEnv-cray/8.5.0 9) craype-accel-nvidia90


[daint][anfink@nid005417 /]$ module avail
$ module avail

----------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/mpi/gnu/12.0/ofi/1.0/cray-mpich/8.0 -----------------------------------------------------------------------------------------------------------------------------------------------------
---- /opt/cray/pe/lmod/modulefiles/mpi/crayclang/17.0/ofi/1.0/cray-mpich/8.0 ----
cray-hdf5-parallel/1.14.3.1 cray-parallel-netcdf/1.12.3.13

----------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/comnet/gnu/12.0/ofi/1.0 -----------------------------------------------------------------------------------------------------------------------------------------------------------
---------- /opt/cray/pe/lmod/modulefiles/comnet/crayclang/17.0/ofi/1.0 ----------
cray-mpich-abi/8.1.30 cray-mpich/8.1.30 (L)

---------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/mix_compilers ----------------------------------------------------------------------------------------------------------------------------------------------------------------
gcc-native-mixed/13.2

-------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/compiler/gnu/12.0 --------------------------------------------------------------------------------------------------------------------------------------------------------------
------------- /opt/cray/pe/lmod/modulefiles/compiler/crayclang/17.0 -------------
cray-hdf5/1.14.3.1 cray-libsci/24.07.0

-------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/cpu/arm-grace/1.0 --------------------------------------------------------------------------------------------------------------------------------------------------------------
cray-fftw/3.3.10.8

----------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/craype-targets/default ------------------------------------------------------------------------------------------------------------------------------------------------------------
craype-accel-amd-gfx908 craype-accel-amd-gfx942 craype-accel-nvidia80 craype-hugepages128M craype-hugepages256M craype-hugepages32M craype-hugepages64M craype-network-ofi (L) craype-x86-milan-x craype-x86-spr-hbm
craype-accel-amd-gfx90a craype-accel-host craype-accel-nvidia90 (L) craype-hugepages16M craype-hugepages2G craype-hugepages4M craype-hugepages8M craype-network-ucx craype-x86-milan craype-x86-spr
craype-accel-amd-gfx940 craype-accel-nvidia70 craype-arm-grace (L) craype-hugepages1G craype-hugepages2M craype-hugepages512M craype-network-none craype-x86-genoa craype-x86-rome craype-x86-trento
------------------ /opt/cray/pe/lmod/modulefiles/mix_compilers ------------------
cce-mixed/18.0.0

-------------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/core ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
PrgEnv-gnu/8.5.0 (L) cray-libsci_acc/24.07.0 cray-pmi/6.1.15.19 cray-python/3.11.7 craype/2.7.32 (L) gcc-native/13.2 (L)
---------------- /opt/cray/pe/lmod/modulefiles/cpu/arm-grace/1.0 ----------------
cray-fftw/3.3.10.8

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/modulefiles ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------- /opt/cray/pe/lmod/modulefiles/craype-targets/default --------------
craype-accel-amd-gfx908 craype-hugepages16M craype-network-none
craype-accel-amd-gfx90a craype-hugepages1G craype-network-ofi (L)
craype-accel-amd-gfx940 craype-hugepages256M craype-network-ucx
craype-accel-amd-gfx942 craype-hugepages2G craype-x86-genoa
craype-accel-host craype-hugepages2M craype-x86-milan-x
craype-accel-nvidia70 craype-hugepages32M craype-x86-milan
craype-accel-nvidia80 craype-hugepages4M craype-x86-rome
craype-accel-nvidia90 (L) craype-hugepages512M craype-x86-spr-hbm
craype-arm-grace (L) craype-hugepages64M craype-x86-spr
craype-hugepages128M craype-hugepages8M craype-x86-trento

---------------------- /opt/cray/pe/lmod/modulefiles/core -----------------------
PrgEnv-cray/8.5.0 (L) cray-libsci_acc/24.07.0 cray-python/3.11.7
cce/18.0.0 (L) cray-pmi/6.1.15.19 craype/2.7.32 (L)

----------------------------- /opt/cray/modulefiles -----------------------------
xpmem/2.9.6 (L)

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cscs/modulefiles ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------- /opt/cscs/modulefiles -----------------------------
cuda/12.6 (L)

Where:
L: Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


[daint][anfink@nid005417 /]$ CC --version
g++-13 (SUSE Linux) 13.3.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[daint][anfink@nid005417 /]$
$ CC --version
Cray clang version 18.0.0 (0e4696aa65fa9549bd5e19c216678cc98185b0f7)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/cray/pe/cce/18.0.0/cce-clang/aarch64/share/../bin
```

The recommended way of using CPE in a container is to start the container, and use `$SCRATCH` and `$STORE` to interact with persistent data. Please remember that any data that is written to a directory that is not mounted from the host system will be lost, after the container stops.

By default, the paths `/capstor`, `/iopsstor` are mounted to the same paths inside the container.
!!! note
By default, the paths `/capstor`, `/iopsstor` are mounted to the same paths inside the container.

Additionally `/users` will be mounted at `/users.host`, so you can access data in your home folder, but with a slightly different path. This is on purpose, and you can override this behaviour by writing your own [EDF file][ref-ce-edf-reference], especially using the key `base_environment`, referencing the predefined CPE environment files and override what you would like to change.
5 changes: 0 additions & 5 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,6 @@ The Alps Research infrastructure hosts multiple platforms and clusters targeting

[:octicons-arrow-right-24: Alps Storage](alps/storage.md)

</div>


<div class="grid cards" markdown>

- :fontawesome-solid-layer-group: __Accounts and Projects__

The first step is to get an account and a project
Expand Down
2 changes: 1 addition & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ nav:
- build-install/index.md
- 'uenv': build-install/uenv.md
- 'Containers': build-install/containers.md
- 'Cray Programming Environment (CPE)': build-install/cpe.md
- 'Cray modules (CPE))': build-install/cpe.md
- 'Python with pip': build-install/pip.md
- 'Debugging and Performance Analysis':
- software/devtools/index.md
Expand Down