-
Notifications
You must be signed in to change notification settings - Fork 41
refactor the cpe docs #101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
34b4a2e
refactor the cpe docs
bcumming b37b447
remove accidental text in software/index
bcumming 62c3574
move EDF note into "how to use"
bcumming 12d9e0b
use output from cpe-cray, not cpe-gnu, in example
bcumming 3610abd
fix another cut and paste error
bcumming 933c8a0
Merge branch 'main' into refactor/cpe
bcumming File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,97 +1,111 @@ | ||
| # Cray Programming Environment (CPE) | ||
|
|
||
| !!! warning | ||
| you don't want to use this | ||
| The Cray Programming Environment (CPE) is a suite of software: programming environments, compilers, libraries and tools. | ||
|
|
||
| !!! mlp | ||
| The CPE is not provided on the machine learning platform. | ||
| The CPE is provided to users on Alps as containers. | ||
|
|
||
| !!! cwp | ||
| The CPE is not provided on the climage and weather platform | ||
| !!! info "CPE is the cray modules" | ||
| The familiar modules that provide the `Prgenv-gnu` and `Prgenv-cray` programming environments, and packages like `cray-python` and `cray-fftw`, that will be familiar from the old Piz Daint system. | ||
|
|
||
| !!! warning "CPE is not supported on Alps" | ||
| The Cray modules were provided on the old Daint system, and CSCS supported their use and provided software built on top of them. | ||
|
|
||
| **Alps is a big change - the CPE modules are not provide as officially supported software.** | ||
| They are provided _as is_ to users who still need to use them, however CSCS will not be able to provide detailed support for issues that arise when using them. | ||
|
|
||
| The recommended method for building and running software is to use [uenv][ref-uenv] or [containers][ref-container-engine]. | ||
|
|
||
| ## CPE in a container | ||
| !!! info | ||
| Currently it is mandatory to export the EDF path manually | ||
| ``` | ||
|
|
||
| [](){#ref-cpe-versions} | ||
| ### Available versions | ||
|
|
||
| The `PrgEnv-gnu` and `PrgEnv-cray` programming environments are provided as separate containers, instead of having both in one container, named `gnu-$version` and `cray-$version` respectively. | ||
| The `version` is the CPE version in the container. | ||
|
|
||
| | | `zen2` | `gh200` | | ||
| |-----------------|----------|---------| | ||
| | `cpe-gnu-24.7` | ❌ | ✅ | | ||
| | `cpe-cray-24.7` | ❌ | ✅ | | ||
|
|
||
| !!! warning "only available on gh200" | ||
| The CPE container is only provided on systems with the "[container engine][ref-container-engine]" container runtime, which is currently the Grace-Hopper systems [Daint][ref-cluster-daint], [Clariden][ref-cluster-clariden] and [Santis][ref-cluster-santis]. | ||
|
|
||
| ### How to use | ||
|
|
||
| !!! info "Before you start" | ||
| To use the CPE containers as documented below, you need to set the `EDF_PATH` environment variable. | ||
| ```console | ||
| export EDF_PATH=/capstor/scratch/cscs/anfink/shared/cpe/edf:$EDF_PATH | ||
| ``` | ||
| Note that the current location of the EDF files is temporary, to work around a file system bug. | ||
| The environment variable __will not be required__ once this issue is fixed. | ||
|
|
||
| To start a container with the cray programming environment, there are pre-defined EDF toml files. Currently they are stored in `/capstor/scratch/cscs/anfink/shared/cpe/edf`, but this will change in the future to a different location. | ||
| Every toml file in this edf path is a programming environment that can be started | ||
| ``` | ||
| [daint][anfink@daint-ln001 ~]$ ls -lh /capstor/scratch/cscs/anfink/shared/cpe/edf | ||
| total 8.0K | ||
| -rw-r--r--+ 1 anfink csstaff 175 Apr 23 11:48 cpe-cray-24.07.toml | ||
| -rw-r--r--+ 1 anfink csstaff 174 Apr 23 11:32 cpe-gnu-24.07.toml | ||
| ``` | ||
| Following the naming scheme from the output, you can spawn a container with | ||
| ``` | ||
| srun --environment=cpe-cray-24.07 --pty bash | ||
| To start a session with the CPE (see [available-versions][ref-cpe-versions] above): | ||
| ```console | ||
| $ srun --environment=cpe-cray-24.07 --pty bash | ||
| ``` | ||
| Once the container starts up you can directly use the programming environment, because there will be modules loaded by default at startup. | ||
| ``` | ||
| [daint][anfink@daint-ln001 ~]$ srun -p debug --environment=cpe-gnu-24.07 --pty bash | ||
| [daint][anfink@nid005417 /]$ module list | ||
|
|
||
| Currently Loaded Modules: | ||
| 1) craype-arm-grace 2) craype-network-ofi 3) xpmem/2.9.6 4) gcc-native/13.2 5) craype/2.7.32 6) PrgEnv-gnu/8.5.0 7) cray-mpich/8.1.30 8) cuda/12.6 9) craype-accel-nvidia90 | ||
| ```console | ||
| $ srun -p debug --environment=cpe-cray-24.07 --pty bash | ||
| $ module list | ||
|
|
||
| Currently Loaded Modules: | ||
| 1) craype-arm-grace 4) cce/18.0.0 7) cray-mpich/8.1.30 | ||
| 2) craype-network-ofi 5) craype/2.7.32 8) cuda/12.6 | ||
| 3) xpmem/2.9.6 6) PrgEnv-cray/8.5.0 9) craype-accel-nvidia90 | ||
|
|
||
|
|
||
| [daint][anfink@nid005417 /]$ module avail | ||
| $ module avail | ||
|
|
||
| ----------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/mpi/gnu/12.0/ofi/1.0/cray-mpich/8.0 ----------------------------------------------------------------------------------------------------------------------------------------------------- | ||
| ---- /opt/cray/pe/lmod/modulefiles/mpi/crayclang/17.0/ofi/1.0/cray-mpich/8.0 ---- | ||
| cray-hdf5-parallel/1.14.3.1 cray-parallel-netcdf/1.12.3.13 | ||
|
|
||
| ----------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/comnet/gnu/12.0/ofi/1.0 ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ||
| ---------- /opt/cray/pe/lmod/modulefiles/comnet/crayclang/17.0/ofi/1.0 ---------- | ||
| cray-mpich-abi/8.1.30 cray-mpich/8.1.30 (L) | ||
|
|
||
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/mix_compilers ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ||
| gcc-native-mixed/13.2 | ||
|
|
||
| -------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/compiler/gnu/12.0 -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ||
| ------------- /opt/cray/pe/lmod/modulefiles/compiler/crayclang/17.0 ------------- | ||
| cray-hdf5/1.14.3.1 cray-libsci/24.07.0 | ||
|
|
||
| -------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/cpu/arm-grace/1.0 -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ||
| cray-fftw/3.3.10.8 | ||
|
|
||
| ----------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/craype-targets/default ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ||
| craype-accel-amd-gfx908 craype-accel-amd-gfx942 craype-accel-nvidia80 craype-hugepages128M craype-hugepages256M craype-hugepages32M craype-hugepages64M craype-network-ofi (L) craype-x86-milan-x craype-x86-spr-hbm | ||
| craype-accel-amd-gfx90a craype-accel-host craype-accel-nvidia90 (L) craype-hugepages16M craype-hugepages2G craype-hugepages4M craype-hugepages8M craype-network-ucx craype-x86-milan craype-x86-spr | ||
| craype-accel-amd-gfx940 craype-accel-nvidia70 craype-arm-grace (L) craype-hugepages1G craype-hugepages2M craype-hugepages512M craype-network-none craype-x86-genoa craype-x86-rome craype-x86-trento | ||
| ------------------ /opt/cray/pe/lmod/modulefiles/mix_compilers ------------------ | ||
| cce-mixed/18.0.0 | ||
|
|
||
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/core --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ||
| PrgEnv-gnu/8.5.0 (L) cray-libsci_acc/24.07.0 cray-pmi/6.1.15.19 cray-python/3.11.7 craype/2.7.32 (L) gcc-native/13.2 (L) | ||
| ---------------- /opt/cray/pe/lmod/modulefiles/cpu/arm-grace/1.0 ---------------- | ||
| cray-fftw/3.3.10.8 | ||
|
|
||
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/modulefiles --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ||
| ------------- /opt/cray/pe/lmod/modulefiles/craype-targets/default -------------- | ||
| craype-accel-amd-gfx908 craype-hugepages16M craype-network-none | ||
| craype-accel-amd-gfx90a craype-hugepages1G craype-network-ofi (L) | ||
| craype-accel-amd-gfx940 craype-hugepages256M craype-network-ucx | ||
| craype-accel-amd-gfx942 craype-hugepages2G craype-x86-genoa | ||
| craype-accel-host craype-hugepages2M craype-x86-milan-x | ||
| craype-accel-nvidia70 craype-hugepages32M craype-x86-milan | ||
| craype-accel-nvidia80 craype-hugepages4M craype-x86-rome | ||
| craype-accel-nvidia90 (L) craype-hugepages512M craype-x86-spr-hbm | ||
| craype-arm-grace (L) craype-hugepages64M craype-x86-spr | ||
| craype-hugepages128M craype-hugepages8M craype-x86-trento | ||
|
|
||
| ---------------------- /opt/cray/pe/lmod/modulefiles/core ----------------------- | ||
| PrgEnv-cray/8.5.0 (L) cray-libsci_acc/24.07.0 cray-python/3.11.7 | ||
| cce/18.0.0 (L) cray-pmi/6.1.15.19 craype/2.7.32 (L) | ||
bcumming marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ----------------------------- /opt/cray/modulefiles ----------------------------- | ||
| xpmem/2.9.6 (L) | ||
|
|
||
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cscs/modulefiles --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ||
| ----------------------------- /opt/cscs/modulefiles ----------------------------- | ||
| cuda/12.6 (L) | ||
|
|
||
| Where: | ||
| L: Module is loaded | ||
|
|
||
| If the avail list is too long consider trying: | ||
|
|
||
| "module --default avail" or "ml -d av" to just list the default modules. | ||
| "module overview" or "ml ov" to display the number of modules for each name. | ||
|
|
||
| Use "module spider" to find all possible modules and extensions. | ||
| Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". | ||
|
|
||
|
|
||
| [daint][anfink@nid005417 /]$ CC --version | ||
| g++-13 (SUSE Linux) 13.3.0 | ||
| Copyright (C) 2023 Free Software Foundation, Inc. | ||
| This is free software; see the source for copying conditions. There is NO | ||
| warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. | ||
|
|
||
| [daint][anfink@nid005417 /]$ | ||
| $ CC --version | ||
| Cray clang version 18.0.0 (0e4696aa65fa9549bd5e19c216678cc98185b0f7) | ||
| Target: aarch64-unknown-linux-gnu | ||
| Thread model: posix | ||
| InstalledDir: /opt/cray/pe/cce/18.0.0/cce-clang/aarch64/share/../bin | ||
| ``` | ||
|
|
||
| The recommended way of using CPE in a container is to start the container, and use `$SCRATCH` and `$STORE` to interact with persistent data. Please remember that any data that is written to a directory that is not mounted from the host system will be lost, after the container stops. | ||
|
|
||
| By default, the paths `/capstor`, `/iopsstor` are mounted to the same paths inside the container. | ||
| !!! note | ||
| By default, the paths `/capstor`, `/iopsstor` are mounted to the same paths inside the container. | ||
|
|
||
| Additionally `/users` will be mounted at `/users.host`, so you can access data in your home folder, but with a slightly different path. This is on purpose, and you can override this behaviour by writing your own [EDF file][ref-ce-edf-reference], especially using the key `base_environment`, referencing the predefined CPE environment files and override what you would like to change. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.