enhance archdetect to support detection of NVIDIA GPUs + using that in EESSI init script by boegel · Pull Request #767 · EESSI/software-layer

boegel · 2024-09-30T16:31:02Z

Example output:

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen3
archdetect found accelerator: accel/nvidia/cc80
Using x86_64/amd/zen3 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all as the site extension directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all as additional directory (for accelerators) to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

{EESSI 2023.06} $ echo $MODULEPATH
/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all:/cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all

{EESSI 2023.06} $ ls -ld $(echo $MODULEPATH | cut -f1 -d:)
drwxr-xr-x 10 cvmfs cvmfs 4096 Sep 25 13:21 /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all

{EESSI 2023.06} $ module avail LAMMPS

---------- /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all ----------
   LAMMPS/2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1 (g)

---------- /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all ----------
   LAMMPS/2Aug2023_update2-foss-2023a-kokkos    LAMMPS/29Aug2024-foss-2023b-kokkos (D)

…n EESSI init script

eessi-bot · 2024-09-30T16:31:06Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

eessi-build-deploy-bot-deucalion · 2024-09-30T16:31:07Z

Instance boegel-bot-deucalion is configured to build for:

architectures: aarch64/a64fx
repositories: eessi.io-2023.06-software

eessi-bot · 2024-09-30T16:31:07Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software, eessi.io-2023.06-compat

ocaisa

I think we can give people the override capabilities to allow mix and match CPU/GPU stacks (in terms of the CPU architectures)

…GPU_SOFTWARE_SUBDIR_OVERRIDE Co-authored-by: ocaisa <alan.ocais@cecam.org>

…ailed to run + take that into account in EESSI init script + allow overriding software subdirectory for accel/* via $EESSI_ACCEL_SOFTWARE_SUBDIR_OVERRIDE

…t, must be 'accel/nvidia/cc[0-9][0-9]'

…with 'No devices were found' if no GPUs are available in Slurm job

boegel · 2024-10-03T18:18:41Z

@ocaisa Don't merge this just yet (although it's ready for re-review + testing).

We should go all the way here, and also set up some CI for this, by using fake nvidia-smi scripts that produce known output & exit code + checking the result of both archdetect and init/bash...

boegel · 2024-10-03T19:45:17Z

Tested extensively:

CPU-only system (zen2), no `nvidia-smi`

Details

$ init/eessi_archdetect.sh accelpath
$ echo $?
2

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 20:00:33 [DEBUG] accelpath: Override variable set as ''
2024-10-03 20:00:33 [DEBUG] accelpath: nvidia-smi command not found
$ echo $?
2

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen2
archdetect could not detect any accelerators
Using x86_64/amd/zen2 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2/modules/all as the site extension directory to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

A100 GPU system (zen3), 1 GPU available

Without overrides:

Details

$ init/eessi_archdetect.sh accelpath
accel/nvidia/cc80
$ echo $?
0

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 21:28:05 [DEBUG] accelpath: Override variable set as ''
2024-10-03 21:28:05 [DEBUG] accelpath: nvidia-smi command found @ /usr/bin/nvidia-smi
2024-10-03 21:28:05 [DEBUG] accelpath: CUDA compute capability '80' derived from nvidia-smi output 'NVIDIA A100-SXM4-80GB, 1, 545.23.08, 8.0'
2024-10-03 21:28:05 [DEBUG] accelpath: result: accel/nvidia/cc80
accel/nvidia/cc80
$ echo $?
0

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen3
archdetect found supported accelerator for CPU target x86_64/amd/zen3: accel/nvidia/cc80
Using x86_64/amd/zen3 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all as the site extension directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all as additional directory (for accelerators) to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

With $EESSI_ACCELERATOR_TARGET_OVERRIDE set (unsupported GPU target):

Details

$ export EESSI_ACCELERATOR_TARGET_OVERRIDE=accel/nvidia/cc12

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 21:41:17 [DEBUG] accelpath: Override variable set as 'accel/nvidia/cc12'
accel/nvidia/cc12
$ echo $?
0

$ init/eessi_archdetect.sh accelpath
accel/nvidia/cc12
$ echo $?
0

$ export EESSI_ACCELERATOR_TARGET_OVERRIDE=accel/nvidia/cc12
$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen3
No matching path found in x86_64/amd/zen3 for accelerator detected by archdetect (accel/nvidia/cc12)
Using x86_64/amd/zen3 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all as the site extension directory to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

With $EESSI_ACCELERATOR_TARGET_OVERRIDE set (incorrectly):

Details

$ export EESSI_ACCELERATOR_TARGET_OVERRIDE=accel/nvidia/cc123

$ init/eessi_archdetect.sh accelpath
2024-10-03 21:43:26 [ERROR] Value of $EESSI_ACCELERATOR_TARGET_OVERRIDE should match 'accel/nvidia/cc[0-9[0-9]', but it does not: 'accel/nvidia/cc123'
$ echo $?
1

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 21:43:43 [DEBUG] accelpath: Override variable set as 'accel/nvidia/cc123'
2024-10-03 21:43:43 [ERROR] Value of $EESSI_ACCELERATOR_TARGET_OVERRIDE should match 'accel/nvidia/cc[0-9[0-9]', but it does not: 'accel/nvidia/cc123'
$ echo $?
1

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen3
2024-10-03 21:44:15 [ERROR] Value of $EESSI_ACCELERATOR_TARGET_OVERRIDE should match 'accel/nvidia/cc[0-9[0-9]', but it does not: 'accel/nvidia/cc123'
archdetect could not detect any accelerators
Using x86_64/amd/zen3 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all as the site extension directory to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

With $EESSI_ACCEL_SOFTWARE_SUBDIR_OVERRIDE set:

Details

export EESSI_ACCEL_SOFTWARE_SUBDIR_OVERRIDE=x86_64/amd/zen2
$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen3
archdetect found supported accelerator for CPU target x86_64/amd/zen2: accel/nvidia/cc80
Using x86_64/amd/zen3 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all as the site extension directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all as additional directory (for accelerators) to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

A2 GPU system (Intel Cascade Lake), 1 GPU available

Details

$ init/eessi_archdetect.sh accelpath
accel/nvidia/cc86
$ echo $?
0

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 21:29:20 [DEBUG] accelpath: Override variable set as ''
2024-10-03 21:29:20 [DEBUG] accelpath: nvidia-smi command found @ /usr/bin/nvidia-smi
2024-10-03 21:29:20 [DEBUG] accelpath: CUDA compute capability '86' derived from nvidia-smi output 'NVIDIA A2, 1, 545.23.08, 8.6'
2024-10-03 21:29:20 [DEBUG] accelpath: result: accel/nvidia/cc86
accel/nvidia/cc86
$ echo $?
0

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/intel/skylake_avx512
No matching path found in x86_64/intel/skylake_avx512 for accelerator detected by archdetect (accel/nvidia/cc86)
Using x86_64/intel/skylake_avx512 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all as the site extension directory to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

V100 GPU system (Intel Cascade Lake), `nvidia-smi` available, but no GPU allocated in Slurm job

Details

$ init/eessi_archdetect.sh accelpath
$ echo $?
3

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 21:30:38 [DEBUG] accelpath: Override variable set as ''
2024-10-03 21:30:38 [DEBUG] accelpath: nvidia-smi command found @ /usr/bin/nvidia-smi
2024-10-03 21:30:38 [DEBUG] accelpath: nvidia-smi command failed, see output in /tmp/nvidia_smi_out.wRvfw
$ echo $?
3

$ cat /tmp/nvidia_smi_out.wRvfw
No devices were found

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/intel/skylake_avx512
archdetect could not detect any accelerators
Using x86_64/intel/skylake_avx512 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all as the site extension directory to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

boegel · 2024-10-06T11:53:37Z

@ocaisa I've added an extensive GitHub Actions workflow for verifying the NVIDIA GPU accelerator detection implemented in this PR, see 24f0620.

There's one issue though: implementing these tests revealed that the EESSI init script now "chokes" when both:

the errexit shell option is enabled (set -e), which it is by default in GitHub Actions and non uncommon in interactive shell session or job scripts;
no accelerator was detected by archdetect, which is a very common case (it's all that we supported up until very recently in EESSI);

For me, this is for enough reason to re-consider your (currently implemented) suggestion to let the accelpath function in archdetect produce a non-zero exit code when no accelerator is detected, and revert back to making it produce no output in that case...

Thoughts?

ocaisa · 2024-10-06T12:56:08Z

I would temporarily disable set -e when making the call to archdetect since we know it may return an error

ocaisa · 2024-10-06T12:57:07Z

Or give archdetect an environment variable or option that allows overriding returning an error code

…ct to detect accelerator

boegel · 2024-10-07T12:31:30Z

I would temporarily disable set -e when making the call to archdetect since we know it may return an error

@ocaisa done in 3bf3b12

boegel · 2024-10-09T07:32:39Z

@ocaisa Anything blocking this now?

ocaisa

LGTM

ocaisa · 2024-10-09T09:34:19Z

bot: build repo:eessi.io-2023.06-software arch:zen2

eessi-bot · 2024-10-09T09:34:22Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:zen2 from ocaisa
- expanded format: build repository:eessi.io-2023.06-software architecture:zen2
handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:
- submitted job 22289, for details & status see enhance archdetect to support detection of NVIDIA GPUs + using that in EESSI init script #767 (comment)

eessi-bot · 2024-10-09T09:34:22Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:zen2 from ocaisa
- expanded format: build repository:eessi.io-2023.06-software architecture:zen2
handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:
- no jobs were submitted

eessi-bot · 2024-10-09T09:34:26Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.10/pr_767/22289

date	job status	comment
Oct 09 09:34:25 UTC 2024	submitted	job id `22289` awaits release by job manager
Oct 09 09:34:32 UTC 2024	released	job awaits launch by Slurm scheduler
Oct 09 09:35:34 UTC 2024	running	job `22289` is running
Oct 09 09:41:41 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-22289.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1728466503.tar.gz` size: 0 MiB (5818 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/bash` `2023.06/init/eessi_archdetect.sh` `2023.06/init/eessi_environment_variables`
Oct 09 09:41:41 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 10/10 test case Details ✅ job output file `slurm-22289.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Oct 09 10:09:54 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen2-1728466503.tar.gz` to S3 bucket succeeded

ocaisa · 2024-10-09T13:32:42Z

Staging PR merged, good to go!

eessi-bot · 2024-10-09T13:32:49Z

PR merged! Moved ['/project/def-users/SHARED/jobs/2024.10/pr_767/22289'] to /project/def-users/SHARED/trash_bin/EESSI/software-layer/2024.10.09

eessi-bot · 2024-10-09T13:32:49Z

PR merged! Moved [] to /project/def-users/SHARED/trash_bin/EESSI/software-layer/2024.10.09

enhance archdetect to support detection of NVIDIA GPUs + using that i…

67bed25

…n EESSI init script

boegel added 2023.06-software.eessi.io 2023.06 version of software.eessi.io accel:nvidia labels Sep 30, 2024

boegel requested a review from ocaisa September 30, 2024 16:31

ocaisa reviewed Sep 30, 2024

View reviewed changes

Comment thread init/eessi_environment_variables Outdated

ocaisa requested changes Oct 1, 2024

View reviewed changes

Comment thread init/eessi_archdetect.sh

Comment thread init/eessi_archdetect.sh Outdated

Comment thread init/eessi_environment_variables Outdated

Comment thread init/eessi_environment_variables Outdated

introduce $EESSI_GPU_SOFTWARE_PATH which can be overriden via $EESSI_…

6571035

…GPU_SOFTWARE_SUBDIR_OVERRIDE Co-authored-by: ocaisa <alan.ocais@cecam.org>

ocaisa reviewed Oct 3, 2024

View reviewed changes

Comment thread init/eessi_archdetect.sh

boegel added 4 commits October 3, 2024 19:29

use non-zero exit code in archdetect if nvidia-smi was not found or f…

228eaa2

…ailed to run + take that into account in EESSI init script + allow overriding software subdirectory for accel/* via $EESSI_ACCEL_SOFTWARE_SUBDIR_OVERRIDE

sanity check value of $EESSI_ACCELERATOR_TARGET_OVERRIDE, if it is se…

f823375

…t, must be 'accel/nvidia/cc[0-9][0-9]'

take into account that nvidia-smi command may be available, but fail …

b04cc29

…with 'No devices were found' if no GPUs are available in Slurm job

only set $EESSI_MODULEPATH_ACCEL if directory exists

d9bca16

add CI workflow for testing NVIDIA GPU accelerator detection

24f0620

temporarily disable errexit shell option when calling out to archdete…

3bf3b12

…ct to detect accelerator

ocaisa reviewed Oct 7, 2024

View reviewed changes

Comment thread .github/workflows/tests_archdetect_nvidia_gpu.yml

Comment thread init/bash

Comment thread init/eessi_environment_variables

ocaisa reviewed Oct 7, 2024

View reviewed changes

Comment thread init/eessi_environment_variables

ocaisa approved these changes Oct 9, 2024

View reviewed changes

ocaisa added the bot:deploy Ask bot to deploy missing software installations to EESSI label Oct 9, 2024

ocaisa merged commit 37226d0 into EESSI:2023.06-software.eessi.io Oct 9, 2024

boegel deleted the 2023.06-software.eessi.io_archdetect_nvidia branch October 9, 2024 16:29

boegel mentioned this pull request Oct 9, 2024

{2023.06}[foss/2023a] waLBerla 6.1 #779

Merged

Conversation

boegel commented Sep 30, 2024

Uh oh!

eessi-bot Bot commented Sep 30, 2024

Uh oh!

eessi-build-deploy-bot-deucalion Bot commented Sep 30, 2024

Uh oh!

eessi-bot Bot commented Sep 30, 2024

Uh oh!

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

boegel commented Oct 3, 2024

Uh oh!

boegel commented Oct 3, 2024

CPU-only system (zen2), no nvidia-smi

A100 GPU system (zen3), 1 GPU available

A2 GPU system (Intel Cascade Lake), 1 GPU available

V100 GPU system (Intel Cascade Lake), nvidia-smi available, but no GPU allocated in Slurm job

Uh oh!

boegel commented Oct 6, 2024

Uh oh!

ocaisa commented Oct 6, 2024

Uh oh!

ocaisa commented Oct 6, 2024

Uh oh!

boegel commented Oct 7, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

boegel commented Oct 9, 2024

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

ocaisa commented Oct 9, 2024

Uh oh!

eessi-bot Bot commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ocaisa commented Oct 9, 2024

Uh oh!

eessi-bot Bot commented Oct 9, 2024

Uh oh!

eessi-bot Bot commented Oct 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

CPU-only system (zen2), no `nvidia-smi`

V100 GPU system (Intel Cascade Lake), `nvidia-smi` available, but no GPU allocated in Slurm job

eessi-bot Bot commented Oct 9, 2024 •

edited

Loading

eessi-bot Bot commented Oct 9, 2024 •

edited

Loading

eessi-bot Bot commented Oct 9, 2024 •

edited

Loading