let bot run EESSI test suite during test phase (part of bot build procedure) #467

casparvl · 2024-02-05T16:30:06Z

A second attempt, based on #366

This is probably a better starting point than #466

Let's see if this runs correctly...

…mas' PR 366

eessi-bot · 2024-02-05T16:30:11Z

Instance eessi-bot-mc-aws is configured to build:

arch x86_64/generic for repo eessi-hpc.org-2023.06-compat
arch x86_64/generic for repo eessi-hpc.org-2023.06-software
arch x86_64/generic for repo eessi.io-2023.06-compat
arch x86_64/generic for repo eessi.io-2023.06-software
arch x86_64/intel/haswell for repo eessi-hpc.org-2023.06-compat
arch x86_64/intel/haswell for repo eessi-hpc.org-2023.06-software
arch x86_64/intel/haswell for repo eessi.io-2023.06-compat
arch x86_64/intel/haswell for repo eessi.io-2023.06-software
arch x86_64/intel/skylake_avx512 for repo eessi-hpc.org-2023.06-compat
arch x86_64/intel/skylake_avx512 for repo eessi-hpc.org-2023.06-software
arch x86_64/intel/skylake_avx512 for repo eessi.io-2023.06-compat
arch x86_64/intel/skylake_avx512 for repo eessi.io-2023.06-software
arch x86_64/amd/zen2 for repo eessi-hpc.org-2023.06-compat
arch x86_64/amd/zen2 for repo eessi-hpc.org-2023.06-software
arch x86_64/amd/zen2 for repo eessi.io-2023.06-compat
arch x86_64/amd/zen2 for repo eessi.io-2023.06-software
arch x86_64/amd/zen3 for repo eessi-hpc.org-2023.06-compat
arch x86_64/amd/zen3 for repo eessi-hpc.org-2023.06-software
arch x86_64/amd/zen3 for repo eessi.io-2023.06-compat
arch x86_64/amd/zen3 for repo eessi.io-2023.06-software
arch aarch64/generic for repo eessi-hpc.org-2023.06-compat
arch aarch64/generic for repo eessi-hpc.org-2023.06-software
arch aarch64/generic for repo eessi.io-2023.06-compat
arch aarch64/generic for repo eessi.io-2023.06-software
arch aarch64/neoverse_n1 for repo eessi-hpc.org-2023.06-compat
arch aarch64/neoverse_n1 for repo eessi-hpc.org-2023.06-software
arch aarch64/neoverse_n1 for repo eessi.io-2023.06-compat
arch aarch64/neoverse_n1 for repo eessi.io-2023.06-software
arch aarch64/neoverse_v1 for repo eessi-hpc.org-2023.06-compat
arch aarch64/neoverse_v1 for repo eessi-hpc.org-2023.06-software
arch aarch64/neoverse_v1 for repo eessi.io-2023.06-compat
arch aarch64/neoverse_v1 for repo eessi.io-2023.06-software

…ger the bot build and test procedures and see if this PR actually works

casparvl · 2024-02-05T16:35:57Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

eessi-bot · 2024-02-05T16:36:00Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 5381, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)

eessi-bot · 2024-02-05T16:36:16Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.02/pr_467/5381

date	job status	comment
Feb 05 16:36:16 UTC 2024	submitted	job id `5381` awaits release by job manager
Feb 05 16:37:18 UTC 2024	released	job awaits launch by Slurm scheduler
Feb 05 16:42:20 UTC 2024	running	job `5381` is running
Feb 05 16:43:21 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-5381.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1707151391.tar.gz` size: 0 MiB (152419 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Feb 05 16:43:21 UTC 2024	test result	🤷 UNKNOWN (click triangle for detailed information) Job test file `_bot_job5381.test` does not exist in job directory or reading it failed.

casparvl · 2024-02-06T09:00:29Z

bot/test.sh: obtaining configuration settings from './cfg/job.cfg'
bot/test.sh: HTTP_PROXY=''
bot/test.sh: HTTPS_PROXY=''
bot/test.sh: LOCAL_TMP='/tmp/$USER/EESSI'
bot/test.sh: BUILD_LOGS_DIR='/project/def-users/SHARED/build-logs'
bot/test.sh: SHARED_FS_PATH='/project/def-users/bot/shared'
bot/test.sh: SINGULARITY_CACHEDIR='/project/def-users/bot/.cache/containers'
bot/test.sh: LOAD_MODULES=''
bot/test.sh: no modules to be loaded
bot/test.sh: EESSI_REPOS_CFG_DIR_OVERRIDE='/project/def-users/SHARED/jobs/2024.02/pr_467/event_a40c8c10-c444-11ee-9238-24dee7b2b685/run_000/linux_x86_64_amd_zen2/eessi.io-2023.06-software/cfg
'
bot/test.sh: EESSI_PILOT_VERSION_OVERRIDE='2023.06'
bot/test.sh: EESSI_CVMFS_REPO_OVERRIDE='software.eessi.io'
bot/test.sh: EESSI_SOFTWARE_SUBDIR_OVERRIDE='x86_64/amd/zen2'
bot/test.sh: EESSI_OS_TYPE='linux'
Executing command to build software:
./eessi_container.sh --verbose --access rw --mode run --container docker://ghcr.io/eessi/build-node:debian11 --repository eessi.io-2023.06-software --save /project/60006/SHARED/jobs/2024.02/pr_467/event_a40c8c10-c444-11ee-9238-24dee7b2b685/run_000/linux_x86_64_amd_zen2/eessi.io-2023.06-software/previous_tmp/test_step --resume /tmp/bot/EESSI/eessi.DKUf0zqdb7
                     -- ./run_tests.sh "" "" 2>&1 | tee -a test.outerr.yeSE
Resuming from previous run using temporary storage at /tmp/bot/EESSI/eessi.DKUf0zqdb7
EESSI_TMPDIR=/tmp/bot/EESSI/eessi.DKUf0zqdb7
HOST_INJECTIONS=/tmp/bot/EESSI/eessi.DKUf0zqdb7/opt-eessi
SINGULARITY_CACHEDIR=/project/def-users/bot/.cache/containers/x86_64
Reusing existing container image /tmp/bot/EESSI/eessi.DKUf0zqdb7/ghcr.io_eessi_build_node_debian11.sif
CONTAINER=/tmp/bot/EESSI/eessi.DKUf0zqdb7/ghcr.io_eessi_build_node_debian11.sif
EESSI_CVMFS_VAR_LIB=/tmp/bot/EESSI/eessi.DKUf0zqdb7/var-lib-cvmfs
EESSI_CVMFS_VAR_RUN=/tmp/bot/EESSI/eessi.DKUf0zqdb7/var-run-cvmfs
SINGULARITY_HOME=/project/60006/SHARED/jobs/2024.02/pr_467/event_a40c8c10-c444-11ee-9238-24dee7b2b685/run_000/linux_x86_64_amd_zen2/eessi.io-2023.06-software:/eessi_bot_job
BIND_PATHS=/tmp/bot/EESSI/eessi.DKUf0zqdb7/var-lib-cvmfs:/var/lib/cvmfs,/tmp/bot/EESSI/eessi.DKUf0zqdb7/var-run-cvmfs:/var/run/cvmfs,/tmp/bot/EESSI/eessi.DKUf0zqdb7/opt-eessi:/opt/eessi,/tmp/bot/EESSI/eessi.DKUf0zqdb7:/tmp
eessi.io/eessi.io.pub --> /etc/cvmfs/keys/eessi.io/eessi.io.pub
eessi.io.conf --> /etc/cvmfs/domain.d/eessi.io.conf
default.local --> /etc/cvmfs/default.local
SINGULARITY_BIND=/project/def-users/SHARED/build-logs,/project/def-users/bot/shared,/tmp/bot/EESSI/eessi.DKUf0zqdb7/var-lib-cvmfs:/var/lib/cvmfs,/tmp/bot/EESSI/eessi.DKUf0zqdb7/var-run-cvmfs:
/var/run/cvmfs,/tmp/bot/EESSI/eessi.DKUf0zqdb7/opt-eessi:/opt/eessi,/tmp/bot/EESSI/eessi.DKUf0zqdb7:/tmp,/tmp/bot/EESSI/eessi.DKUf0zqdb7/repos_cfg/eessi.io/eessi.io.pub:/etc/cvmfs/keys/eessi.
io/eessi.io.pub,/tmp/bot/EESSI/eessi.DKUf0zqdb7/repos_cfg/eessi.io.conf:/etc/cvmfs/domain.d/eessi.io.conf,/tmp/bot/EESSI/eessi.DKUf0zqdb7/repos_cfg/default.local:/etc/cvmfs/default.local
Launching container with command (next line):
singularity  run  --fusemount container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch --fusemount container:cvmfs2 software.eessi.io /cvmfs_ro/software.eessi.io --fusemount containe
r:fuse-overlayfs -o lowerdir=/cvmfs_ro/software.eessi.io -o upperdir=/tmp/overlay-upper -o workdir=/tmp/overlay-work /cvmfs/software.eessi.io /tmp/bot/EESSI/eessi.DKUf0zqdb7/ghcr.io_eessi_bui
ld_node_debian11.sif ./run_tests.sh
INFO:    Environment variable SINGULARITY_BIND is set, but APPTAINER_BIND is preferred
INFO:    Environment variable SINGULARITY_HOME is set, but APPTAINER_HOME is preferred
INFO:    Environment variable SINGULARITY_TMPDIR is set, but APPTAINER_TMPDIR is preferred
FATAL:   permission denied
tar: ./var-lib-cvmfs/shared/cvmfs_io.cvmfs-config.cern.ch: socket ignored
tar: ./var-lib-cvmfs/shared/cvmfs_io.software.eessi.io: socket ignored
tar: ./var-run-cvmfs/cvmfs.cvmfs-config.cern.ch: socket ignored
tar: ./var-run-cvmfs/cvmfs.software.eessi.io: socket ignored
Saved contents of tmp directory '/tmp/bot/EESSI/eessi.DKUf0zqdb7' to tarball '/project/60006/SHARED/jobs/2024.02/pr_467/event_a40c8c10-c444-11ee-9238-24dee7b2b685/run_000/linux_x86_64_amd_zen
2/eessi.io-2023.06-software/previous_tmp/test_step/eessi.io-2023.06-software-1707151395.tgz' (to resume session add '--resume /project/60006/SHARED/jobs/2024.02/pr_467/event_a40c8c10-c444-11ee-9238-24dee7b2b685/run_000/linux_x86_64_amd_zen2/eessi.io-2023.06-software/previous_tmp/test_step/eessi.io-2023.06-software-1707151395.tgz')
bot/test.sh finished
bot/check-test.sh script found in '/project/60006/SHARED/jobs/2024.02/pr_467/event_a40c8c10-c444-11ee-9238-24dee7b2b685/run_000/linux_x86_64_amd_zen2/eessi.io-2023.06-software', so running it!
/var/spool/slurmd/job05381/slurm_script: line 61: bot/check-test.sh: Permission denied
check test step finished

And indeed, the two scripts don't have execute permission:

[casparvl@login1 5381]$ ls -al *test*
-rw-rw-r--. 1 bot def-users  408 Feb  5 16:36 run_tests.sh
-rw-------. 1 bot def-users 3247 Feb  5 16:43 test.outerr.yeSE
-rw-rw-r--. 1 bot def-users 6220 Feb  5 16:36 test_suite.sh

casparvl · 2024-02-06T09:00:40Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

eessi-bot · 2024-02-06T09:00:43Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 5435, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)

eessi-bot · 2024-02-06T09:00:47Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.02/pr_467/5435

date	job status	comment
Feb 06 09:00:46 UTC 2024	submitted	job id `5435` awaits release by job manager
Feb 06 09:01:38 UTC 2024	released	job awaits launch by Slurm scheduler
Feb 06 09:06:41 UTC 2024	running	job `5435` is running
Feb 06 09:07:42 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-5435.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1707210434.tar.gz` size: 0 MiB (152419 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Feb 06 09:07:42 UTC 2024	test result	Test step run succesfully

casparvl · 2024-02-06T09:17:27Z

Sounds good but not quite there yet:

ESC[32m/cvmfs/software.eessi.io available, OK!ESC[0m
ESC[31mERROR: Not running in Gentoo Prefix environment, run '/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/startprefix' first!ESC[0m
Saved contents of tmp directory '/tmp/bot/EESSI/eessi.RPtwJH7y3g' to tarball '/project/60006/SHARED/jobs/2024.02/pr_467/event_342e9840-c4ce-11ee-8955-cd4f7390ce84/run_000/linux_x86_64_amd_zen2/eessi.io-2023.06-software/previous_tmp/test_step/eessi.io-2023.06-software-1707210440.tgz' (to resume session add '--resume /project/60006/SHARED/jobs/2024.02/pr_467/event_342e9840-c4ce-11e
e-8955-cd4f7390ce84/run_000/linux_x86_64_amd_zen2/eessi.io-2023.06-software/previous_tmp/test_step/eessi.io-2023.06-software-1707210440.tgz')
bot/test.sh finished

I don't really want to run the tests in a prefix environment. I'll need to see where this is checked and supress the error.

The other thing I need to check is why this led to a 'Test step run succesfully'. It should have led to a "Failure to execute test step".

casparvl · 2024-02-06T09:30:36Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

eessi-bot · 2024-02-06T09:30:40Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 5439, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)

eessi-bot · 2024-02-06T09:30:44Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.02/pr_467/5439

date	job status	comment
Feb 06 09:30:43 UTC 2024	submitted	job id `5439` awaits release by job manager
Feb 06 09:31:35 UTC 2024	released	job awaits launch by Slurm scheduler
Feb 06 09:32:42 UTC 2024	running	job `5439` is running
Feb 06 09:33:49 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-5439.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1707211968.tar.gz` size: 0 MiB (152420 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Feb 06 09:33:49 UTC 2024	test result	FAILED (test step failed to execute)

casparvl · 2024-02-06T09:39:09Z

Ok, runs as expected:

# slurm-5439.out
...
ESC[32m/cvmfs/software.eessi.io available, OK!ESC[0m
ESC[31mERROR: Not running in Gentoo Prefix environment, run '/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/startprefix' first!ESC[0m

This results in a pattern match for this error:

[casparvl@login1 5439]$ GP_error='ERROR: '
[casparvl@login1 5439]$ grep_out=$(grep -v "^>> searching for " ${job_dir}/${job_out} | grep "${GP_error}");
[casparvl@login1 5439]$ [[ $? -eq 0 ]] && ERROR=1 || ERROR=0
[casparvl@login1 5439]$ echo $ERROR
1

And:

$ cat _bot_job5439.test
[TEST]
comment_description = FAILED (test step failed to execute)
status = FAILURE

As expected. Next, I'll fix the requirement of this thing running in prefix, since that is not needed, so we shouldn't check for it.

casparvl · 2024-02-06T09:43:39Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

eessi-bot · 2024-02-06T09:43:42Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 5440, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)

eessi-bot · 2024-02-06T09:43:46Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.02/pr_467/5440

date	job status	comment
Feb 06 09:43:45 UTC 2024	submitted	job id `5440` awaits release by job manager
Feb 06 09:44:26 UTC 2024	released	job awaits launch by Slurm scheduler
Feb 06 09:45:33 UTC 2024	running	job `5440` is running
Feb 06 09:46:42 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-5440.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1707212738.tar.gz` size: 0 MiB (152417 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Feb 06 09:46:42 UTC 2024	test result	😢 FAILED (EESSI test suite was not run, test step itself failed to execute)

casparvl · 2024-02-06T09:49:28Z

Ok, the detailed sentence is quite long, I'll need to see how I can hide that behind a 'details' thing like is done for the check-build.sh. But first, let's see what the issue is:

ESC[32m/cvmfs/software.eessi.io available, OK!ESC[0m
>> Determining software subdirectory to use for current build/test host...
>> Picking up pre-defined $EESSI_SOFTWARE_SUBDIR_OVERRIDE: x86_64/amd/zen2
ESC[32m>> Using x86_64/amd/zen2 as software subdirectory!ESC[0m
>> Initializing Lmod...
ESC[32m>> Found Lmod 8.7.23ESC[0m
>> Setting up $MODULEPATH...
ESC[32m>> MODULEPATH set up: /modules/allESC[0m
Lmod has detected the following error: The following module(s) are unknown:
"ReFrame/4.3.3"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore_cache load "ReFrame/4.3.3"

Also make sure that all modulefiles written in TCL start with the string
#%Module

Hm, did I actually initialize the EESSI environment here or not (yet)?

casparvl · 2024-02-06T10:18:17Z

Ah,

echo ">> Setting up environment..."

source $TOPDIR/init/minimal_eessi_env

Doesn't set modulepath. That is arranged here:

echo ">> Setting up \$MODULEPATH..."
# make sure no modules are loaded
module --force purge
# ignore current $MODULEPATH entirely
module unuse $MODULEPATH
module use $EASYBUILD_INSTALLPATH/modules/all
if [[ -z ${MODULEPATH} ]]; then
    fatal_error "Failed to set up \$MODULEPATH?!"
else
    echo_green ">> MODULEPATH set up: ${MODULEPATH}"
fi

But, I stripped the sourcing of the configure_easybuild script, since I figured "I don't need to build anything". Two options:

Put in that segment again to configure easybuild. This keeps the script similar to the build step.
source .../init/bash instead of the .../init/minimal_eessi_env

casparvl · 2024-02-06T10:18:53Z

Ok, there's actually a minimal change that I can do: Change EASYBUILD_INSTALLPATH to EESSI_SOFTWARE_PATH. That keeps the script as similar as possible, but doesn't require me to configure EasyBuild.

…odulepath. It should be the same, and we no longer configure EasyBuild, since we don't use it when running the test suite

casparvl · 2024-02-06T10:35:38Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

test_suite.sh

run_tests.sh

bot/check-test.sh

bot/test.sh

reframe_config_bot.py.tmpl

test_suite.sh

bot/test.sh

run_tests.sh

test_suite.sh

…sed for running test suite

…re run with generic software installations when intended

boegel · 2024-03-26T18:05:45Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/generic
bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3
bot: build repo:eessi.io-2023.06-software arch:aarch64/generic
bot: build repo:eessi.io-2023.06-software arch:aarch64/neoverse_v1

eessi-bot · 2024-03-26T18:05:48Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/generic from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/generic
received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
received bot command build repo:eessi.io-2023.06-software arch:aarch64/generic from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/generic
received bot command build repo:eessi.io-2023.06-software arch:aarch64/neoverse_v1 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_v1
handling command build repository:eessi.io-2023.06-software architecture:x86_64/generic resulted in:
- submitted job 8605, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- submitted job 8606, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)
handling command build repository:eessi.io-2023.06-software architecture:aarch64/generic resulted in:
- submitted job 8607, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)
handling command build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_v1 resulted in:
- submitted job 8608, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)

eessi-bot · 2024-03-26T18:05:52Z

New job on instance eessi-bot-mc-aws for architecture x86_64-generic for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_467/8605

date	job status	comment
Mar 26 18:05:51 UTC 2024	submitted	job id `8605` awaits release by job manager
Mar 26 18:06:20 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 26 18:07:34 UTC 2024	running	job `8605` is running
Mar 26 18:22:19 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8605.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-generic-1711476390.tar.gz` size: 0 MiB (206987 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/generic/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/generic/software no software packages in tarball other under 2023.06/software/linux/x86_64/generic `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Mar 26 18:22:19 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8605.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-26T18:05:55Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_467/8606

date	job status	comment
Mar 26 18:05:54 UTC 2024	submitted	job id `8606` awaits release by job manager
Mar 26 18:06:16 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 26 18:07:30 UTC 2024	running	job `8606` is running
Mar 26 18:17:59 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8606.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1711476385.tar.gz` size: 0 MiB (207126 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen3/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen3/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen3 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Mar 26 18:17:59 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8606.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-26T18:05:59Z

New job on instance eessi-bot-mc-aws for architecture aarch64-generic for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_467/8607

date	job status	comment
Mar 26 18:05:58 UTC 2024	submitted	job id `8607` awaits release by job manager
Mar 26 18:06:07 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 26 18:07:23 UTC 2024	running	job `8607` is running
Mar 26 18:15:47 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8607.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-generic-1711476375.tar.gz` size: 0 MiB (205354 bytes) entries: 3 modules under 2023.06/software/linux/aarch64/generic/modules/all no module files in tarball software under 2023.06/software/linux/aarch64/generic/software no software packages in tarball other under 2023.06/software/linux/aarch64/generic `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Mar 26 18:15:47 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8607.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-26T18:06:02Z

New job on instance eessi-bot-mc-aws for architecture aarch64-neoverse_v1 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_467/8608

date	job status	comment
Mar 26 18:06:02 UTC 2024	submitted	job id `8608` awaits release by job manager
Mar 26 18:06:10 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 26 18:07:27 UTC 2024	running	job `8608` is running
Mar 26 18:14:39 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8608.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_v1-1711476378.tar.gz` size: 0 MiB (205924 bytes) entries: 3 modules under 2023.06/software/linux/aarch64/neoverse_v1/modules/all no module files in tarball software under 2023.06/software/linux/aarch64/neoverse_v1/software no software packages in tarball other under 2023.06/software/linux/aarch64/neoverse_v1 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Mar 26 18:14:39 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8608.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

boegel · 2024-03-26T19:52:09Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/intel/haswell
bot: build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512
bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2
bot: build repo:eessi.io-2023.06-software arch:aarch64/neoverse_n1

eessi-bot · 2024-03-26T19:52:14Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/haswell from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/haswell
received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512
received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
received bot command build repo:eessi.io-2023.06-software arch:aarch64/neoverse_n1 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_n1
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/haswell resulted in:
- submitted job 8621, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512 resulted in:
- submitted job 8622, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 8623, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)
handling command build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_n1 resulted in:
- submitted job 8624, for details & status see let bot run EESSI test suite during test phase (part of bot build procedure) #467 (comment)

eessi-bot · 2024-03-26T19:52:18Z

New job on instance eessi-bot-mc-aws for architecture x86_64-intel-haswell for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_467/8621

date	job status	comment
Mar 26 19:52:17 UTC 2024	submitted	job id `8621` awaits release by job manager
Mar 26 19:52:22 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 26 19:57:36 UTC 2024	running	job `8621` is running
Mar 26 20:13:40 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8621.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-haswell-1711483030.tar.gz` size: 0 MiB (207413 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/intel/haswell/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/intel/haswell/software no software packages in tarball other under 2023.06/software/linux/x86_64/intel/haswell `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Mar 26 20:13:40 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8621.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-26T19:52:22Z

New job on instance eessi-bot-mc-aws for architecture x86_64-intel-skylake_avx512 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_467/8622

date	job status	comment
Mar 26 19:52:21 UTC 2024	submitted	job id `8622` awaits release by job manager
Mar 26 19:53:41 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 26 19:58:59 UTC 2024	running	job `8622` is running
Mar 26 20:13:42 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8622.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1711483113.tar.gz` size: 0 MiB (207719 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/intel/skylake_avx512/software no software packages in tarball other under 2023.06/software/linux/x86_64/intel/skylake_avx512 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Mar 26 20:13:42 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8622.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-26T19:52:25Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_467/8623

date	job status	comment
Mar 26 19:52:24 UTC 2024	submitted	job id `8623` awaits release by job manager
Mar 26 19:53:37 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 26 19:58:49 UTC 2024	running	job `8623` is running
Mar 26 20:16:25 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8623.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1711483088.tar.gz` size: 0 MiB (207127 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Mar 26 20:16:25 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8623.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-26T19:52:29Z

New job on instance eessi-bot-mc-aws for architecture aarch64-neoverse_n1 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_467/8624

date	job status	comment
Mar 26 19:52:28 UTC 2024	submitted	job id `8624` awaits release by job manager
Mar 26 19:53:33 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 26 19:58:43 UTC 2024	running	job `8624` is running
Mar 26 20:07:44 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8624.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_n1-1711483089.tar.gz` size: 0 MiB (205929 bytes) entries: 3 modules under 2023.06/software/linux/aarch64/neoverse_n1/modules/all no module files in tarball software under 2023.06/software/linux/aarch64/neoverse_n1/software no software packages in tarball other under 2023.06/software/linux/aarch64/neoverse_n1 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Mar 26 20:07:44 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8624.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

truib and others added 4 commits October 14, 2023 21:09

first version of bot/test.sh script

6cc2e3e

scripts to run tests after a build job has finished

9a32dc6

small adjustments to test script

17cfd02

Second attempt at having the bot run the test suite, now based on Tho…

0f3646e

…mas' PR 366

Caspar van Leeuwen added 2 commits February 5, 2024 17:33

Merge upstream into old feature branch based on pr366

e8c91eb

Make a very simple change to an easystack file so we are able to trig…

34fca15

…ger the bot build and test procedures and see if this PR actually works

Make scripts executable

3faae9d

Check for existence of SLURM output first

081e2b1

Clarify return messages from the bot

9b6fa73

Use EESSI_SOFTWARE_PATH instead of EASYBUILD_INSTALLPATH to set the m…

cb48b36

…odulepath. It should be the same, and we no longer configure EasyBuild, since we don't use it when running the test suite

casparvl commented Mar 26, 2024

View reviewed changes

test_suite.sh Outdated Show resolved Hide resolved

casparvl commented Mar 26, 2024

View reviewed changes

run_tests.sh Outdated Show resolved Hide resolved

boegel mentioned this pull request Mar 26, 2024

clean up duplicate code in various scripts #509

Open

boegel requested changes Mar 26, 2024

View reviewed changes

boegel added 6 commits March 26, 2024 10:32

also dump status to test result file

330db7a

add Caspar as author in bot/test.sh script

f3c6207

use read-only mode to access software installations via container

13072df

fix typo: build -> test

1fa323a

disable CPU auto-detection in template ReFrame configuration script u…

8d05473

…sed for running test suite

pass down --generic to script used for running test suite, so tests a…

c53f578

…re run with generic software installations when intended

boegel changed the title ~~Run test suite (based on pr366)~~ let bot run EESSI test suite during test phase Mar 26, 2024

boegel approved these changes Mar 26, 2024

View reviewed changes

boegel merged commit 73905db into EESSI:2023.06-software.eessi.io Mar 26, 2024
33 checks passed

boegel added the 2023.06-software.eessi.io 2023.06 version of software.eessi.io label Apr 4, 2024

boegel changed the title ~~let bot run EESSI test suite during test phase~~ let bot run EESSI test suite during test phase (part of bot build procedure) Apr 4, 2024

boegel mentioned this pull request Apr 4, 2024

add scripts to run test suite after build job has finished #366

Merged

2 tasks

trz42 mentioned this pull request Apr 5, 2024

Syncing code-base and CVMFS repository with EESSI (April 2024) NorESSI/software-layer#294

Open

51 tasks

casparvl deleted the run_test_suite_based_on_pr366 branch August 15, 2024 15:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

let bot run EESSI test suite during test phase (part of bot build procedure) #467

let bot run EESSI test suite during test phase (part of bot build procedure) #467

casparvl commented Feb 5, 2024

eessi-bot bot commented Feb 5, 2024

casparvl commented Feb 5, 2024

eessi-bot bot commented Feb 5, 2024 •

edited

Loading

eessi-bot bot commented Feb 5, 2024 •

edited

Loading

casparvl commented Feb 6, 2024

casparvl commented Feb 6, 2024

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

casparvl commented Feb 6, 2024

casparvl commented Feb 6, 2024

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

casparvl commented Feb 6, 2024

casparvl commented Feb 6, 2024

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

casparvl commented Feb 6, 2024

casparvl commented Feb 6, 2024

casparvl commented Feb 6, 2024 •

edited

Loading

casparvl commented Feb 6, 2024

boegel commented Mar 26, 2024

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

boegel commented Mar 26, 2024

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

let bot run EESSI test suite during test phase (part of bot build procedure) #467

let bot run EESSI test suite during test phase (part of bot build procedure) #467

Conversation

casparvl commented Feb 5, 2024

eessi-bot bot commented Feb 5, 2024

casparvl commented Feb 5, 2024

eessi-bot bot commented Feb 5, 2024 • edited Loading

eessi-bot bot commented Feb 5, 2024 • edited Loading

casparvl commented Feb 6, 2024

casparvl commented Feb 6, 2024

eessi-bot bot commented Feb 6, 2024 • edited Loading

eessi-bot bot commented Feb 6, 2024 • edited Loading

casparvl commented Feb 6, 2024

casparvl commented Feb 6, 2024

eessi-bot bot commented Feb 6, 2024 • edited Loading

eessi-bot bot commented Feb 6, 2024 • edited Loading

casparvl commented Feb 6, 2024

casparvl commented Feb 6, 2024

eessi-bot bot commented Feb 6, 2024 • edited Loading

eessi-bot bot commented Feb 6, 2024 • edited Loading

casparvl commented Feb 6, 2024

casparvl commented Feb 6, 2024

casparvl commented Feb 6, 2024 • edited Loading

casparvl commented Feb 6, 2024

boegel commented Mar 26, 2024

eessi-bot bot commented Mar 26, 2024 • edited Loading

eessi-bot bot commented Mar 26, 2024 • edited Loading

eessi-bot bot commented Mar 26, 2024 • edited Loading

eessi-bot bot commented Mar 26, 2024 • edited Loading

eessi-bot bot commented Mar 26, 2024 • edited Loading

boegel commented Mar 26, 2024

eessi-bot bot commented Mar 26, 2024 • edited Loading

eessi-bot bot commented Mar 26, 2024 • edited Loading

eessi-bot bot commented Mar 26, 2024 • edited Loading

eessi-bot bot commented Mar 26, 2024 • edited Loading

eessi-bot bot commented Mar 26, 2024 • edited Loading

eessi-bot bot commented Feb 5, 2024 •

edited

Loading

eessi-bot bot commented Feb 5, 2024 •

edited

Loading

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

eessi-bot bot commented Feb 6, 2024 •

edited

Loading

casparvl commented Feb 6, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading

eessi-bot bot commented Mar 26, 2024 •

edited

Loading