{2023.06}[foss/2023a] OpenFOAM v2312 #563

boegel · 2024-05-07T13:21:40Z

2 out of 113 required modules missing:

* KaHIP/3.16-gompi-2023a (KaHIP-3.16-gompi-2023a.eb)
* OpenFOAM/v2312-foss-2023a (OpenFOAM-v2312-foss-2023a.eb)

eessi-bot-aws · 2024-05-07T13:21:45Z

Instance eessi-bot-mc-aws is configured to build:

arch x86_64/generic for repo eessi-hpc.org-2023.06-compat
arch x86_64/generic for repo eessi-hpc.org-2023.06-software
arch x86_64/generic for repo eessi.io-2023.06-compat
arch x86_64/generic for repo eessi.io-2023.06-software
arch x86_64/intel/haswell for repo eessi-hpc.org-2023.06-compat
arch x86_64/intel/haswell for repo eessi-hpc.org-2023.06-software
arch x86_64/intel/haswell for repo eessi.io-2023.06-compat
arch x86_64/intel/haswell for repo eessi.io-2023.06-software
arch x86_64/intel/skylake_avx512 for repo eessi-hpc.org-2023.06-compat
arch x86_64/intel/skylake_avx512 for repo eessi-hpc.org-2023.06-software
arch x86_64/intel/skylake_avx512 for repo eessi.io-2023.06-compat
arch x86_64/intel/skylake_avx512 for repo eessi.io-2023.06-software
arch x86_64/amd/zen2 for repo eessi-hpc.org-2023.06-compat
arch x86_64/amd/zen2 for repo eessi-hpc.org-2023.06-software
arch x86_64/amd/zen2 for repo eessi.io-2023.06-compat
arch x86_64/amd/zen2 for repo eessi.io-2023.06-software
arch x86_64/amd/zen3 for repo eessi-hpc.org-2023.06-compat
arch x86_64/amd/zen3 for repo eessi-hpc.org-2023.06-software
arch x86_64/amd/zen3 for repo eessi.io-2023.06-compat
arch x86_64/amd/zen3 for repo eessi.io-2023.06-software
arch aarch64/generic for repo eessi-hpc.org-2023.06-compat
arch aarch64/generic for repo eessi-hpc.org-2023.06-software
arch aarch64/generic for repo eessi.io-2023.06-compat
arch aarch64/generic for repo eessi.io-2023.06-software
arch aarch64/neoverse_n1 for repo eessi-hpc.org-2023.06-compat
arch aarch64/neoverse_n1 for repo eessi-hpc.org-2023.06-software
arch aarch64/neoverse_n1 for repo eessi.io-2023.06-compat
arch aarch64/neoverse_n1 for repo eessi.io-2023.06-software
arch aarch64/neoverse_v1 for repo eessi-hpc.org-2023.06-compat
arch aarch64/neoverse_v1 for repo eessi-hpc.org-2023.06-software
arch aarch64/neoverse_v1 for repo eessi.io-2023.06-compat
arch aarch64/neoverse_v1 for repo eessi.io-2023.06-software

eessi-bot-aws · 2024-05-07T13:21:45Z

Instance eessi-bot-mc-azure is configured to build:

arch x86_64/amd/zen4 for repo eessi-hpc.org-2023.06-compat
arch x86_64/amd/zen4 for repo eessi-hpc.org-2023.06-software
arch x86_64/amd/zen4 for repo eessi.io-2023.06-compat
arch x86_64/amd/zen4 for repo eessi.io-2023.06-software

boegel · 2024-05-07T13:23:17Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3

eessi-bot-aws · 2024-05-07T13:23:20Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- submitted job 10305, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)

eessi-bot-aws · 2024-05-07T13:23:20Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- no jobs were submitted

eessi-bot-aws · 2024-05-07T13:23:45Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.05/pr_563/10305

date	job status	comment
May 07 13:23:44 UTC 2024	submitted	job id `10305` awaits release by job manager
May 07 13:24:28 UTC 2024	released	job awaits launch by Slurm scheduler
May 07 13:29:56 UTC 2024	running	job `10305` is running
May 07 14:50:10 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-10305.out` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1715092555.tar.gz` size: 11 MiB (11585324 bytes) entries: 44 modules under 2023.06/software/linux/x86_64/amd/zen3/modules/all `KaHIP/3.16-gompi-2023a.lua` software under 2023.06/software/linux/x86_64/amd/zen3/software `KaHIP/3.16-gompi-2023a` other under 2023.06/software/linux/x86_64/amd/zen3 no other files in tarball
May 07 14:50:10 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-10305.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.1-2023a.yml

Co-authored-by: ocaisa <alan.ocais@cecam.org>

…io_OpenFOAM-v2312

bedroge · 2024-06-07T12:20:05Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3

eessi-bot-aws · 2024-06-07T12:20:08Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- submitted job 12272, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)

eessi-bot-aws · 2024-06-07T12:20:08Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- no jobs were submitted

eessi-bot-aws · 2024-06-07T12:20:12Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/12272

date	job status	comment
Jun 07 12:20:11 UTC 2024	submitted	job id `12272` awaits release by job manager
Jun 07 12:20:44 UTC 2024	released	job awaits launch by Slurm scheduler
Jun 07 12:21:49 UTC 2024	running	job `12272` is running
Jun 07 12:25:53 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-12272.out` ❌ found message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts No artefacts were created or found.
Jun 07 12:25:53 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-12272.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

bedroge · 2024-06-07T12:27:51Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3

eessi-bot-aws · 2024-06-07T12:27:54Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- submitted job 12274, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)

eessi-bot-aws · 2024-06-07T12:27:54Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- no jobs were submitted

eessi-bot-aws · 2024-06-07T12:27:58Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/12274

date	job status	comment
Jun 07 12:27:57 UTC 2024	submitted	job id `12274` awaits release by job manager
Jun 07 12:29:00 UTC 2024	released	job awaits launch by Slurm scheduler
Jun 07 12:30:05 UTC 2024	running	job `12274` is running
Jun 07 13:41:59 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-12274.out` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1717767335.tar.gz` size: 11 MiB (11586200 bytes) entries: 44 modules under 2023.06/software/linux/x86_64/amd/zen3/modules/all `KaHIP/3.16-gompi-2023a.lua` software under 2023.06/software/linux/x86_64/amd/zen3/software `KaHIP/3.16-gompi-2023a` other under 2023.06/software/linux/x86_64/amd/zen3 no other files in tarball
Jun 07 13:41:59 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-12274.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

bedroge · 2024-06-07T14:11:31Z

Sanity check is failing because of missing binaries (e.g. boundaryFoam and engineFoam). They should be in OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/bin, but they're not, while the same version on generoso does have them:

$ ls -1 /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/bin/ | wc -l
133

$ ls -1 /project/boegelbot/Rocky8/haswell/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/bin | wc -l
316
$

But I can't find any explanation for this in the logs...

bedroge · 2024-06-07T14:19:49Z

Oh, actually, it looks like the build step did fail, even though EB says it completed. In the log file of the build step itself I see lots of errors near the bottom, most of them are like this:

/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld.bfd: insertFeaturePoints.C:(.text+0xc19): undefined reference to `__gmpn_mul'
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld.bfd: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/OpenFO
AM/v2312-foss-2023a/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libconformalVoronoiMesh.so: undefined reference to `mpfr_get_emin'
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld.bfd: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/OpenFO
AM/v2312-foss-2023a/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libconformalVoronoiMesh.so: undefined reference to `__gmpn_com'
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld.bfd: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/OpenFO
AM/v2312-foss-2023a/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libconformalVoronoiMesh.so: undefined reference to `__gmpq_swap'
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld.bfd: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/OpenFO
AM/v2312-foss-2023a/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libconformalVoronoiMesh.so: undefined reference to `__gmpq_init'
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld.bfd: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libconformalVoronoiMesh.so: undefined reference to `__gmpn_copyi'
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld.bfd: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libconformalVoronoiMesh.so: undefined reference to `mpfr_set_q'
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld.bfd: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libconformalVoronoiMesh.so: undefined reference to `__gmpn_add_n'

ocaisa · 2024-06-07T14:23:10Z

Ah, GMP is not included in the list of dependencies but it clearly is one

bedroge · 2024-06-07T14:25:08Z

Ah, GMP is not included in the list of dependencies but it clearly is one

Yep, and MPFR.

bedroge · 2024-06-07T14:30:17Z

Doing an interactive build now with both of them added to the deps:

    ('GMP', '6.2.1'),
    ('MPFR', '4.2.0'),

It's worrying though that the build actually failed, but apparently this Allwmake command returned with an exit code 0?

boegel · 2024-06-07T14:40:49Z

Doing an interactive build now with both of them added to the deps:
    ('GMP', '6.2.1'),
    ('MPFR', '4.2.0'),
It's worrying though that the build actually failed, but apparently this Allwmake command returned with an exit code 0?

If I remember correctly this was one of the motivations for implementing the sanity check concept in EasyBuild :)

bedroge · 2024-06-07T17:59:48Z

Adding those dependencies didn't help, still getting the same issue/errors.

bedroge · 2024-06-07T18:26:10Z

It's probably related to CGAL, which could previously be built in different ways. Since version 5 it's header-only, but when using this with OpenFOAM, it looks like MPFR and GMP are still required. There seems to be functionality in OpenFOAM that should detect which variant of CGAL you have, but I find lots of similar issues, e.g. this one with Spack: https://develop.openfoam.com/Development/openfoam/-/issues/3101.

bedroge · 2024-06-11T15:39:08Z

Confirmed, it's related to CGAL. In Spack they even stick to CGAL 4, probably because of similar issues:
https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/openfoam/package.py#L372

I suspect it pops up now because we're filtering $LD_LIBRARY_PATH. As a sort of a quick/dirty workaround I've added a command to preconfigopts that injects the MPFR and GMP directories to CGAL_LIBS somewhere, and that allowed the build to complete. I'll clean this up and make a PR for it.

…M-v2312 move OpenFOAM to EB 4.9.2 easystack

bedroge · 2024-06-19T10:03:19Z

bot: build repo:eessi.io-2023.06-software arch:aarch64/generic
bot: build repo:eessi.io-2023.06-software arch:aarch64/neoverse_n1
bot: build repo:eessi.io-2023.06-software arch:aarch64/neoverse_v1
bot: build repo:eessi.io-2023.06-software arch:x86_64/generic
bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2
bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3
bot: build repo:eessi.io-2023.06-software arch:x86_64/intel/haswell
bot: build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512

eessi-bot-aws · 2024-06-19T10:03:22Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:aarch64/generic from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/generic
received bot command build repo:eessi.io-2023.06-software arch:aarch64/neoverse_n1 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_n1
received bot command build repo:eessi.io-2023.06-software arch:aarch64/neoverse_v1 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_v1
received bot command build repo:eessi.io-2023.06-software arch:x86_64/generic from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/generic
received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/haswell from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/haswell
received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512
handling command build repository:eessi.io-2023.06-software architecture:aarch64/generic resulted in:
- submitted job 13044, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)
handling command build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_n1 resulted in:
- submitted job 13045, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)
handling command build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_v1 resulted in:
- submitted job 13046, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/generic resulted in:
- submitted job 13047, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 13048, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- submitted job 13049, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/haswell resulted in:
- submitted job 13050, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512 resulted in:
- submitted job 13051, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)

eessi-bot-aws · 2024-06-19T10:03:22Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:aarch64/generic from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/generic
received bot command build repo:eessi.io-2023.06-software arch:aarch64/neoverse_n1 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_n1
received bot command build repo:eessi.io-2023.06-software arch:aarch64/neoverse_v1 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_v1
received bot command build repo:eessi.io-2023.06-software arch:x86_64/generic from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/generic
received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/haswell from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/haswell
received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512
handling command build repository:eessi.io-2023.06-software architecture:aarch64/generic resulted in:
- no jobs were submitted
handling command build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_n1 resulted in:
- no jobs were submitted
handling command build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_v1 resulted in:
- no jobs were submitted
handling command build repository:eessi.io-2023.06-software architecture:x86_64/generic resulted in:
- no jobs were submitted
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- no jobs were submitted
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- no jobs were submitted
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/haswell resulted in:
- no jobs were submitted
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512 resulted in:
- no jobs were submitted

eessi-bot-aws · 2024-06-19T10:03:26Z

New job on instance eessi-bot-mc-aws for architecture aarch64-generic for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/13044

date	job status	comment
Jun 19 10:03:25 UTC 2024	submitted	job id `13044` awaits release by job manager
Jun 19 10:04:12 UTC 2024	released	job awaits launch by Slurm scheduler
Jun 19 10:05:37 UTC 2024	running	job `13044` is running
Jun 19 12:06:38 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-13044.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-generic-1718797312.tar.gz` size: 527 MiB (552641970 bytes) entries: 65533 modules under 2023.06/software/linux/aarch64/generic/modules/all `KaHIP/3.16-gompi-2023a.lua` `OpenFOAM/v2312-foss-2023a.lua` software under 2023.06/software/linux/aarch64/generic/software `KaHIP/3.16-gompi-2023a` `OpenFOAM/v2312-foss-2023a` other under 2023.06/software/linux/aarch64/generic no other files in tarball
Jun 19 12:06:38 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 12/12 test case(s) from 12 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-13044.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2024-06-19T10:03:30Z

New job on instance eessi-bot-mc-aws for architecture aarch64-neoverse_n1 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/13045

date	job status	comment
Jun 19 10:03:29 UTC 2024	submitted	job id `13045` awaits release by job manager
Jun 19 10:04:15 UTC 2024	released	job awaits launch by Slurm scheduler
Jun 19 10:09:04 UTC 2024	running	job `13045` is running
Jun 19 12:12:07 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-13045.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_n1-1718797678.tar.gz` size: 528 MiB (553793208 bytes) entries: 65533 modules under 2023.06/software/linux/aarch64/neoverse_n1/modules/all `KaHIP/3.16-gompi-2023a.lua` `OpenFOAM/v2312-foss-2023a.lua` software under 2023.06/software/linux/aarch64/neoverse_n1/software `KaHIP/3.16-gompi-2023a` `OpenFOAM/v2312-foss-2023a` other under 2023.06/software/linux/aarch64/neoverse_n1 no other files in tarball
Jun 19 12:12:07 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 12/12 test case(s) from 12 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-13045.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2024-06-19T10:03:34Z

New job on instance eessi-bot-mc-aws for architecture aarch64-neoverse_v1 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/13046

date	job status	comment
Jun 19 10:03:33 UTC 2024	submitted	job id `13046` awaits release by job manager
Jun 19 10:04:17 UTC 2024	released	job awaits launch by Slurm scheduler
Jun 19 10:09:08 UTC 2024	running	job `13046` is running
Jun 19 11:37:14 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-13046.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_v1-1718796057.tar.gz` size: 528 MiB (553730757 bytes) entries: 65533 modules under 2023.06/software/linux/aarch64/neoverse_v1/modules/all `KaHIP/3.16-gompi-2023a.lua` `OpenFOAM/v2312-foss-2023a.lua` software under 2023.06/software/linux/aarch64/neoverse_v1/software `KaHIP/3.16-gompi-2023a` `OpenFOAM/v2312-foss-2023a` other under 2023.06/software/linux/aarch64/neoverse_v1 no other files in tarball
Jun 19 11:37:14 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 12/12 test case(s) from 12 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-13046.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2024-06-19T10:03:39Z

New job on instance eessi-bot-mc-aws for architecture x86_64-generic for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/13047

date	job status	comment
Jun 19 10:03:38 UTC 2024	submitted	job id `13047` awaits release by job manager
Jun 19 10:04:24 UTC 2024	released	job awaits launch by Slurm scheduler
Jun 19 10:11:37 UTC 2024	running	job `13047` is running
Jun 19 12:20:39 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-13047.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-generic-1718798102.tar.gz` size: 529 MiB (555660549 bytes) entries: 65533 modules under 2023.06/software/linux/x86_64/generic/modules/all `KaHIP/3.16-gompi-2023a.lua` `OpenFOAM/v2312-foss-2023a.lua` software under 2023.06/software/linux/x86_64/generic/software `KaHIP/3.16-gompi-2023a` `OpenFOAM/v2312-foss-2023a` other under 2023.06/software/linux/x86_64/generic no other files in tarball
Jun 19 12:20:39 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 12/12 test case(s) from 12 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-13047.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2024-06-19T10:03:43Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/13048

date	job status	comment
Jun 19 10:03:42 UTC 2024	submitted	job id `13048` awaits release by job manager
Jun 19 10:04:19 UTC 2024	released	job awaits launch by Slurm scheduler
Jun 19 10:11:31 UTC 2024	running	job `13048` is running
Jun 19 12:32:00 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-13048.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1718798853.tar.gz` size: 531 MiB (557146732 bytes) entries: 65533 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all `KaHIP/3.16-gompi-2023a.lua` `OpenFOAM/v2312-foss-2023a.lua` software under 2023.06/software/linux/x86_64/amd/zen2/software `KaHIP/3.16-gompi-2023a` `OpenFOAM/v2312-foss-2023a` other under 2023.06/software/linux/x86_64/amd/zen2 no other files in tarball
Jun 19 12:32:00 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 12/12 test case(s) from 12 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-13048.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2024-06-19T10:03:47Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/13049

date	job status	comment
Jun 19 10:03:46 UTC 2024	submitted	job id `13049` awaits release by job manager
Jun 19 10:04:21 UTC 2024	released	job awaits launch by Slurm scheduler
Jun 19 10:11:34 UTC 2024	running	job `13049` is running
Jun 19 12:08:51 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-13049.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1718797561.tar.gz` size: 531 MiB (557097477 bytes) entries: 65533 modules under 2023.06/software/linux/x86_64/amd/zen3/modules/all `KaHIP/3.16-gompi-2023a.lua` `OpenFOAM/v2312-foss-2023a.lua` software under 2023.06/software/linux/x86_64/amd/zen3/software `KaHIP/3.16-gompi-2023a` `OpenFOAM/v2312-foss-2023a` other under 2023.06/software/linux/x86_64/amd/zen3 no other files in tarball
Jun 19 12:08:51 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 12/12 test case(s) from 12 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-13049.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2024-06-19T10:03:51Z

New job on instance eessi-bot-mc-aws for architecture x86_64-intel-haswell for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/13050

date	job status	comment
Jun 19 10:03:50 UTC 2024	submitted	job id `13050` awaits release by job manager
Jun 19 10:04:26 UTC 2024	released	job awaits launch by Slurm scheduler
Jun 19 10:12:53 UTC 2024	running	job `13050` is running
Jun 19 12:24:50 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-13050.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-haswell-1718798220.tar.gz` size: 531 MiB (557655917 bytes) entries: 65533 modules under 2023.06/software/linux/x86_64/intel/haswell/modules/all `KaHIP/3.16-gompi-2023a.lua` `OpenFOAM/v2312-foss-2023a.lua` software under 2023.06/software/linux/x86_64/intel/haswell/software `KaHIP/3.16-gompi-2023a` `OpenFOAM/v2312-foss-2023a` other under 2023.06/software/linux/x86_64/intel/haswell no other files in tarball
Jun 19 12:24:50 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 12/12 test case(s) from 12 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-13050.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2024-06-19T10:03:55Z

New job on instance eessi-bot-mc-aws for architecture x86_64-intel-skylake_avx512 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/13051

date	job status	comment
Jun 19 10:03:54 UTC 2024	submitted	job id `13051` awaits release by job manager
Jun 19 10:04:28 UTC 2024	released	job awaits launch by Slurm scheduler
Jun 19 10:12:56 UTC 2024	running	job `13051` is running
Jun 19 10:45:32 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-13051.out` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1718792346.tar.gz` size: 10 MiB (11356671 bytes) entries: 44 modules under 2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all `KaHIP/3.16-gompi-2023a.lua` software under 2023.06/software/linux/x86_64/intel/skylake_avx512/software `KaHIP/3.16-gompi-2023a` other under 2023.06/software/linux/x86_64/intel/skylake_avx512 no other files in tarball
Jun 19 10:45:32 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 12/12 test case(s) from 12 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-13051.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

bedroge · 2024-06-20T07:29:55Z

New job on instance eessi-bot-mc-aws for architecture x86_64-intel-skylake_avx512 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.06/pr_563/13051

date job status comment
Jun 19 10:03:54 UTC 2024 submitted job id 13051 awaits release by job manager
Jun 19 10:04:28 UTC 2024 released job awaits launch by Slurm scheduler
Jun 19 10:12:56 UTC 2024 running job 13051 is running
Jun 19 10:45:32 UTC 2024 finished
😢 FAILURE (click triangle for details)
Jun 19 10:45:32 UTC 2024 test result
😁 SUCCESS (click triangle for details)

make: /bin/sh: Argument list too long
make: *** [/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/wmake/makefiles/general:210: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so] Error 127

Looks like the path for skylake is too long...

marconetto · 2024-06-27T13:33:32Z

HI, just to double check, once this is issue is fixed, will we be able to run OpenFoam on zen4?

ocaisa · 2024-06-27T14:38:56Z

@marconetto Yes and no, the Zen4 builds are happening separately as we still have to bring that stack in line with the other architectures. Having said that, zen4 can't go through without getting this PR over the line. It's not an easy one though as this is a difficult issue and will require a little design

marconetto · 2024-06-27T18:24:23Z

thanks for the clarification @ocaisa!

bedroge · 2024-07-05T20:13:35Z

Based on my comments in #296 (comment), I was hoping to fix this by only unsetting some environment variables. That didn't work though, and I'm still not fully sure why. The failing command does work fine when running it interactively, also with the same environment variables set as during the build. My only guess is that there's a bunch of makefiles calling/including other ones, which are also somehow increasing the size of the environment and apparently exceeding some limit.

While analyzing and debugging this, I came up with a workaround though. The failing command is compiling libOpenFOAM.so, and this needs to link together a whole bunch of object files. It uses the full path to these object files, but since the command is running in the build/installation directory, it can use relative paths. This greatly reduces the length of the command. The following hook uses that approach and worked on skylake (still need to be cleaned/extended a bit):

def pre_build_hook_openfoam(self, *args, **kwargs):
    if self.name == 'OpenFOAM':
        objects_regex = (
            r'(\$E \$\(LINKLIBSO\) \$\(FOAM_EXTRA_LDFLAGS\)) \$\(OBJECTS\) (-L\$\(LIB_PLATFORMS\))',
            r'\1 $(subst $(WM_PROJECT_DIR),$(shell realpath --relative-to=$(PWD) $(WM_PROJECT_DIR)),$(OBJECTS)) \2'
        )
        apply_regex_substitutions('wmake/makefiles/general', [objects_regex])

boegel · 2024-07-05T20:17:12Z

@bedroge Awesome work!

Instead of implementing this in a hook, why don't we bake it into the OpenFOAM easyblock instead, since this should be a general fix, it's not specific to EESSI at all?

bedroge · 2024-07-05T20:19:13Z

@bedroge Awesome work!

Instead of implementing this in a hook, why don't we bake it into the OpenFOAM easyblock instead, since this should be a general fix, it's not specific to EESSI at all?

Yes, you're absolutely right. I can open a PR in the easyblocks repo.

…io_OpenFOAM-v2312

bedroge · 2024-07-09T13:48:16Z

I've opened an easyblocks PR: easybuilders/easybuild-easyblocks#3388.
Let's give it a spin...

bedroge · 2024-07-09T13:48:25Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512

eessi-bot-aws · 2024-07-09T13:48:28Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512 resulted in:
- submitted job 14154, for details & status see {2023.06}[foss/2023a] OpenFOAM v2312 #563 (comment)

eessi-bot-aws · 2024-07-09T13:48:28Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512 from bedroge
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512 resulted in:
- no jobs were submitted

eessi-bot-aws · 2024-07-09T13:48:32Z

New job on instance eessi-bot-mc-aws for architecture x86_64-intel-skylake_avx512 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.07/pr_563/14154

date	job status	comment
Jul 09 13:48:31 UTC 2024	submitted	job id `14154` awaits release by job manager
Jul 09 13:49:03 UTC 2024	released	job awaits launch by Slurm scheduler
Jul 09 13:55:06 UTC 2024	running	job `14154` is running
Jul 09 15:49:48 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-14154.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1720539408.tar.gz` size: 531 MiB (556978919 bytes) entries: 65535 modules under 2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all `KaHIP/3.16-gompi-2023a.lua` `OpenFOAM/v2312-foss-2023a.lua` software under 2023.06/software/linux/x86_64/intel/skylake_avx512/software `KaHIP/3.16-gompi-2023a` `OpenFOAM/v2312-foss-2023a` other under 2023.06/software/linux/x86_64/intel/skylake_avx512 `2023.06/init/easybuild/eb_hooks.py`
Jul 09 15:49:48 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 14/14 test case(s) from 14 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-14154.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

{2023.06}[foss/2023a] OpenFOAM v2312

ae924e0

boegel added the 2023.06-software.eessi.io 2023.06 version of software.eessi.io label May 7, 2024

ocaisa reviewed May 14, 2024

View reviewed changes

easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.1-2023a.yml Outdated Show resolved Hide resolved

bedroge and others added 2 commits June 7, 2024 14:19

use PRs with fixes for easyconfig+easyblock

4084173

Co-authored-by: ocaisa <alan.ocais@cecam.org>

Merge branch '2023.06-software.eessi.io' into 2023.06-software.eessi.…

4a04398

…io_OpenFOAM-v2312

add missing colon

958e78a

bedroge mentioned this pull request Jun 17, 2024

make sure that OpenFOAM's wmake can find MPFR and GMP if CGAL >= 5.0 (header-only) is used easybuilders/easybuild-easyblocks#3366

Merged

bedroge and others added 3 commits June 18, 2024 10:26

move OpenFOAM to EB 4.9.2 easystack

3af05e9

Merge pull request #31 from bedroge/2023.06-software.eessi.io_OpenFOA…

70c024f

…M-v2312 move OpenFOAM to EB 4.9.2 easystack

use from-commit instead of from-pr

76c963f

bedroge mentioned this pull request Jun 21, 2024

Reducing the size of the end user environment + avoid long installation paths #296

Open

bedroge mentioned this pull request Jul 9, 2024

use relative paths to object files when compiling shared libraries in the OpenFOAM easyblock easybuilders/easybuild-easyblocks#3388

Open

bedroge added 2 commits July 9, 2024 15:47

use easyblock from PR 3388

391d5ed

Merge branch '2023.06-software.eessi.io' into 2023.06-software.eessi.…

2801ecd

…io_OpenFOAM-v2312

{2023.06}[foss/2023a] OpenFOAM v2312 #563

Are you sure you want to change the base?

{2023.06}[foss/2023a] OpenFOAM v2312 #563

Conversation

boegel commented May 7, 2024 • edited Loading

eessi-bot-aws bot commented May 7, 2024

eessi-bot-aws bot commented May 7, 2024

boegel commented May 7, 2024

eessi-bot-aws bot commented May 7, 2024 • edited Loading

eessi-bot-aws bot commented May 7, 2024 • edited Loading

eessi-bot-aws bot commented May 7, 2024 • edited Loading

bedroge commented Jun 7, 2024

eessi-bot-aws bot commented Jun 7, 2024 • edited Loading

eessi-bot-aws bot commented Jun 7, 2024 • edited Loading

eessi-bot-aws bot commented Jun 7, 2024 • edited Loading

bedroge commented Jun 7, 2024

eessi-bot-aws bot commented Jun 7, 2024 • edited Loading

eessi-bot-aws bot commented Jun 7, 2024 • edited Loading

eessi-bot-aws bot commented Jun 7, 2024 • edited Loading

bedroge commented Jun 7, 2024

bedroge commented Jun 7, 2024

ocaisa commented Jun 7, 2024

bedroge commented Jun 7, 2024

bedroge commented Jun 7, 2024

boegel commented Jun 7, 2024

bedroge commented Jun 7, 2024

bedroge commented Jun 7, 2024 • edited Loading

bedroge commented Jun 11, 2024 • edited Loading

bedroge commented Jun 19, 2024

eessi-bot-aws bot commented Jun 19, 2024 • edited Loading

eessi-bot-aws bot commented Jun 19, 2024 • edited Loading

eessi-bot-aws bot commented Jun 19, 2024 • edited Loading

eessi-bot-aws bot commented Jun 19, 2024 • edited Loading

eessi-bot-aws bot commented Jun 19, 2024 • edited Loading

eessi-bot-aws bot commented Jun 19, 2024 • edited Loading

eessi-bot-aws bot commented Jun 19, 2024 • edited Loading

eessi-bot-aws bot commented Jun 19, 2024 • edited Loading

eessi-bot-aws bot commented Jun 19, 2024 • edited Loading

eessi-bot-aws bot commented Jun 19, 2024 • edited Loading

bedroge commented Jun 20, 2024

marconetto commented Jun 27, 2024

ocaisa commented Jun 27, 2024

marconetto commented Jun 27, 2024 • edited Loading

bedroge commented Jul 5, 2024

boegel commented Jul 5, 2024

bedroge commented Jul 5, 2024

bedroge commented Jul 9, 2024

bedroge commented Jul 9, 2024

eessi-bot-aws bot commented Jul 9, 2024 • edited Loading

eessi-bot-aws bot commented Jul 9, 2024 • edited Loading

eessi-bot-aws bot commented Jul 9, 2024 • edited Loading

boegel commented May 7, 2024 •

edited

Loading

eessi-bot-aws bot commented May 7, 2024 •

edited

Loading

eessi-bot-aws bot commented May 7, 2024 •

edited

Loading

eessi-bot-aws bot commented May 7, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 7, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 7, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 7, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 7, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 7, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 7, 2024 •

edited

Loading

bedroge commented Jun 7, 2024 •

edited

Loading

bedroge commented Jun 11, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 19, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 19, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 19, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 19, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 19, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 19, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 19, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 19, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 19, 2024 •

edited

Loading

eessi-bot-aws bot commented Jun 19, 2024 •

edited

Loading

marconetto commented Jun 27, 2024 •

edited

Loading

eessi-bot-aws bot commented Jul 9, 2024 •

edited

Loading

eessi-bot-aws bot commented Jul 9, 2024 •

edited

Loading

eessi-bot-aws bot commented Jul 9, 2024 •

edited

Loading