Disable `--with-sysroot` option for UCX #133

bedroge · 2025-11-20T11:45:38Z

In both EESSI/software-layer#1299 (comment) and EESSI/software-layer#1294 we've seen the following error for applications that try to link against UCX:

../../libtool: line 3232: cd: =/cvmfs/software.eessi.io/versions/2025.06/software/linux/aarch64/nvidia/grace/software/UCX/1.16.0-GCCcore-13.3.0/lib: No such file or directory
libtool: link: warning: cannot determine absolute directory name of `=/cvmfs/software.eessi.io/versions/2025.06/software/linux/aarch64/nvidia/grace/software/UCX/1.16.0-GCCcore-13.3.0/lib'
grep: =/cvmfs/software.eessi.io/versions/2025.06/software/linux/aarch64/nvidia/grace/software/UCX/1.16.0-GCCcore-13.3.0/lib/libuct.la: No such file or directory
/cvmfs/software.eessi.io/versions/2025.06/compat/linux/aarch64/bin/sed: can't read =/cvmfs/software.eessi.io/versions/2025.06/software/linux/aarch64/nvidia/grace/software/UCX/1.16.0-GCCcore-13.3.0/lib/libuct.la: No such file or directory
libtool: link: `=/cvmfs/software.eessi.io/versions/2025.06/software/linux/aarch64/nvidia/grace/software/UCX/1.16.0-GCCcore-13.3.0/lib/libuct.la' is not a valid libtool archive

UCX's $EBROOTUCX/lib/*.la files contain these paths prefixed with a =, which are, according to ChatGPT, "paths relative to the installed libdir of this library". They're not relative though, but absolute paths, and this seems to be caused by the --with-sysroot option. Compiling it without that option should be fine, it should even use the compiler's sysroot in this case:

  --with-sysroot=DIR Search for dependent libraries within DIR
                        (or the compiler's sysroot if not specified).

To make sure that this works, I'll do test builds of a few UCX versions here.

bedroge · 2025-11-20T11:59:59Z

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2

eessi-bot-aws · 2025-11-20T12:00:07Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2023.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.11/pr_133/106263

date	job status	comment
Nov 20 12:00:05 UTC 2025	submitted	job id `106263` awaits release by job manager
Nov 20 12:01:09 UTC 2025	released	job awaits launch by Slurm scheduler
Nov 20 12:08:15 UTC 2025	running	job `106263` is running
Nov 20 12:12:22 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-106263.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.* created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-17636405410.tar.zst` size: 0 MiB (23033 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball reprod directories under 2023.06/software/linux/x86_64/amd/zen2/reprod no reprod directories in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/easybuild/eb_hooks.py`
Nov 20 12:12:22 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86_64_amd_zen2+default P: perf: 446.764 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86_64_amd_zen2+default P: perf: 451.962 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86_64_amd_zen2+default P: latency: 4.41 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86_64_amd_zen2+default P: latency: 2.92 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86_64_amd_zen2+default P: latency: 6.2 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86_64_amd_zen2+default P: latency: 5.85 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.85 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.8 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 6374.81 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 6377.45 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-106263.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2025-11-20T12:00:12Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.11/pr_133/106264

date	job status	comment
Nov 20 12:00:10 UTC 2025	submitted	job id `106264` awaits release by job manager
Nov 20 12:01:07 UTC 2025	released	job awaits launch by Slurm scheduler
Nov 20 12:07:12 UTC 2025	running	job `106264` is running
Nov 20 12:09:17 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-106264.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.* created!` Artefacts `eessi-2025.06-software-linux-x86_64-amd-zen2-17636404700.tar.zst` size: 0 MiB (23035 bytes) entries: 1 modules under 2025.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2025.06/software/linux/x86_64/amd/zen2/software no software packages in tarball reprod directories under 2025.06/software/linux/x86_64/amd/zen2/reprod no reprod directories in tarball other under 2025.06/software/linux/x86_64/amd/zen2 `2025.06/init/easybuild/eb_hooks.py`
Nov 20 12:09:17 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] (1/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:x86_64_amd_zen2+default P: latency: 1.53 us (r:0, l:None, u:None) [ OK ] (2/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:x86_64_amd_zen2+default P: latency: 2.07 us (r:0, l:None, u:None) [ OK ] (3/4) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.18 us (r:0, l:None, u:None) [ OK ] (4/4) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7850.72 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 4/4 test case(s) from 4 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-106264.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

bedroge · 2025-11-20T12:45:20Z

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2

eessi-bot-aws · 2025-11-20T12:45:26Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2023.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.11/pr_133/106265

date	job status	comment
Nov 20 12:45:25 UTC 2025	submitted	job id `106265` awaits release by job manager
Nov 20 12:46:29 UTC 2025	released	job awaits launch by Slurm scheduler
Nov 20 12:47:34 UTC 2025	running	job `106265` is running
Nov 20 13:05:11 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-106265.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.* created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-17636437180.tar.zst` size: 49 MiB (51490143 bytes) entries: 575 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all `UCX/1.13.1-GCCcore-12.2.0.lua` `UCX/1.14.1-GCCcore-12.3.0.lua` `UCX/1.15.0-GCCcore-13.2.0.lua` software under 2023.06/software/linux/x86_64/amd/zen2/software `UCX/1.13.1-GCCcore-12.2.0` `UCX/1.14.1-GCCcore-12.3.0` `UCX/1.15.0-GCCcore-13.2.0` reprod directories under 2023.06/software/linux/x86_64/amd/zen2/reprod no reprod directories in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/easybuild/eb_hooks.py`
Nov 20 13:05:11 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86_64_amd_zen2+default P: perf: 445.11 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86_64_amd_zen2+default P: perf: 455.545 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86_64_amd_zen2+default P: latency: 2.63 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86_64_amd_zen2+default P: latency: 2.68 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86_64_amd_zen2+default P: latency: 6.02 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86_64_amd_zen2+default P: latency: 6.17 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.97 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.66 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 6370.57 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 6369.04 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-106265.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot-aws · 2025-11-20T12:45:31Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.11/pr_133/106266

date	job status	comment
Nov 20 12:45:30 UTC 2025	submitted	job id `106266` awaits release by job manager
Nov 20 12:46:28 UTC 2025	released	job awaits launch by Slurm scheduler
Nov 20 12:47:32 UTC 2025	running	job `106266` is running
Nov 20 13:02:06 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-106266.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.* created!` Artefacts `eessi-2025.06-software-linux-x86_64-amd-zen2-17636436070.tar.zst` size: 35 MiB (37598208 bytes) entries: 494 modules under 2025.06/software/linux/x86_64/amd/zen2/modules/all `UCX/1.16.0-GCCcore-13.3.0.lua` `UCX/1.18.0-GCCcore-14.2.0.lua` software under 2025.06/software/linux/x86_64/amd/zen2/software `UCX/1.16.0-GCCcore-13.3.0` `UCX/1.18.0-GCCcore-14.2.0` reprod directories under 2025.06/software/linux/x86_64/amd/zen2/reprod `UCX/1.16.0-GCCcore-13.3.0/20250924_092946UTC` `UCX/1.16.0-GCCcore-13.3.0/20251030_130512UTC` `UCX/1.16.0-GCCcore-13.3.0/20251120_125316UTC` `UCX/1.18.0-GCCcore-14.2.0/20251012_084446UTC` `UCX/1.18.0-GCCcore-14.2.0/20251030_131147UTC` `UCX/1.18.0-GCCcore-14.2.0/20251120_125956UTC` other under 2025.06/software/linux/x86_64/amd/zen2 `2025.06/init/easybuild/eb_hooks.py`
Nov 20 13:02:06 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] (1/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:x86_64_amd_zen2+default P: latency: 1.46 us (r:0, l:None, u:None) [ OK ] (2/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:x86_64_amd_zen2+default P: latency: 2.07 us (r:0, l:None, u:None) [ OK ] (3/4) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.19 us (r:0, l:None, u:None) [ OK ] (4/4) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7764.95 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 4/4 test case(s) from 4 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-106266.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

bedroge · 2025-11-20T15:10:26Z

So, UCX builds fine without the --with-sysroot flag. One more build to verify that Cgl builds without issues now.

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2

eessi-bot-aws · 2025-11-20T15:10:33Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.11/pr_133/106267

date	job status	comment
Nov 20 15:10:32 UTC 2025	submitted	job id `106267` awaits release by job manager
Nov 20 15:11:25 UTC 2025	released	job awaits launch by Slurm scheduler
Nov 20 15:17:28 UTC 2025	running	job `106267` is running
Nov 20 15:53:13 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-106267.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.* created!` Artefacts `eessi-2025.06-software-linux-x86_64-amd-zen2-17636538840.tar.zst` size: 30 MiB (31736484 bytes) entries: 825 modules under 2025.06/software/linux/x86_64/amd/zen2/modules/all `Cgl/0.60.8-foss-2024a.lua` `Clp/1.17.10-foss-2024a.lua` `CoinUtils/2.11.12-GCC-13.3.0.lua` `METIS/5.1.0-GCCcore-13.3.0.lua` `MUMPS/5.7.2-foss-2024a-metis.lua` `Osi/0.108.11-GCC-13.3.0.lua` `SCOTCH/7.0.6-gompi-2024a.lua` `UCX/1.16.0-GCCcore-13.3.0.lua` software under 2025.06/software/linux/x86_64/amd/zen2/software `Cgl/0.60.8-foss-2024a` `Clp/1.17.10-foss-2024a` `CoinUtils/2.11.12-GCC-13.3.0` `METIS/5.1.0-GCCcore-13.3.0` `MUMPS/5.7.2-foss-2024a-metis` `Osi/0.108.11-GCC-13.3.0` `SCOTCH/7.0.6-gompi-2024a` `UCX/1.16.0-GCCcore-13.3.0` reprod directories under 2025.06/software/linux/x86_64/amd/zen2/reprod `Cgl/0.60.8-foss-2024a/20251120_155112UTC` `Clp/1.17.10-foss-2024a/20251120_154633UTC` `CoinUtils/2.11.12-GCC-13.3.0/20251120_152528UTC` `METIS/5.1.0-GCCcore-13.3.0/20251120_152756UTC` `MUMPS/5.7.2-foss-2024a-metis/20251120_154258UTC` `Osi/0.108.11-GCC-13.3.0/20251120_152652UTC` `SCOTCH/7.0.6-gompi-2024a/20251120_153323UTC` `UCX/1.16.0-GCCcore-13.3.0/20250924_092946UTC` `UCX/1.16.0-GCCcore-13.3.0/20251030_130512UTC` `UCX/1.16.0-GCCcore-13.3.0/20251120_152342UTC` other under 2025.06/software/linux/x86_64/amd/zen2 `2025.06/init/easybuild/eb_hooks.py`
Nov 20 15:53:13 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] (1/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:x86_64_amd_zen2+default P: latency: 1.32 us (r:0, l:None, u:None) [ OK ] (2/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:x86_64_amd_zen2+default P: latency: 2.05 us (r:0, l:None, u:None) [ OK ] (3/4) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.22 us (r:0, l:None, u:None) [ OK ] (4/4) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7811.5 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 4/4 test case(s) from 4 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-106267.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

bedroge · 2025-11-20T15:18:33Z

Also just checked the libuct.la file from one of the UCX builds, and it now looks like this:

# Libraries that this one depends upon.
dependency_libs=' -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/numactl/2.0.16-GCCcore-13.2.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/numactl/2.0.16-GCCcore-13.2.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/pkgconf/2.0.3-GCCcore-13.2.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/pkgconf/2.0.3-GCCcore-13.2.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/GCCcore/13.2.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/GCCcore/13.2.0/lib /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/UCX/1.15.0-GCCcore-13.2.0/lib/libucs.la /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/UCX/1.15.0-GCCcore-13.2.0/lib/libucm.la -ldl -lrt -lm -lpthread'

No more paths starting with a =.

bedroge · 2025-11-20T15:59:54Z

That also worked, so let's do the final build.

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2

eessi-bot-aws · 2025-11-20T16:00:01Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2023.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.11/pr_133/106268

date	job status	comment
Nov 20 16:00:00 UTC 2025	submitted	job id `106268` awaits release by job manager
Nov 20 16:00:19 UTC 2025	released	job awaits launch by Slurm scheduler
Nov 20 16:07:26 UTC 2025	running	job `106268` is running
Nov 20 16:10:30 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-106268.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.* created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-17636548320.tar.zst` size: 0 MiB (23033 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball reprod directories under 2023.06/software/linux/x86_64/amd/zen2/reprod no reprod directories in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/easybuild/eb_hooks.py`
Nov 20 16:10:30 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86_64_amd_zen2+default P: perf: 447.549 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86_64_amd_zen2+default P: perf: 453.299 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86_64_amd_zen2+default P: latency: 2.43 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86_64_amd_zen2+default P: latency: 2.56 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86_64_amd_zen2+default P: latency: 6.34 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86_64_amd_zen2+default P: latency: 5.87 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.85 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.79 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 6309.38 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 6373.43 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-106268.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Nov 20 16:59:43 UTC 2025	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen2-17636548320.tar.zst` to S3 bucket succeeded

eessi-bot-aws · 2025-11-20T16:00:07Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.11/pr_133/106269

date	job status	comment
Nov 20 16:00:06 UTC 2025	submitted	job id `106269` awaits release by job manager
Nov 20 16:00:17 UTC 2025	released	job awaits launch by Slurm scheduler
Nov 20 16:01:22 UTC 2025	running	job `106269` is running
Nov 20 16:02:23 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-106269.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.* created!` Artefacts `eessi-2025.06-software-linux-x86_64-amd-zen2-17636544410.tar.zst` size: 0 MiB (23035 bytes) entries: 1 modules under 2025.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2025.06/software/linux/x86_64/amd/zen2/software no software packages in tarball reprod directories under 2025.06/software/linux/x86_64/amd/zen2/reprod no reprod directories in tarball other under 2025.06/software/linux/x86_64/amd/zen2 `2025.06/init/easybuild/eb_hooks.py`
Nov 20 16:02:23 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] (1/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:x86_64_amd_zen2+default P: latency: 4.22 us (r:0, l:None, u:None) [ OK ] (2/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:x86_64_amd_zen2+default P: latency: 2.04 us (r:0, l:None, u:None) [ OK ] (3/4) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.18 us (r:0, l:None, u:None) [ OK ] (4/4) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7877.44 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 4/4 test case(s) from 4 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-106269.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Nov 20 16:59:51 UTC 2025	uploaded	transfer of `eessi-2025.06-software-linux-x86_64-amd-zen2-17636544410.tar.zst` to S3 bucket succeeded

ocaisa

LGTM

boegel · 2025-11-20T19:01:47Z

@bedroge Did you check the details on why we started using --with-sysroot in the first place with UCX?
See EESSI/software-layer#75 (comment)

I'm not opposing this change, but we should try and make try and make sure it won't hit us in the face somewhere else later...

bedroge · 2025-11-20T19:05:28Z

@bedroge Did you check the details on why we started using --with-sysroot in the first place with UCX?

See EESSI/software-layer#75 (comment)

I'm not opposing this change, but we should try and make try and make sure it won't hit us in the face somewhere else later...

Couldn't really find a particular reason, only the comment in the hooks file itself. It seems like it was added in EESSI/software-layer#75, but that doesn't say anything about it either.

bedroge · 2025-11-20T19:14:29Z

Also see EESSI/software-layer#63. It was probably added in an attempt to pick up the right rdmacm libraries, but I don't think this flag was required for that (as I apparently had already suggested in that PR).

boegel · 2025-11-20T19:31:59Z

Updated hooks have been ingested, so this can be merged...

bedroge added 3 commits November 20, 2025 12:42

disable --with-sysroot for UCX

196b4b4

2023.06 UCX versions

168e6e0

2025.06 UCX versions

1ba694c

Add rebuild: true

498e76d

bedroge added 2 commits November 20, 2025 16:09

build Cgl (which was affected by this issue)

4fd832e

remove 2023.06 easystack

65241d6

remove easystack

4f621d6

ocaisa approved these changes Nov 20, 2025

View reviewed changes

ocaisa added the bot:deploy label Nov 20, 2025

boegel merged commit 710b8fd into EESSI:main Nov 20, 2025
67 of 70 checks passed

bedroge deleted the ucx_remove_sysroot branch November 20, 2025 21:15

Disable --with-sysroot option for UCX #133

Disable --with-sysroot option for UCX #133

Uh oh!

Conversation

bedroge commented Nov 20, 2025

Uh oh!

bedroge commented Nov 20, 2025

Uh oh!

eessi-bot-aws bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot-aws bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedroge commented Nov 20, 2025

Uh oh!

eessi-bot-aws bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot-aws bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedroge commented Nov 20, 2025

Uh oh!

eessi-bot-aws bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedroge commented Nov 20, 2025

Uh oh!

bedroge commented Nov 20, 2025

Uh oh!

eessi-bot-aws bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot-aws bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

boegel commented Nov 20, 2025

Uh oh!

bedroge commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedroge commented Nov 20, 2025

Uh oh!

boegel commented Nov 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Disable `--with-sysroot` option for UCX #133

Disable `--with-sysroot` option for UCX #133

eessi-bot-aws bot commented Nov 20, 2025 •

edited

Loading

eessi-bot-aws bot commented Nov 20, 2025 •

edited

Loading

eessi-bot-aws bot commented Nov 20, 2025 •

edited

Loading

eessi-bot-aws bot commented Nov 20, 2025 •

edited

Loading

eessi-bot-aws bot commented Nov 20, 2025 •

edited

Loading

eessi-bot-aws bot commented Nov 20, 2025 •

edited

Loading

eessi-bot-aws bot commented Nov 20, 2025 •

edited

Loading

bedroge commented Nov 20, 2025 •

edited

Loading