[INSTALL]: builds using spack-stack 1.8.0 and 1.9.3 on derecho.hpc.ucar.edu no longer work

[jim-p-w]

Package name

spack-stack

Package version/tag

1.6.0, 1.8.0, 1.9.3

Build options

current

Installation timeframe

The jcsda software no longer builds on derecho. My regression testing is blocked.

Other information

jedi-bundle and mpas-bundle no longer build using spack-stack 1.8.0 or 1.9.3 on derecho.
The builds stopped working after the derecho OS upgrade on Nov 4-5.
See https://github.com/JCSDA-internal/jedi-bundle/issues/123 for details.

It appears librt.so and libpthreads.so have been removed from the /usr/lib64 directory.
I see /usr/lib64/librt.so and /usr/lib64/libpthreads.so in the CMakeCache.txt files for a number of modules in the spack-stack install directory:
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/envs/ue-gcc-12.4/install/gcc/12.4.0, e.g.
eckit-1.28.3-7sbode2/.spack/archived-files/spack-build-7sbode2/CMakeCache.txt

[climbfuji]

I wonder if a reinstall of everything does the trick, but:

Can't believe the sysadmins remove these libraries without thinking about the impact. Can you create a ticket with UCAR IT first and check if those libraries can be put back, please.

[rickgrubin-noaa]

derecho% cat /etc/os-release
NAME="SLES"
VERSION="15-SP6"
VERSION_ID="15.6"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"

Ugh; from a quick websearch:

For most SLES 15 Service Packs (prior to SP6), the functionalities of libpthread are part of the main C library, libc.so, which is provided by the glibc package. Dynamic linking still works as expected using the -lpthread flag during compilation.

Specifically:

• The actual shared library file is typically libpthread.so.0, which is part of the glibc package.
• The developer files (.so symlinks for linking purposes) were previously provided by the glibc-devel package.
• Starting with SLES 15 SP6, the libpthread, libdl, libutil, and libanl libraries were merged into the main libc library. The .so symlinks (like /usr/lib64/libpthread.so) are no longer provided by glibc-devel in SP6 and later, though the static .a files are still available for compatibility.

and

After upgrading to SLES 15 SP6, some libraries appear to be missing

Perhaps a rebuild might solve it.

[jim-p-w]

I wonder if a reinstall of everything does the trick, but:

Can't believe the sysadmins remove these libraries without thinking about the impact. Can you create a ticket with UCAR IT first and check if those libraries can be put back, please.

@climbfuji I opened up a ticket w/ UCAR last Fri: https://ithelp.ucar.edu/plugins/servlet/desk/portal/3/RC-35545
I am also working with CISL to see if we can find some cmake workaround which will enable us to use the current spack-stack installs. So far no progress.

Rick's comments make me think the libraries (locations and existence) are controlled by the SUSE packaging, not by UCAR IT. Given that it seems unlikely that the symlinks in /usr/lib64 are going to come back.

[vanderwb]

Hey all - I understand the frustration here. Certainly we were also surprised that upgrading from SLES SP4 to SP6 would result in this change. Unfortunately none of our tests caught this as none of them use eckit or ecbuild, and I yet to find another package in our stack or yours that contains this direct reference to /usr/lib64/librt.so.

I've been discussing with the system admins how we might fix or work around the problem. Hope to have more details on that today or tomorrow.

[climbfuji]

Thanks Brian! Hopefully a rebuild will do

[rickgrubin-noaa]

Rebuilding on derecho:

oneAPI: fails to configure nemsiogfs

==> Installing nemsiogfs-2.5.3-kfotberatd3ixj3n2ytwd563adbvsnme [129/298]
-- The Fortran compiler identification is Intel 2021.1.0.20240703
[...]
-- Could NOT find MPI_Fortran (missing: MPI_Fortran_LIB_NAMES MPI_Fortran_WORKS) 
CMake Error at /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/envs/rebuild-ue-oneapi-2024.2.1/install/oneapi/2024.2.1/cmake-3.27.9-jbb4rqk/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find MPI (missing: MPI_Fortran_FOUND Fortran)

gcc: fails to configure wgrib2

==> Installing wgrib2-3.6.0-dlxsiaknkfviiqz7hjuvriwasferbem4 [127/338]
-- The C compiler identification is GNU 12.4.0
[...]
-- The Fortran compiler identification is GNU 12.4.0
[...]
-- Found OpenMP_Fortran: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5") found components: Fortran 
[...]
-- Could NOT find MPI_C (missing: MPI_C_LIB_NAMES MPI_C_WORKS) 
-- Found MPI_Fortran: /opt/cray/pe/mpich/8.1.29/ofi/gnu/12.3/lib/libmpifort_gnu_123.so (found version "3.1") 
CMake Error at /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/envs/rebuild-ue-gcc-12.4/install/gcc/12.4.0/cmake-3.27.9-e7o4lre/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find MPI (missing: MPI_C_FOUND) (found version "3.1")

Similar failures (albeit in different packages).

The configuration for each of these envs is identical to that which built:

• glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/envs/ue-gcc-12.4
• /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/envs/ue-oneapi-2024.2.1

and those spack-stack envs were in place / being used prior to upgrading from SLES SP4 to SP6.

[rickgrubin-noaa]

Adding for some amount of completeness:

gaea-c6

gaea% cat /etc/os-release
NAME="SLES"
VERSION="15-SP6"
VERSION_ID="15.6"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"

and confirmed that the offending libraries at the base of this issue are indeed no longer present on the host.

A oneAPI@2024.2.1 / cray-mpich@8.1.32 stack build is chugging along, and has successfully built (among other packages) nemsiogfs and wgrib2 so far.

---

Edited to add: successful build of a oneAPI / cray-mpich stack on gaea-c6; this was expected, have been building it for UFS-WM testing in recent days.

[vanderwb]

@rickgrubin-noaa - I'm not sure why your rebuild are having issues with MPI Fortran support. Nothing should have changed as far as the Cray packages go. Do you have CMakeConfigureLog.yaml output from these configure tests you can share?

[rickgrubin-noaa]

@vanderwb I suspect that the following will be easier than uploading a somewhat unwieldy file.

gcc / wgrib2 configuration failure

• /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/cache/build_stage/spack-stage-wgrib2-3.6.0-dlxsiaknkfviiqz7hjuvriwasferbem4/spack-build-dlxsiak/CMakeFiles/CMakeConfigureLog.yaml

oneAPI / nemsiogfs configuration failure

• /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/cache/build_stage/spack-stage-nemsiogfs-2.5.3-kfotberatd3ixj3n2ytwd563adbvsnme/spack-build-kfotber/CMakeFiles/CMakeConfigureLog.yaml

[vanderwb]

Thanks @rickgrubin-noaa, but I cannot open those currently. Can you chmod -R a+rX /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/cache/build_stage, or if you prefer to keep it locked down generally, setfacl -R u:vanderwb:rX /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/cache/build_stage?

[rickgrubin-noaa]

Thanks @rickgrubin-noaa, but I cannot open those currently. Can you chmod -R a+rX /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/cache/build_stage, or if you prefer to keep it locked down generally, setfacl -R u:vanderwb:rX /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/cache/build_stage?

Done.

[vanderwb]

Thanks. I suspect the issue comes from build environment contamination as a result of your having our default module environment loaded. Here's the old wgrib2 build env:

[09:15] ~$ grep LOADEDMODULES /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/envs/ue-gcc-12.4/install/gcc/12.4.0/wgrib2-3.6.0-carni3h/.spack/spack-build-env.txt 
[09:17] ~$

and here is the new build env:

[09:16] $ grep LOADEDMODULES /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/cache/build_stage/spack-stage-wgrib2-3.6.0-dlxsiaknkfviiqz7hjuvriwasferbem4/spack-build-env.txt 
LOADEDMODULES=ncarenv/24.12:gcc/12.4.0:hdf5/1.12.3:netcdf/4.9.2:craype/2.7.31:ncarcompilers/1.0.0:libfabric/1.15.2.0:cray-mpich/8.1.29; export LOADEDMODULES

[rickgrubin-noaa]

Thanks. I suspect the issue comes from build environment contamination as a result of your having our default module environment loaded. Here's the old wgrib2 build env:

[09:15] ~$ grep LOADEDMODULES /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/envs/ue-gcc-12.4/install/gcc/12.4.0/wgrib2-3.6.0-carni3h/.spack/spack-build-env.txt 
[09:17] ~$

and here is the new build env:

[09:16] $ grep LOADEDMODULES /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/cache/build_stage/spack-stage-wgrib2-3.6.0-dlxsiaknkfviiqz7hjuvriwasferbem4/spack-build-env.txt 
LOADEDMODULES=ncarenv/24.12:gcc/12.4.0:hdf5/1.12.3:netcdf/4.9.2:craype/2.7.31:ncarcompilers/1.0.0:libfabric/1.15.2.0:cray-mpich/8.1.29; export LOADEDMODULES

Thanks for the analysis -- reconcretizing and rebuilding after module --force purge

[rickgrubin-noaa]

@jim-p-w please try:

derecho: /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.3/envs/rebuild-ue-gcc-12.4

A oneAPI v1.9.3 env (re)build is underway.