Linker fails due to missing -rpath

[jh66637]

I've been trying to find the problem myself for quite a while, but I can't get to the bottom of it. If someone could take a look, I would be very grateful.

I encounter linking issues on a cluster with Spack using GCC and OpenMPI. After deal.II compiled successfully, I get the linker error:

fatal error: /gpfs/opt/sw/zen/spack-0.19.0/opt/spack/linux-almalinux8-zen2/gcc-9.5.0/hwloc-2.8.0-s7uap66oudz2vyjwxvjnyo46zahht3x/lib:pread failed: Is a directory

I attached the log file for the minimal set of dependencies, and I am happy to provide any additional information that might be helpful.
While the log is produced with GCC9, I encounter the same problems with GCC12. I tried on Intel Skylake as well as AMD Zen architecture and observe the same behavior.

What stands out to me is that I am loading openmpi-4.1.4 and the recognized MPI_VERSION = 3.1. However, I don't know if this is related.

detailed.log

[jh66637]

The problem is in the linker flags. Changing -Wl, to -Wl,-rpath -Wl, in a lot of places seems to be a step into the right direction.

[drwells]

I'm not sure if this is something wrong with deal.II or an upstream package - the problem occurs somewhere in hwloc which we do not directly use (I think Kokkos uses it). pread() is passed a directory instead of a file for something in hwloc which makes me think a dependency has something messed up.

[jh66637]

@drwells Thank you for taking a look. I am currently testing some things, which is quite easy now that deal.II does not have to be recompiled every time. I figured that g++ is used as a compiler even though MPI is found. Fixing the compiler explicitly to the mpic++ binary seems to resolve the problem. Btw. I did not use any TPL except MPI to reproduce the issue.

I still have to wait until everything compiled, and I am able to check if the tests are running.

If everything works: Should I close the issue or keep it open because mpic++ is not used even though MPI was found?

[tamiko]

@drwells This looks like a -Wl,--rpath being stripped from a -Wl,--rpath -Wl,.../lib.
We should detect such occurences and escape them correctly by now, but somehow this doesn't seem to happen.

Correction - I did not implemented this logic for -Wl, only for -Xlinker... pull request incoming.

[drwells]

I must be missing something. The message is fatal error: /gpfs/[...]/hwloc-2.8.0-[...]/lib:pread failed: Is a directory. Doesn't that indicate a failure in calling pread(), not that pread is a directory we failed to read from?

[tamiko]

@drwells It means that we supplied a plain -Wl,/.../lib to the linker. For examle:

# g++ -Wl,/usr test.cc 
/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: error: /usr: read: Is a directory
collect2: error: ld returned 1 exit status

(The :pread in above output is confusing.)

[jh66637]

@tamiko @drwells Tanks you for the big help.
I can test the behavior of #15257 on mentioned system, tomorrow.

[jh66637]

Fixes the issue for me. Thank you for the help.