Spack sets RUNPATH instead of RPATH on latest Ubuntu LTS

[alalazo]

I tried to check the RPATH of a few applications installed with Spack on my Ubuntu 18.04 and found out that Spack was setting RUNPATH instead of RPATH.

Steps to reproduce the issue

I was able to reproduce the issue using a docker image, and it seems related to the Ubuntu version:

$ docker run -it --rm ubuntu:18.04 /bin/bash
root@ab5ae0326b89:~# apt update && apt upgrade
root@ab5ae0326b89:~# apt install gcc g++ git curl make python tar xz-utils
root@ab5ae0326b89:~# git clone https://github.com/spack/spack.git
root@ab5ae0326b89:~# . spack/share/spack/setup-env.sh
root@ab5ae0326b89:~# spack install pkgconf
==> Installing pkgconf
==> Using cached archive: /spack/var/spack/cache/pkgconf/pkgconf-1.4.2.tar.xz
==> Staging archive: /spack/var/spack/stage/pkgconf-1.4.2-gavqfquorv7kyfjbib5w3w5odhfkuggc/pkgconf-1.4.2.tar.xz
==> Created stage in /spack/var/spack/stage/pkgconf-1.4.2-gavqfquorv7kyfjbib5w3w5odhfkuggc
==> No patches needed for pkgconf
==> Building pkgconf [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed pkgconf
  Fetch: 0.00s.  Build: 7.17s.  Total: 7.17s.
[+] /spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/pkgconf-1.4.2-gavqfquorv7kyfjbib5w3w5odhfkuggc

Error Message

At this point, trying to check the dynamic section of the ELF executable:

root@ab5ae0326b89:~# objdump -x /spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/pkgconf-1.4.2-gavqfquorv7kyfjbib5w3w5odhfkuggc/bin/pkgconf
/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/pkgconf-1.4.2-gavqfquorv7kyfjbib5w3w5odhfkuggc/bin/pkgconf:     file format elf64-x86-64
/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/pkgconf-1.4.2-gavqfquorv7kyfjbib5w3w5odhfkuggc/bin/pkgconf
architecture: i386:x86-64, flags 0x00000150:
HAS_SYMS, DYNAMIC, D_PAGED
start address 0x0000000000003200

Program Header:
    PHDR off    0x0000000000000040 vaddr 0x0000000000000040 paddr 0x0000000000000040 align 2**3
         filesz 0x00000000000001f8 memsz 0x00000000000001f8 flags r--
  INTERP off    0x0000000000000238 vaddr 0x0000000000000238 paddr 0x0000000000000238 align 2**0
         filesz 0x000000000000001c memsz 0x000000000000001c flags r--
    LOAD off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**21
         filesz 0x00000000000075a0 memsz 0x00000000000075a0 flags r-x
    LOAD off    0x0000000000007bf0 vaddr 0x0000000000207bf0 paddr 0x0000000000207bf0 align 2**21
         filesz 0x0000000000000b10 memsz 0x0000000000000c78 flags rw-
 DYNAMIC off    0x0000000000007c10 vaddr 0x0000000000207c10 paddr 0x0000000000207c10 align 2**3
         filesz 0x0000000000000210 memsz 0x0000000000000210 flags rw-
    NOTE off    0x0000000000000254 vaddr 0x0000000000000254 paddr 0x0000000000000254 align 2**2
         filesz 0x0000000000000044 memsz 0x0000000000000044 flags r--
EH_FRAME off    0x0000000000006d58 vaddr 0x0000000000006d58 paddr 0x0000000000006d58 align 2**2
         filesz 0x0000000000000144 memsz 0x0000000000000144 flags r--
   STACK off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**4
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
   RELRO off    0x0000000000007bf0 vaddr 0x0000000000207bf0 paddr 0x0000000000207bf0 align 2**0
         filesz 0x0000000000000410 memsz 0x0000000000000410 flags r--

Dynamic Section:
  NEEDED               libpkgconf.so.3
  NEEDED               libc.so.6
  RUNPATH              /spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/pkgconf-1.4.2-gavqfquorv7kyfjbib5w3w5odhfkuggc/lib:/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/pkgconf-1.4.2-gavqfquorv7kyfjbib5w3w5odhfkuggc/lib64
  INIT                 0x0000000000001cf8
  FINI                 0x0000000000004f04
...

Additional information

Repeating the same procedure on the previous LTS (Ubuntu 16.04) gives instead the expected result:

root@aaf643ede7cc:/spack# objdump -x /spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v/bin/pkgconf

/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v/bin/pkgconf:     file format elf64-x86-64
/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v/bin/pkgconf
architecture: i386:x86-64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000402780

Program Header:
    PHDR off    0x0000000000000040 vaddr 0x0000000000400040 paddr 0x0000000000400040 align 2**3
         filesz 0x00000000000001f8 memsz 0x00000000000001f8 flags r-x
  INTERP off    0x0000000000000238 vaddr 0x0000000000400238 paddr 0x0000000000400238 align 2**0
         filesz 0x000000000000001c memsz 0x000000000000001c flags r--
    LOAD off    0x0000000000000000 vaddr 0x0000000000400000 paddr 0x0000000000400000 align 2**21
         filesz 0x0000000000007224 memsz 0x0000000000007224 flags r-x
    LOAD off    0x0000000000007df0 vaddr 0x0000000000607df0 paddr 0x0000000000607df0 align 2**21
         filesz 0x00000000000003fc memsz 0x0000000000000578 flags rw-
 DYNAMIC off    0x0000000000007e08 vaddr 0x0000000000607e08 paddr 0x0000000000607e08 align 2**3
         filesz 0x00000000000001f0 memsz 0x00000000000001f0 flags rw-
    NOTE off    0x0000000000000254 vaddr 0x0000000000400254 paddr 0x0000000000400254 align 2**2
         filesz 0x0000000000000044 memsz 0x0000000000000044 flags r--
EH_FRAME off    0x0000000000006a10 vaddr 0x0000000000406a10 paddr 0x0000000000406a10 align 2**2
         filesz 0x000000000000013c memsz 0x000000000000013c flags r--
   STACK off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**4
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
   RELRO off    0x0000000000007df0 vaddr 0x0000000000607df0 paddr 0x0000000000607df0 align 2**0
         filesz 0x0000000000000210 memsz 0x0000000000000210 flags r--

Dynamic Section:
  NEEDED               libpkgconf.so.3
  NEEDED               libc.so.6
  RPATH                /spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v/lib:/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v/lib64
...

so this seems to be related in some way to the version of the OS.

Information on your system

The bug can be reproduced on the latest Ubuntu LTS docker image (ubuntu:18.04), while it's not there in the previous LTS (ubuntu:16.04).

[tgamblin]

I think to fix this we need to both strip --enable-new-dtags and add --disable-new-dtags in the compiler wrappers to get it right. At some point binutils changed the default behavior of -rpath in ld, and I think it is starting to make its way into newer releases.

@alalazo: do you want to give this a shot?

[alalazo]

@tgamblin Ok, I'll look into that.

[rmsds]

It might have trickled down from Debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=835859

Some people there also think it's a bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859732

[neilflood]

My understanding is that RPATH is now deprecated, in favour of RUNPATH.
http://man7.org/linux/man-pages/man8/ld.so.8.html

I think this is a good thing, because RPATH is set like concrete and can never be over-ridden, whereas RUNPATH happens after LD_LIBRARY_PATH, thus allowing at least the possibility of over-riding it when a different .so version is required, without having to re-link everything which uses it. Which is kind of the point of dynamic linking.

I would be a bit dismayed if Spack started somehow enforcing RPATH.

[tgamblin]

We should make this configurable so users can use one or the other, but we shouldn’t ever allow both. They don’t mix well. Fermilab users like @gartung want Spack to use RUNPATH (they want to be able to load modules to override), but LLNL and other HPC sites are very strongly on the RPATH side.

I realize that the ld.so manual says that RPATH is “deprecated”, but that’s been there for over 10 years and the functionality isn’t going away. We remind the glibc maintainers of this via our contacts at red hat. The trouble with RUNPATH and the reason we won’t use it is that it’s search semantics are fundamentally broken. It isn’t just a difference in precedence. Whereas an RPATH’d binary will search its RPATH, it’s loading object’s RPATH, all the way up to the executable if need be, RUNPATH gives up after searching the RUNPATH of the object itself. This means that the executable can’t enforce consistent link semantics, and that if libraries are broken or list incomplete paths, you can’t fix that with your executable’s RUNPATH like you can with RPATH. You HAVE to use LD_LIBRARY_PATH, which puts the burden on the user.

For libraries like OpenMPI, which come with only partial RPATHs and expect the mpirun script to “finish” the link with LD_LIBRARY_PATH, this means we can’t build an executable that will work properly without help from the user (or the launch script). We use multiple MPIs at LLNL, and we don’t want the users to have to remember which one they built with in order to run with srun under SLURM. So we force RPATH at LLNL.

Worse, RPATHs are ignored if there are ANY RUNPATHs in binaries within a DAG. We actually have an ld wrapper that eats the new dtags flag, because of insanity like this.

There are more reasons why RUNPATH is broken here. In that case it breaks plugin loading in Qt.

Finally, it may seem nice that LD_LIBRARY_PATH can “override” your RUNPATH, because you can experiment, and that is great for testing, but LD_LIBRARY_PATH is it’s own kind of evil in practice. It’s a global setting for your environment, so all the executables in an environment will use it. This means that if you have spack things built from a while ago with old versions of libraries, and if you have newer versions in LD_LIBRARY_PATH, running the old things will break. You can no longer have a workflow where two applications have different versions of common dependencies. Users frequently botch this somehow — when a long-forgotten LD_LIBRARY_PATH breaks some new program, and we end up getting support calls about the unexpected behavior. We like RPATH because we trust the builder of the application, much more than the user, to know the dependencies.

See also here and here.

At any rate, though, some sites do like RUNPATH, and if we’re doing this it should be settable. @alalazo: do you mind adding a config option to pick which we use? That should suffice initially but for reasons mentioned above we should probably eventually wnforce the no mixing rule by preventing users from linking RUNPATH’d libraries into RPATH DAGs.

[neilflood]

Ah - I think I stand corrected - all good points.

I was aware of the alternative views on LD_LIBRARY_PATH, but had not known of the deeper differences between RPATH and RUNPATH. I probably need to be more knowledgable. Thank you for the more comprehensive explanation.

As it happens, on my SLES-12 system, Spack ends up setting RUNPATH only.

[gartung]

@tgamblin I use $ORIGIN when making binaries with relative paths. I think @hegner mentioned using RUNPATH so that LD_LIBRARY_PATH could be used to override as needed.

[gartung]

PR #909 and issue #639 are related to this

[neilflood]

@alalazo
I tested #9168 on my SLES-12 cluster, and it behaves exactly as desired. The config switch makes it generate RPATH or RUNPATH, as per selection.