Users can decide whether to set RPATH or RUNPATH #9168

alalazo · 2018-09-03T13:18:35Z

fixes #9160
fixes #909
fixes #639

This PR adds a new entry in config.yaml:

config:
   shared_linking: 'rpath'

If this variable is set to rpath (the default) Spack will set RPATH in ELF binaries. If set to runpath it will set RUNPATH.

lib/spack/env/cc

tgamblin

Looks like a good start -- see my comment below on ld vs ccld.

The other thing that should be handled here is if the build provides --enable-new-dtags. We need to strip that, because I dunno what happens if both --enable-new-dtags and --disable-new-dtags are provided together on the link line. We want disable to be all that is left.

lib/spack/env/cc

alalazo · 2018-09-05T15:04:52Z

Ready for a second review

alalazo · 2018-09-05T15:14:46Z

lib/spack/env/cc

@@ -325,6 +343,8 @@ while [ -n "$1" ]; do
                    die "-Wl,-rpath was not followed by -Wl,*"
                fi
                rp="${arg#-Wl,}"
+            elif [[ "$arg" = --enable-new-dtags ]] ; then


Reading bash is not my best skill, but I think we are currently not treating NAG the same as the other compilers:

RPATHs are not collected in rp as there's no condition that matches -Wl,-Wl,,-rpath,,XXX

Every -Wl,-Wl,,-rpath,,XXX will be appended to other_args

Am I misreading something? In case: do we want to fix this here or, having the issue a minimal impact, can we defer it to a later PR?

I wish I understood the NAG linker argument syntax there -- I wouldn't mind actually fixing this here if you're up for it. I just don't have a NAG compiler to test on. I looked at the NAG manual and it doesn't seem to indicate that we need two -Wl,'s. Do you know who we could ask? I suspect we don't have many NAG users.

I think @adamjstewart is our best bet. As far as I remember he had to install NAG in his previous job.

NAG calls gcc for linking, which calls ld. Therefore, when we specify a flag with a single prefix -Wl for NAG, it will pass it to gcc, which won't understand an ld-specific flag. So, we have to wrap an ld-specific flag twice: the first level will be unwrapped by NAG, the second one - by gcc.

skosukhin · 2018-09-05T18:40:57Z

I was going to start working on implementing something like this. I've stumbled upon this issue on a Cray machine. BTW, the colleagues seem to have implemented something addressing this problem: https://easybuild.readthedocs.io/en/latest/RPATH-support.html?highlight=disable-dtags#implementation
I didn't look at their code yet but maybe they have something for us to borrow :)

alalazo · 2018-09-06T08:28:00Z

@skosukhin

I was going to start working on implementing something like this. I've stumbled upon this issue on a Cray machine.

Can you check if this PR solves your issue?

I didn't look at their code yet but maybe they have something for us to borrow :)

I took a look at that while preparing this PR. It seems to me we are doing the same things conceptually, except that we are also taking care of the reordering of flags as explained here:

spack/lib/spack/env/cc

Lines 279 to 295 in b39ff56

    
           # 
        
           # Parse the command line arguments. 
        
           # 
        
           # We extract -L, -I, and -Wl,-rpath arguments from the command line and 
        
           # recombine them with Spack arguments later.  We parse these out so that 
        
           # we can make sure that system paths come last, that package arguments 
        
           # come first, and that Spack arguments are injected properly. 
        
           # 
        
           # All other arguments, including -l arguments, are treated as 
        
           # 'other_args' and left in their original order.  This ensures that 
        
           # --start-group, --end-group, and other order-sensitive flags continue to 
        
           # work as the caller expects. 
        
           # 
        
           # The libs variable is initialized here for completeness, and it is also 
        
           # used later to inject flags supplied via `ldlibs` on the command 
        
           # line. These come into the wrappers via SPACK_LDLIBS. 
        
           #

To have an easy reference, Easybuild's code is here.

alalazo · 2018-09-07T06:32:44Z

@scheibelp @tgamblin @becker33

alalazo · 2018-09-12T10:00:02Z

@gartung If you are interested, this branch should permit you to choose between RPATH and RUNPATH.

gartung · 2018-09-12T14:54:26Z

@alalazo Thanks for pointing this out. I prefer to use RPATH myself but others in HEP community have expressed interest in using RUNPATH.

tgamblin

@alalazo: some general comments -- I think the code could be cleaned up a little if it didn't use 2 variables. I also think fixing NAG (if we're missing RPATHs for it) would be good.

@skosukhin: do you know how frequently people need to do dynamic links with NAG? I'm curious how much RPATHs are used with NAG.

tgamblin · 2018-09-13T05:32:21Z

etc/spack/defaults/config.yaml

+  # https://gms.tf/ld_library_path-considered-harmful.html
+  #
+  # for more information on the subject.
+  rpath: true


Let's not make this the first entry in the default config.yaml, as I suspect it's not the first thing most users want to look at. The comment is a bit obscure -- how about:

# Control whether Spack embeds RPATH or RUNPATH attributes in ELF binaries # so that they can find their dependencies. Has no effect on macOS. # # There are two options: # # 1. rpath: use RPATH: force --disable-new-tags flag to be passed to ld # 2. runpath: use RUNPATH: force --enable-new-tags flag to be passed to ld # # RPATH search paths have higher precedence than LD_LIBRARY_PATH # and ld.so will search for libraries in transitive RPATHs of # parent objects. # # RUNPATH search paths have lower precedence than LD_LIBRARRY_PATH, # and ld.so will ONLY search for dependencies in the RUNPATH of # the loading object. # # DO NOT MIX these within the same install tree. See the Spack # documentation for details.

And make the two possible options rpath and runpath (enforced by schema). This at least gives us room to choose other values in the future.

I think the links there, and maybe also https://bugs.launchpad.net/ubuntu/+source/eglibc/+bug/1253638, could be included in the Spack documentation. Maybe some of the text above could be too... the only important parts for hte config file are really the first couple sentences, the options, and the instructions not to mix them (since if you do, RUNPATH breaks everything).

See the Spack documentation for details.

Please point to a specific part of the documentation

Well it doesn't exist yet, but the idea was that he'd write it :).

I meant that the comment should have a more specific notion on where I should look within the documentation. Having to look through all of it (or using the search function) just seems needlessly mean to the user ;)

tgamblin · 2018-09-13T05:36:33Z

lib/spack/env/cc

+# opposite stance that Spack does):
+#
+# http://blog.qt.io/blog/2011/10/28/rpath-and-runpath/
+# https://wiki.debian.org/RpathIssue


This whole account is from the point of view of a single-prefix, system package manager. Not one that has to maintain relationships between combinatorial sets of modules... so it's a completely different link situation.

Agree completely. I put the links there more as reference for future developers to get some info on RPATH vs. RUNPATH rather than to explain Spack's philosophy towards linking. Let me know if you want me to remove them, or just rephrase the comment.

tgamblin · 2018-09-13T05:50:26Z

lib/spack/env/cc

@@ -51,6 +51,9 @@ parameters=(
    SPACK_CXX_RPATH_ARG
    SPACK_F77_RPATH_ARG
    SPACK_FC_RPATH_ARG
+    SPACK_DTAGS_TO_ENABLE
+    SPACK_DTAGS_TO_DISABLE


This feels very specific to me, and it feels kludgy to have two variables... Could we generalize this? it actually seems like we could do it more cleanly by just adding either --enable-new-dtags/--disable-new-dtags to SPACK_LDFLAGS in the build environment, then a single more general option like SPACK_STRIP_LDFLAGS, which would strip a set of flags from the link line - not just this one. That mechanism would be a similar amount of code, but there are other places where it would be useful -- For example, I've seen users want to do things like strip fPIC for certain compilers on certain platforms, and we could (in future PRs) allow packages, compilers, or platforms to set some of these things to ease porting. What do you think?

Fine to generalize the mechanics here. I'll work towards that.

This feels very specific to me, and it feels kludgy to have two variables...

Eheh, I wouldn't say kludgy, but very specific yes. From my point of view we are broadening the scope of this PR the third time, more or less along the lines:

Make sure Spack sets RPATH as it says it does

Give the possibility to users to select between RPATH and RUNPATH

Extend the mechanism by which we strip flags to ld to prepare for functionality that could be added later

To achieve point 2. two variables seemed the right thing to have.

For example, I've seen users want to do things like strip fPIC for certain compilers on certain platforms, and we could

I think fpic shouldn't be part of SPACK_STRIP_LDFLAGS but maybe SPACK_STRIP_<LANG>FLAGS. Anyhow: I agree with the overall idea of intercepting flags we don't want to be in the compiler wrapper and strip them.

skosukhin · 2018-09-13T07:22:04Z

@tgamblin, I would say that NAG users need dynamic linking as often as users of any other compiler. I wrote a short text on dynamic linking and Spack explaining why: https://gitlab.dkrz.de/m300488/icon-bootstrap#dynamic-linking. What I have not written there is that in order to support static linking properly, we need the method libs of the packages to return not only their own (static) libs but the libs of their dependencies too.

lib/spack/env/cc

skosukhin · 2018-09-13T20:07:56Z

@alalazo if I understand correctly, you want to add -Wl, prefixes to the linking flags assuming they are specified in the form understandable by ld. What is the plan on the compiler linking flags like -pthread, -openmp (and probably other) that need to be passed in the ccld mode but without the prefix? And what prefix will be added when I have a mixed toolchain like NAG+gcc, which require different prefixes (and different flags)?

alalazo · 2018-09-13T20:21:29Z

if I understand correctly, you want to add -Wl, prefixes to the linking flags assuming they are specified in the form understandable by ld.

That's how I interpreted the request in #9168 (comment). The mechanism should work only for the flags that are injected in the compiler wrapper.

What is the plan on the compiler linking flags like -pthread, -openmp (and probably other) that need to be passed in the ccld mode but without the prefix?

Can't they be injected with SPACK_<LANG>FLAGS?

And what prefix will be added when I have a mixed toolchain like NAG+gcc, which require different prefixes (and different flags)?

This is a case I effectively didn't consider. If we want to support that we probably need to split linker_arg (a variable that usually contains the string -Wl,) into the 4 languages supported by Spack's compilers - so for instance cc_linker_arg, fc_linker_arg, etc. What do you think?

skosukhin · 2018-09-14T11:28:25Z

@alalazo I think we should start with making it possible to override every property of the Compiler class for every compiler (C, C++, Fortran) in the compilers.yaml. This would fix a problem with pic flags right away, for example. Regarding the per-compiler linking flags, we could wait with this: for now, every compiler could prepend its own prefix to the common linking flags when in the ccld mode. After that we can introduce per-compiler linker flags, which if specified, would override this default behaviour.

And a small remark. We can replace this:

spack/lib/spack/spack/compiler.py

Lines 129 to 136 in 0f053e0

    
           @property 
        
           def fc_rpath_arg(self): 
        
               return '-Wl,-rpath,' 
        
           @property 
        
           def linker_arg(self): 
        
               """Flag that need to be used to pass an argument to the linker.""" 
        
               return '-Wl,'

With this (that's how, for example, libtool does this):

@property
def fc_rpath_arg(self):
    return self.linker_arg + '-rpath ' + self.linker_arg

@property
def linker_arg(self):
    """Flag that need to be used to pass an argument to the linker."""
    return '-Wl,'

For NAG, for example, that would result in prepending: -Wl,-Wl,,-rpath -Wl,-Wl,, to the path to a library.

alalazo · 2018-09-16T06:20:21Z

@skosukhin @tgamblin To summarize, I see many sensible needs being discussed here:

Select between RPATH and RUNPATH
Being able to strip arbitrary flags from the command line at compile time
Being able to override in compilers.yaml any property of Compiler

As @neilflood confirms that this PR already achieves 1. (and closes the 3 issues in the description) I'd propose the following:

Remove ba0d18b and 0f053e0 from this PR (these 2 commits are going towards 2., not 1., and are not user-facing)
Open new issues / PRs to track 2. and 3. and work them out immediately after this PR

@tgamblin Would that be fine with you, or do you prefer to work all three issues here? What I would like to avoid is having a PR that stays idle for a long time and becomes bigger and bigger...

alalazo · 2019-10-09T06:50:55Z

@tgamblin @becker33 There's one thing that we might need to discuss: this setting doesn't enter the hash. We might have to decide on a policy, or extend this feature, when we'll start distributing package binaries.

alalazo · 2019-10-09T06:52:15Z

This PR also lacks documentation. As soon as you consider the design of the feature complete I'll start working on adding that.

tgamblin · 2019-10-09T07:47:13Z

@alalazo: on the hash issue, I think this should probably not have to affect the hash. I say that because we should be able to decide at binary install time whether we want to relocate using RPATH or RUNPATH. Last I checked patchelf had an option to choose one or the other; I am not sure whether it actually works so you should probably check that. 😄

alalazo · 2019-10-09T07:48:50Z

@tgamblin I wasn't aware that you could switch from one to the other when relocating binaries. I'll try that out and report here.

alalazo · 2019-10-18T14:51:33Z

@tgamblin @becker33 Just tried out manually and patchelf v0.9 does work. It has the behavior shown here though - basically we need to --remove-rpaths before doing anything else if we need to switch from RPATH to RUNPATH or vice-versa.

fixes spack#9160

As noted by Todd, ccld needs a prefix (e.g. `-Wl,`) before the linker argument, while ld does not. This prefix changes in principle depending on the compiler. Using (or not) `--disable-new-dtags` is now OS dependent.

If that flag is present in the options passed to the compiler or the linker the wrapper will strip it explicitly.

A new option as been added to config.yaml to select between RPATH and RUNPATH. The default configuration file contains links that point to relevant discussions on the topic.

The option is now named 'shared_linking' and can take two options: 'rpath' and 'runpath'.

Add a new entry in `config.yaml`: config: shared_linking: 'rpath' If this variable is set to `rpath` (the default) Spack will set RPATH in ELF binaries. If set to `runpath` it will set RUNPATH. Details: * Spack cc wrapper explicitly adds `--disable-new-dtags` when linking * cc wrapper also strips `--enable-new-dtags` from the compile line when disabling (and vice versa) * We specifically do *not* add any dtags flags on macOS, which uses Mach-O binaries, not ELF, so there's no RUNPATH)

alalazo added bug Something isn't working don't-merge-yet labels Sep 3, 2018

alalazo requested review from tgamblin, scheibelp, adamjstewart and becker33 September 3, 2018 13:18

alalazo commented Sep 3, 2018

View reviewed changes

lib/spack/env/cc Outdated Show resolved Hide resolved

tgamblin requested changes Sep 3, 2018

View reviewed changes

lib/spack/env/cc Outdated Show resolved Hide resolved

alalazo commented Sep 5, 2018

View reviewed changes

alalazo removed the don't-merge-yet label Sep 6, 2018

alalazo mentioned this pull request Sep 11, 2018

Spack sets RUNPATH instead of RPATH on latest Ubuntu LTS #9160

Closed

alalazo changed the title ~~Explicitly disable new dtags in compiler wrappers~~ Users can decide whether to set RPATH or RUNPATH Sep 12, 2018

alalazo force-pushed the fixes/disable_new_dtags branch from b39ff56 to 3a10b6d Compare September 12, 2018 09:22

adamjstewart added the rpath label Sep 12, 2018

tgamblin requested changes Sep 13, 2018

View reviewed changes

alalazo commented Sep 13, 2018

View reviewed changes

lib/spack/env/cc Outdated Show resolved Hide resolved

alalazo force-pushed the fixes/disable_new_dtags branch from 0f053e0 to 04fbf82 Compare October 1, 2018 15:28

alalazo force-pushed the fixes/disable_new_dtags branch from 04fbf82 to a2a5c6d Compare November 12, 2018 11:01

alalazo added the ready label Nov 12, 2018

alalazo force-pushed the fixes/disable_new_dtags branch from f7d597f to f354176 Compare October 18, 2019 14:13

tgamblin moved this from In Progress to In Review in Spack v0.13.0 release Oct 22, 2019

becker33 approved these changes Oct 22, 2019

View reviewed changes

tgamblin assigned alalazo Oct 22, 2019

tgamblin approved these changes Oct 23, 2019

View reviewed changes

alalazo added 9 commits October 23, 2019 11:28

Spack CC wrapper explicitly disables new dtags when linking

a95cff3

fixes spack#9160

Fixed unit tests + added regression test for spack#9160

df6992a

Differentiate flags to be used depending on compiler and OS.

3aeb024

As noted by Todd, ccld needs a prefix (e.g. `-Wl,`) before the linker argument, while ld does not. This prefix changes in principle depending on the compiler. Using (or not) `--disable-new-dtags` is now OS dependent.

Strip --enable-new-dflags within the compiler wrapper

e1e4c5e

If that flag is present in the options passed to the compiler or the linker the wrapper will strip it explicitly.

Users can decide whether to have RPATH or RUNPATH

be3756f

A new option as been added to config.yaml to select between RPATH and RUNPATH. The default configuration file contains links that point to relevant discussions on the topic.

Changed the name of the configure option in config.yaml

68104e0

The option is now named 'shared_linking' and can take two options: 'rpath' and 'runpath'.

Removing the unwanted dtag also in case of '-Xlinker <dtag>'

533be8a

Renamed a few variables for improved readability

72df81c

Added a unit test to raise coverage of build_environment.py

f89a951

alalazo force-pushed the fixes/disable_new_dtags branch from f354176 to f89a951 Compare October 23, 2019 10:23

tgamblin force-pushed the fixes/disable_new_dtags branch from d00ffee to 5a4c6d4 Compare October 23, 2019 17:33

bugfix: do not add dtags arguments on mac os

aa27fad

tgamblin force-pushed the fixes/disable_new_dtags branch from 5a4c6d4 to aa27fad Compare October 23, 2019 17:40

Shortened comments in config.yaml. Moved most part of it to the docs.

fbad075

tgamblin merged commit b29eb42 into spack:develop Oct 23, 2019

Spack v0.13.0 release automation moved this from In Review to Done Oct 23, 2019

alalazo deleted the fixes/disable_new_dtags branch October 23, 2019 20:41

adamjstewart mentioned this pull request Mar 13, 2020

Language-specific PIC flags #15474

Merged

adamjstewart mentioned this pull request Apr 11, 2020

Install openmpi with NAG compiler #15776

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Users can decide whether to set RPATH or RUNPATH #9168

Users can decide whether to set RPATH or RUNPATH #9168

alalazo commented Sep 3, 2018 •

edited

tgamblin left a comment

alalazo commented Sep 5, 2018

alalazo Sep 5, 2018

tgamblin Sep 13, 2018

alalazo Sep 13, 2018

skosukhin Sep 13, 2018

skosukhin commented Sep 5, 2018

alalazo commented Sep 6, 2018

alalazo commented Sep 7, 2018

alalazo commented Sep 12, 2018

gartung commented Sep 12, 2018

tgamblin left a comment

tgamblin Sep 13, 2018

healther Sep 13, 2018

tgamblin Sep 13, 2018

healther Sep 13, 2018

tgamblin Sep 13, 2018

alalazo Sep 13, 2018

tgamblin Sep 13, 2018

alalazo Sep 13, 2018 •

edited

skosukhin commented Sep 13, 2018

skosukhin commented Sep 13, 2018

alalazo commented Sep 13, 2018 •

edited

skosukhin commented Sep 14, 2018

alalazo commented Sep 16, 2018

alalazo commented Oct 9, 2019

alalazo commented Oct 9, 2019

tgamblin commented Oct 9, 2019

alalazo commented Oct 9, 2019

alalazo commented Oct 18, 2019

Users can decide whether to set RPATH or RUNPATH #9168

Users can decide whether to set RPATH or RUNPATH #9168

Conversation

alalazo commented Sep 3, 2018 • edited

tgamblin left a comment

Choose a reason for hiding this comment

alalazo commented Sep 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skosukhin commented Sep 5, 2018

alalazo commented Sep 6, 2018

alalazo commented Sep 7, 2018

alalazo commented Sep 12, 2018

gartung commented Sep 12, 2018

tgamblin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alalazo Sep 13, 2018 • edited

Choose a reason for hiding this comment

skosukhin commented Sep 13, 2018

skosukhin commented Sep 13, 2018

alalazo commented Sep 13, 2018 • edited

skosukhin commented Sep 14, 2018

alalazo commented Sep 16, 2018

alalazo commented Oct 9, 2019

alalazo commented Oct 9, 2019

tgamblin commented Oct 9, 2019

alalazo commented Oct 9, 2019

alalazo commented Oct 18, 2019

alalazo commented Sep 3, 2018 •

edited

alalazo Sep 13, 2018 •

edited

alalazo commented Sep 13, 2018 •

edited