Setting LD_LIBRARY_PATH to be "helpful" considered harmful

[hartzell]

(with apologies to Djikstra and everyone else who's recycled that title meme)

TL;DR: the library that one of my spack binaries uses depends on what other spack packages I've module load-ed. YIKES. See also #3926.

I was trying to understand why @JusticeForMikeBrown was having trouble building bowtie2 (see #3950) when I've built it successfully with gcc@4.x.y.

His problem with gcc@4.x.y was zlib related; I checked the package and noticed that it doesn't have a dependency on zlib. Perhaps it should, I thought. Wonder what zlib my "production" copy was linked against?

$ ldd bowtie2-align-l  | grep libz
	libz.so.1 => /blah/spack/v0.0.8/opt/spack/linux-centos7-x86_64/gcc-5.4.0/zlib-1.2.11-ec535e2ikkpl7hd4y454t3yydjqorja6/lib/libz.so.1 (0x00002aaaaaf32000)

That surprised me, because there's no zlib dependency in the package.

Sure enough, it's because I have something else module load-ed that has the side effect of adding zlib's directory to LD_LIBRARY_PATH.

$ (unset LD_LIBRARY_PATH; ldd bowtie2-align-l) | grep libz
	libz.so.1 => /lib64/libz.so.1 (0x00002aaaaaf2f000)

My "newer" version of CentOS has a /lib64/libz.so.1 that includes gzbuffer (nm didn't help, library's stripped...):

$ strings /lib64/libz.so.1 | grep buffer
gzbuffer
buffer error

so it (probably) works for me either way.

But imagine if there were two versions of a library (perhaps something mathematical) that give different results. Now you have a program giving different results depending on what other Spack applications are also loaded.

THAT would be fun to track down (assuming you even noticed...).

W.R.T. the main problem, bowtie2 should probably have a dependency on a new-ish version of zlib, but stuff like this is why LD_LIBRARY_PATH is a slippery tool to reach for.

I'll argue that this kind of unpredictability is a bigger negative than being helpful and always setting LD_LIBRARY_PATH. This comment in the docs isn't actually correct:

Spack avoids library misconfiguration by using RPATH to link dependencies. When a user links a library or runs a program, it is tied to the dependencies it was built with, so there is no need to manipulate LD_LIBRARY_PATH at runtime.

clipped from here

What would happen if LD_LIBRARY_PATH became opt-in, packages that need it specify it in their package definitions?

Looking at the list of cases where RPATH support doesn't work, it seems like 1) is not relevant (I think it's referring to PERL5LIB, etc...) and 3) are simply bugs. That leaves 2), python extensions. Is RPATH unworkable there or just not yet working?

[adamjstewart]

I agree that the real problem here is with the missing zlib dependency for bowtie2. This is a great argument for why we need to add dependencies for everything we can, even if we expect it to be installed on every operating system ever.

This comment in the docs isn't actually correct:

Spack avoids library misconfiguration by using RPATH to link dependencies. When a user links a library or runs a program, it is tied to the dependencies it was built with, so there is no need to manipulate LD_LIBRARY_PATH at runtime.

What's incorrect about that comment? As long as the dependency is stated, Spack will use RPATHs, making LD_LIBRARY_PATH unnecessary. We don't RPATH things in /lib, /usr/lib64, /usr/local/lib, etc. for the reasons stated in #3910, #2083, and the issues they fix. Basically, if -Wl,-rpath, is used with a system location, it can override Spack-built dependencies if it comes first. This is true for any directory that contains multiple installations, not just /, /usr/, and /usr/local.

1. is not relevant (I think it's referring to PERL5LIB, etc...)

Yes, it's referring to things like PERL5LIB and PYTHONPATH. Important, but not relevant to what you're worried about.

That leaves 2), python extensions. Is RPATH unworkable there or just not yet working?

Unfortunately, Python doesn't really have any way of RPATHing dependencies. I don't know the full details, but @wscullin or @citibeth can explain more if you're interested.

What would happen if LD_LIBRARY_PATH became opt-in, packages that need it specify it in their package definitions?

So there's two ways to think about this. You're thinking about it from the point of view of someone using the packages you installed. Yes, some packages don't work unless you have LD_LIBRARY_PATH set properly, but that's not the only reason our module files add it. Think about another scenario, in which you install some libraries like MPI and your users build their own applications outside of Spack that link to your MPI libraries. If they don't know about -Wl,-rpath, their applications won't work unless LD_LIBRARY_PATH is set properly. Maybe we should add that use case to the documentation as well.

But, if you want it to be opt-in, you can do that. All you have to do is create a modules.yaml that looks like:

modules:
  lmod:
    all:
      filter:
        environment_blacklist: 'LD_LIBRARY_PATH'
    bowtie2:
      environment:
        LD_LIBRARY_PATH: '$prefix/lib'

I haven't tested this, but it should work. See Modules and especially Module Configuration Tutorial for details.

[hartzell]

[edited, jumping back to the top of my string of replies to @adamjstewart to make sure I'm not coming across a ranting maniac]

@adamjstewart made a bunch of points in the comment above and I've responded to them one by one. I really don't see the value in this (even in automagically helping a set of programmers) and it does bother me. I know it's not going to be the downfall of the known universe or Spack or ....

Hopefully the points (arguments?) that follow won't overshadow how valuable all of this is.

[end of edit]

---

I agree that the real problem here is with the missing zlib dependency for bowtie2.

I think that will fix bowtie2's problem with the older gcc's. I think that there is [perhaps also] a different issue with gcc@6.x.y.

[hartzell]

This is a great argument for why we need to add dependencies for everything we can, even if we expect it to be installed on every operating system ever.

So you're suggesting that every time someone adds a new library to spack that "someone" needs to audit every package in the tree that might use it and update them?

If it were necessary to set LD_LIBRARY_PATH then perhaps that would be reasonable, but since Spack did/does the [hard] work of using RPATH everywhere it can then the only justification is "in case someone wants to build against this library from outside of Spack.

An alternative would be to provide some documentation about using Spack libraries in non-Spack projects.

[hartzell]

What's incorrect about that comment?

Two things:

1. packages that use RPATH are only safe until someone adds a new library to Spack for them to trip over (e.g. bowtie2). The fact that bowtie2 quietly depends on the system's zlib is only a bug (for those of us with modern-ish zlibs) because Spack sets LD_LIBRARY_PATH (though the quoted passage strongly suggests that it doesn't). What happens when someone adds an updated copy of libc to the tree and everything in my world automagically picks it up.

   Spack includes OpenSSL and I recently skimmed past something where someone (@citibeth?) mentioned that Spack's best practice was for people to hardwire their system OpenSSL into their packages.yaml to avoid security surprises. If I look at the package for the the software I want (perhaps something like nginx or the shiny server or ... where I need to start it as root) and it doesn't use OpenSSL, I might think that I was safe.

2. This bit: "so there is no need to manipulate LD_LIBRARY_PATH at runtime".

   Because apparently there is a need.

3. (What are the two most common bugs in computer programs? Off by one errors....)

   That passage gave me false comfort. I consider LD_LIBRARY_PATH slippery, dangerous, sloppy, Perl coders (and perhaps others) refer to this sort of thing as a "code smell". I know that software's full of compromises and you all have helped me get things into the tree that are less than perfect (as always, thank you!). If a package won't work without it and the package is important then so be it.

   But one of the things that drew me to Spack was it being tight and clean, being nicely self contained/immune-to-its-neighbors and being able to put together a set of tools that aren't a hazard to the rest of the community. The thing that I've replaced was an unprincipled mess of spaghetti and I was proud (by association) that I had found a better solution.

   I think that's why all of this is so surprising to me.

[hartzell]

Python

Spack breaks system utilities.

At first I thought that this was a PYTHONPATH thing, but it's LD_LIBRARY_PATH.

I understand that Python apps are a separate case (I wonder if the other dynamic languages are too, we just haven't noticed yet?), but....

$ yum --version
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named yum

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, Apr 15 2017, 18:37:03)
[GCC 5.4.0]

If you cannot solve this problem yourself, please go to
the yum faq at:
  http://yum.baseurl.org/wiki/Faq


$ unset LD_LIBRARY_PATH
$ yum --version
3.4.3
  Installed: rpm-4.11.3-17.el7.x86_64 at 2017-02-21 22:42
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2015-11-20 11:15
  Committed: Florian Festi <ffesti@redhat.com> at 2015-09-11

  Installed: subscription-manager-1.15.9-15.el7.centos.0.1.x86_64 at 2017-02-21 22:42
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2015-12-08 18:57
  Committed: Johnny Hughes <johnny@centos.org> at 2015-12-08

  Installed: yum-3.4.3-132.el7.centos.0.1.noarch at 2017-02-21 22:42
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2015-12-03 15:33
  Committed: Johnny Hughes <johnny@centos.org> at 2015-12-03

  Installed: yum-plugin-fastestmirror-1.1.31-34.el7.noarch at 2017-02-21 22:42
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2015-11-20 15:34
  Committed: Valentina Mukhamedzhanova <vmukhame@redhat.com> at 2015-10-12

Why subject a community to this unless we have to?

[hartzell]

Think about another scenario, in which you install some libraries like MPI and your users build their own applications outside of Spack that link to your MPI libraries. If they don't know about -Wl,-rpath, their applications won't work unless LD_LIBRARY_PATH is set properly.

Two things (avoiding the joke above):

1. What is the ratio of users of Spack based application trees to people who don't know about -Wl,rpath who are compiling applications that depend on libraries in a Spack deployment?

   The first group is potentially a victim of spooky-action-at-a-distance, the second group is relying on it.

2. What happens to these applications that were built by someone who doesn't understand/control these dependencies when someone changes the set of modules that are being loaded? kerblooie.

   I'd argue that making things automagically work isn't doing them any favors

[hartzell]

All you have to do is create a modules.yaml that looks like:

Yes, I can do that (mentioned playing with it in #3926).

The problem is that it puts the burden of figuring out which modules really need LD_LIBRARY_PATH and which don't on me. I'm poorly suited for that. The module authors (including me, at times) are in the best position to know what their packages need and to state it clearly in the package definition for all to see. If they actually need LD_LIBRARY_PATH, that need should (sez me...) go in setup_environment or setup_dependent_environment and be vetted by the reviewers.

If I blacklist LD_LIBRARY_PATH everywhere, then I'm constantly going to be reacting to problems (probably reported to me by my users).

Package authors/updaters, on the other hand, know ahead of time.

[alalazo]

If it were necessary to set LD_LIBRARY_PATH then perhaps that would be reasonable, but since Spack did/does the [hard] work of using RPATH everywhere it can then the only justification is "in case someone wants to build against this library from outside of Spack.

Let me add a few words to your description: LD_LIBRARY_PATH is set in module files "in case someone wants to build against this library from outside of Spack, and doesn't necessarily know that Spack exists".

Users on a cluster may be unaware of how you built the software that is available through module files, and I don't think telling them to add several lines of directories with strange hashes to manually set the RPATH will help.

An alternative would be to provide some documentation about using Spack libraries in non-Spack projects.

I think this will be what the environment project will try to achieve: set a development environment managed by Spack. And that means also that Spack will take care of setting RPATHs for you.

[alalazo]

What happens when someone adds an updated copy of libc to the tree and everything in my world automagically picks it up.

I see the point here. If I remember well there was a similar discussion sometimes ago (in a PR that was meant to introduce libc, but was never finished).

My personal point of view is that the current state in Spack is a trade off between usability and theoretical correctness, and we should rely on whatever libc (or other similar kind of low level libraries) we have in our OS. We must stop at some point in the software stack, otherwise to be completely isolated from the environment where we install software we'll turn the Spack PM project into the Spack OS project.

This bit: "so there is no need to manipulate LD_LIBRARY_PATH at runtime".

There should be no need if you use a Spack built application. If you use a Spack built library things must be put in perspective: you don't need to set search paths for transitive dependencies (they are RPATHed). So maybe "so there is no need to manipulate LD_LIBRARY_PATH at runtime for the software that was built by Spack" sounds better?

[citibeth]

1. is not relevant (I think it's referring to PERL5LIB, etc...)

Yes, it's referring to things like PERL5LIB and PYTHONPATH. Important, but not relevant to what you're worried about.

PYTHONPATH is evil for the same reason LD_LIBRARY_PATH is evil. I did think up a proposal on how to get RPATH-like functionality in Python --- basically edit every .py file when it's being installed, to set sys.path correctly for imports. Or someting like that. It should work in theory, but someone would have to do it and find all the corner cases.

In the meantime, PYTHONPATH isn't quite as evil as LD_LIBRARY_PATH because Python linking is less evil. If you run with py-numpy@1.1.2 instead of py-numpy@1.1.1, probably nothing bad will happen. But if you dynamically link to the wrong version of a shared lib, the program will probably crash.

That leaves 2), python extensions. Is RPATH unworkable there or just not yet working?

Unfortunately, Python doesn't really have any way of RPATHing dependencies. I don't know the full details, but @wscullin or @citibeth can explain more if you're interested.

Yes that's the end result, see #719 for background. This is probably fixable, but someone would have to look more closely at Python's setuptools and distutils, and figure out how to do it right. In the meantime, Python extensions need LD_LIBRARY_PATH to run properly.

your users build their own applications outside of Spack that link to your MPI libraries. If they don't know about -Wl,-rpath, their applications won't work unless LD_LIBRARY_PATH is set properly. Maybe we should add that use case to the documentation as well.

Everyone should build their applications in Spack. Yes, I'm mostly serious. Some more work on Spack Environments, and we might be closer to that Nirvana.

[tgamblin]

FWIW, I actually relate to what @hartzell's referring to here, and I wouldn't be completely opposed to removing LD_LIBRARY_PATH from modules by default, if we documented it sufficiently. I think most module systems rely on it, though, and I think most HPC users will expect it in their modules. So maybe we should call more attention to how to unset this.

I think the answer here is to put in more checks to ensure that we're actually RPATH-ing all the dependencies in Spack builds, as that will at least ensure that Spack-built binaries work.

[hartzell]

@tgamblin --

I think most module systems rely on it, though

When you say "module systems", do you mean creatures like Lmod?

I can see where the environments which e.g. Lmod or Tcl-modules are trying to set up aren't built with RPATH-ing and depend on LD_LIBRARY_PATH but I can't see how Lmod or Tcl-modules inherently need it.

Would a more nit-picky wording of your statement be: "most HPC computing enviroments rely on it" or some such (putting the blame on how the apps were built, not on the tool being used to provision the user's environment for them)?

I think most HPC users will expect it in their modules.

If a) always setting LD_LIBRARY_PATH is useful because it helps people build software and b) "most HPC users will expect it" does that mean that most HPC users are building software?

Most of my users are not building their own software (be careful what you wish for, I've been told, you might get it...), which might be why I seem to be landing in the minority.

[ifelsefi]

@hartzell

I cannot replicate your yum issue:

[root@node103 spack]# spack load python@3.5.2
[root@node103 spack]# spack load bowtie2
[root@node103 spack]# spack load gcc@6.3.0
[root@node103 spack]# yum --version
3.2.29

Unable to read consumer identity
  Installed: rpm-4.8.0-27.el6.x86_64 at 2014-09-22 01:19
  Built    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> at 2012-04-04 08:43
  Committed: Panu Matilainen <pmatilai@redhat.com> at 2012-04-04

  Installed: subscription-manager-0.99.19-1.el6.x86_64 at 2014-09-22 01:27
  Built    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> at 2012-05-22 19:19
  Committed: Devan Goodwin <dgoodwin@rm-rf.ca> at 2012-05-22

  Installed: yum-3.2.29-30.el6.noarch at 2014-09-22 01:20
  Built    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> at 2012-05-09 11:42
  Committed: Zdenek Pavlas <zpavlas@redhat.com> at 2012-05-09

  Installed: yum-plugin-aliases-1.1.30-14.el6.noarch at 2014-09-22 01:32
  Built    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> at 2012-04-26 11:35
  Committed: Zdenek Pavlas <zpavlas@redhat.com> at 2012-04-26

[root@node103 spack]# echo $LD_LIBRARY_PATH 
/pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.6/gcc-6.3.0-han2ovjpjdnuwkhtfhugdq66cjkte7q3/lib64:/pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.6/gcc-6.3.0-xnoykzftyh27xs6vqkwvtft2cce5yd4k/lib64

I do not think glibc should be added to spack unless there's extensive testing and it's proven to not break things. If users need a newer version of glibc then I think the better solution would be to build a separate spack environment against the newer operating system which has that newer version of glibc. For example I have separate spack environments for centos 6 and centos 7 servers.

To allow users to switch between the separate operating system environments I tell them to put this in their bashrc:

# detect kernel then source spack file if it exists
KERNEL=$(uname -r | cut -d '.' -f1,2)

if [ "${KERNEL}" = "2.6" ] ; then
        if [ -f /pbtech_mounts/softlib001/apps/EL6/spack/share/spack/setup-env.sh ] ; then
                . /pbtech_mounts/softlib001/apps/EL6/spack/share/spack/setup-env.sh
        fi
fi

if [ "${KERNEL}" = "3.10" ] ; then
        if [ -f /pbtech_mounts/softlib001/apps/EL7/spack/share/spack/setup-env.sh ] ; then
                . /pbtech_mounts/softlib001/apps/EL7/spack/share/spack/setup-env.sh
        fi
fi

Moreover, if you need a very new version of glibc, one could try out something like containers as we've been using Singularity http://singularity.lbl.gov then use spack within that. We haven't tested yet but I think it would work.

[hartzell]

I cannot replicate your yum issue:

[root@node103 spack]# spack load python@3.5.2
[...]

I suspect that's because yum uses the system Python, which is Python2.

hartzelg@lb097hmdev:~$ head /usr/bin/yum
#!/usr/bin/python
import sys
try:

The Python you tested with is @3.5.2, and presumably whatever thing that's tripping yum up is sufficiently different.

I can replicate it fairly easily:

hartzelg@lb097hmdev:~$ module purge
hartzelg@lb097hmdev:~$ yum |& head
Loaded plugins: fastestmirror, package_upload, priorities, product-id, search-
              : disabled-repos, subscription-manager
You need to give some command
Usage: yum [options] COMMAND

List of Commands:

check          Check for problems in the rpmdb
check-update   Check for available package updates
clean          Remove cached data
hartzelg@lb097hmdev:~$ module load gcc
hartzelg@lb097hmdev:~$ module load python/2.7.13
hartzelg@lb097hmdev:~$ yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named yum

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, Apr 23 2017, 09:47:24)
[GCC 5.4.0]

If you cannot solve this problem yourself, please go to
the yum faq at:
  http://yum.baseurl.org/wiki/Faq

Can you try it with a python@2?

g.

[hartzell]

I do not think glibc should be added to spack

I'm starting to regret using the example of glibc. It's a worst-case scenario and I spent a lot of time chasing problems in linuxbrew related to building it but it's not categorically different than other packages that Spack installs.

The issue of being surprised by dynamically linking things that you didn't expect is real in other cases, if less catastrophic.

[hartzell]

Thanks for the discussion! I think that I understand the pros and cons.

[ifelsefi]

@hartzell

Discussion is fun!

Anyway, you closed but still cannot replicate on RHEL 6.3 node.

user@node103[~/repos/spack]$ which python
/usr/bin/python

user@node103[~/repos/spack]$ python -V
Python 2.6.6

user@node103[~/repos/spack]$ spack load python@2.7.13
                                                                                                                                                                                      
user@node103[~/repos/spack]$ python -V                                                                                                                                                                                                     
Python 2.7.13

user@node103[~/repos/spack]$ yum
Loaded plugins: aliases, changelog, downloadonly, kabi, presto, product-id, refresh-packagekit, security, subscription-manager, tmprepo, verify, versionlock
Not root, certificate-based repositories not updated
Loading support for Red Hat kernel ABI
You need to give some command
Usage: yum [options] COMMAND

user@node103[~/repos/spack]$ echo $LD_LIBRARY_PATH 
/pbtech_mounts/homes027/user/repos/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.6/python-2.7.13-wyawocw4q6fthbhfbmoe6uecx4njg76e/lib

[hartzell]

Hmmm. I wonder if the difference is my module load (via Lmod) vs. your spack load?

Do you end up with /.../opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib on your LD_LIBRARY_PATH? (hash might differ)?

I do, and if I surgically delete just that directory from the LD_LIBRARY_PATH then yum works again.

[ifelsefi]

@hartzell

I do

Old Centos 6.3 node:

user@node103[~/repos/spack]$ echo $LD_LIBRARY_PATH 
/pbtech_mounts/homes027/dod2014/repos/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.6/python-2.7.13-wyawocw4q6fthbhfbmoe6uecx4njg76e/lib

Python 2.7.13 (default, May 11 2017, 15:07:28) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
>>> import sys
>>> import yum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named yum

So it seems that the Spack-provided Python indeed will break yum. I don't understand though why mine's while your's isn't.

Though I do now have the same problem on Centos 7.3 box.

user@france[~]$ yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named yum

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, Apr 25 2017, 10:47:26) 
[GCC 6.3.0]

If you cannot solve this problem yourself, please go to 
the yum faq at:
  http://yum.baseurl.org/wiki/Faq
  


echo $LD_LIBRARY_PATH 
/pbtech_mounts/softlib001/apps/EL7/spack/opt/spack/linux-centos7-x86_64/gcc-6.3.0/python-2.7.13-j34ptvqierishxtfdndhf4n2ia6vk55v/lib:/pbtech_mounts/softlib001/apps/EL7/spack/opt/spack/linux-centos7-x86_64/gcc-6.3.0/python-2.7.13-vb7ici6xj2dni5voyan5kotc7yilgi2y/lib:/pbtech_mounts/softlib001/apps/EL7/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/python-2.7.13-hrkwpc3zx6736aapgstxuatwdm5vhszs/l

user@france[~]$ spack load python@2.7 %gcc@4.8.5

user@france[~]$ python -V
Python 2.7.13

user@france[~]$ yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named yum

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, May 16 2017, 11:15:46) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

If you cannot solve this problem yourself, please go to 
the yum faq at:
  http://yum.baseurl.org/wiki/Faq

[adamjstewart]

It could be that one was RPATHed and one wasn't. But I'm just speculating.

[hartzell]

If I run strace yum after loading python@2.7.13, it gives up after trying to load yum.{so,py,pyc}|yummodule.so.

stat("/usr/bin", {st_mode=S_IFDIR|0555, st_size=36864, ...}) = 0
stat("/usr/bin", {st_mode=S_IFDIR|0555, st_size=36864, ...}) = 0
stat("/usr/bin/yum", {st_mode=S_IFREG|0755, st_size=801, ...}) = 0
open("/usr/bin/yum.so", O_RDONLY)       = -1 ENOENT (No such file or directory)
open("/usr/bin/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/bin/yum.py", O_RDONLY)       = -1 ENOENT (No such file or directory)
open("/usr/bin/yum.pyc", O_RDONLY)      = -1 ENOENT (No such file or directory)
stat("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/httpie-0.9.8-k27pwfdxo3wumv2nbtmp4n7sjhl2qy7s/lib/python2.7/site-packages/yum", 0x7ffffffeaed0) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/httpie-0.9.8-k27pwfdxo3wumv2nbtmp4n7sjhl2qy7s/lib/python2.7/site-packages/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/httpie-0.9.8-k27pwfdxo3wumv2nbtmp4n7sjhl2qy7s/lib/python2.7/site-packages/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/httpie-0.9.8-k27pwfdxo3wumv2nbtmp4n7sjhl2qy7s/lib/python2.7/site-packages/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/httpie-0.9.8-k27pwfdxo3wumv2nbtmp4n7sjhl2qy7s/lib/python2.7/site-packages/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-setuptools-34.4.1-wuvgblqg2ttm7qar7x43qun5cyi356i5/lib/python2.7/site-packages/yum", 0x7ffffffeaed0) = -1 ENOENT (No such file or directory)
[...]
stat("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib/python2.7/site-packages/yum", 0x7ffffffeaed0) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib/python2.7/site-packages/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib/python2.7/site-packages/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib/python2.7/site-packages/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib/python2.7/site-packages/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "There was a problem importing on"..., 500There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named yum

Please install a package which provides this module, or

I've trimmed the above, but trust me that it never looks for it in any of the system directories.

If I unset LD_LIBRARY_PATH then strace yum then it runs successfully (here's the analogous bit to the above. Note that it finds what it's looking for in /usr/lib/python2.7/site-packages/yum.

stat("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-pysocks-1.6.6-cmakuuvdpuktjn2m4rdfr4kh2p32nxlf/lib/python2.7/site-packages/yum", 0x7ffffffec760) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-pysocks-1.6.6-cmakuuvdpuktjn2m4rdfr4kh2p32nxlf/lib/python2.7/site-packages/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-pysocks-1.6.6-cmakuuvdpuktjn2m4rdfr4kh2p32nxlf/lib/python2.7/site-packages/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-pysocks-1.6.6-cmakuuvdpuktjn2m4rdfr4kh2p32nxlf/lib/python2.7/site-packages/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-pysocks-1.6.6-cmakuuvdpuktjn2m4rdfr4kh2p32nxlf/lib/python2.7/site-packages/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/yum", 0x7ffffffec760) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/plat-linux2/yum", 0x7ffffffec760) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/lib-dynload/yum", 0x7ffffffec760) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/site-packages/yum", 0x7ffffffec760) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/site-packages/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/site-packages/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/site-packages/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/site-packages/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib/python2.7/site-packages/yum", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/usr/lib/python2.7/site-packages/yum/__init__.py", {st_mode=S_IFREG|0755, st_size=305199, ...}) = 0
stat("/usr/lib/python2.7/site-packages/yum/__init__", 0x7ffffffec710) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/site-packages/yum/__init__.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/site-packages/yum/__init__module.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/site-packages/yum/__init__.py", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=305199, ...}) = 0
open("/usr/lib/python2.7/site-packages/yum/__init__.pyc", O_RDONLY) = 4

Without LD_LIBRARY_PATH to distract it, it looks in the system dirs and finds what it needs.

Why would setting LD_LIBRARY_PATH (used to search for shared libraries) influence Python's search for a python package?

[ifelsefi]

@hartzell @adamjstewart

On Centos 7.3 the rpm yum-3.4.3-150.el7.centos.noarch contains files within:

user@france[~]$ rpm -ql yum-3.4.3-150.el7.centos.noarch | grep py | head
/usr/lib/python2.7/site-packages/rpmUtils
/usr/lib/python2.7/site-packages/rpmUtils/__init__.py
/usr/lib/python2.7/site-packages/rpmUtils/__init__.pyc
/usr/lib/python2.7/site-packages/rpmUtils/arch.py
/usr/lib/python2.7/site-packages/rpmUtils/arch.pyc
/usr/lib/python2.7/site-packages/rpmUtils/miscutils.py
/usr/lib/python2.7/site-packages/rpmUtils/miscutils.pyc
/usr/lib/python2.7/site-packages/rpmUtils/oldUtils.py
/usr/lib/python2.7/site-packages/rpmUtils/oldUtils.pyc
/usr/lib/python2.7/site-packages/rpmUtils/transaction.py

user@france[~]$ export PYTHONPATH=/usr/lib/python2.7/site-packages

user@france[~]$ echo $PYTHONPATH 
/usr/lib/python2.7/site-packages

user@france[~]$ yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named rpm

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, Apr 25 2017, 10:47:26) 
[GCC 6.3.0]

If you cannot solve this problem yourself, please go to 
the yum faq at:
  http://yum.baseurl.org/wiki/Faq

user@france[~]$ rpm -ql rpm-python-4.11.3-21.el7.x86_64 | grep -e rpm | head                                                                                                                                                               
/usr/lib64/python2.7/site-packages/rpm
/usr/lib64/python2.7/site-packages/rpm/__init__.py
/usr/lib64/python2.7/site-packages/rpm/__init__.pyc
/usr/lib64/python2.7/site-packages/rpm/__init__.pyo
/usr/lib64/python2.7/site-packages/rpm/_rpm.so
/usr/lib64/python2.7/site-packages/rpm/_rpmb.so
/usr/lib64/python2.7/site-packages/rpm/_rpms.so
/usr/lib64/python2.7/site-packages/rpm/transaction.py
/usr/lib64/python2.7/site-packages/rpm/transaction.pyc
/usr/lib64/python2.7/site-packages/rpm/transaction.py

user@france[~]$ export PYTHONPATH=/usr/lib/python2.7/site-packages:/usr/lib64/python2.7/site-packages

user@france[~]$ echo $PYTHONPATH 
/usr/lib/python2.7/site-packages:/usr/lib64/python2.7/site-packages

user@france[~]$ yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   /usr/lib64/python2.7/site-packages/rpm/_rpm.so: undefined symbol: PyUnicodeUCS4_AsUTF8String

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, Apr 25 2017, 10:47:26) 
[GCC 6.3.0]

If you cannot solve this problem yourself, please go to 
the yum faq at:
  http://yum.baseurl.org/wiki/Faq

So it seems to be an issue with the fact that yum's python things are not installed within Spack's Python environment.

I don't consider this to be a problem as our users are not using yum though it would be nice if we could fix somehow since I source Spack via ~/.bashrc. Though I always do "sudo yum" so not, actually...

[hartzell]

Nice digging.

I'm content to close this (again). There's enough documentation here if any ever trips over it and it's not a day-to-day problem (yet).

[hartzell]

Here's another cute one (CentOS 7), where Spack's libtiff is screwing up the system's ps2pdf application:

hartzelg@blah:spack-graphviz (bug/graphviz-and-swig)$ ps2pdf foo.gv.ps2 foo.pdf
/usr/bin/gs: /doh/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/libtiff-4.0.6-g7gpsqww4hfkg5tg4nr5izmc2sj76c2z/lib/libtiff.so.5: no version information available (required by /lib64/libgs.so.9)
hartzelg@blah:spack-graphviz (bug/graphviz-and-swig)$ unset LD_LIBRARY_PATH; ps2pdf foo.gv.ps2 foo.pdf
hartzelg@blah:spack-graphviz (bug/graphviz-and-swig)$ file foo.pdf
foo.pdf: PDF document, version 1.4

I think that I'll take a test drive of @adamjstewart's blacklisting suggestion and see what breaks.

[ChristianTackeGSI]

I stumbled over this, because LD_LIBRARY_PATH breaks nearly anything using ncurses on my Debian-10 machine:

$ /usr/bin/clear_console
/usr/bin/clear_console: /opt/spack/install-tree/debian10/gcc-9.3.0/ncurses-6.2-zqozh/lib/libtinfo.so.6: no version information available (required by /usr/bin/clear_console)

So I would like to blacklist LD_LIBRARY_PATH for spack load. I just haven't found an easy way to do that?

And on the other side, modules.yaml / prefix_inspections doesn't seem to be honored.

[ReinhardPrix]

I'm seeing exactly the same issue, on Debian stable (10) and testing/bullseye (11), I'm getting:

bash: /home/gitlab-runner/builds/c-5dWVoW/0/cw-group/cw-software/.spack/opt/spack/linux-debiantesting-x86_64/gcc-10.1.0/ncurses-6.2-g663epv4unoj2sohfwgpumxnyvvnverl/lib/libtinfo.so.6: no version information available (required by bash)

which seems to come from spack load ncurses, as

$ echo $LD_LIBRARY_PATH

$ ldd /bin/bash
	linux-vdso.so.1 (0x00007ffc33317000)
	libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f156e59a000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f156e595000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f156e3d4000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f156e734000)
$ spack load ncurses
$ ldd /bin/bash
/bin/bash: /home/repr/cw-software-0.15/.spack/opt/spack/linux-debian10-x86_64/gcc-8.3.0/ncurses-6.2-mkf5o7z3jz7zm4vyylvlawgdbvry5zzj/lib/libtinfo.so.6: no version information available (required by /bin/bash)
	linux-vdso.so.1 (0x00007ffcf359d000)
	libtinfo.so.6 => /home/repr/cw-software-0.15/.spack/opt/spack/linux-debian10-x86_64/gcc-8.3.0/ncurses-6.2-mkf5o7z3jz7zm4vyylvlawgdbvry5zzj/lib/libtinfo.so.6 (0x00007f06faecd000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f06fae86000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f06facc5000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f06fb036000)
$ echo $LD_LIBRARY_PATH
/home/repr/cw-software-0.15/.spack/opt/spack/linux-debian10-x86_64/gcc-8.3.0/ncurses-6.2-mkf5o7z3jz7zm4vyylvlawgdbvry5zzj/lib:/home/repr/cw-software-0.15/.spack/opt/spack/linux-debian10-x86_64/gcc-8.3.0/ncurses-6.2-mkf5o7z3jz7zm4vyylvlawgdbvry5zzj/lib

So spack load sets LD_LIBRARY_PATH, which then 'breaks' system tools. Is there any easy way to turn off LD_LIBRARY_PATH setting in spack load ?

[ChristianTackeGSI]

If #17457 (comment) would get implemented, you could override the prefix_inspections in modules.yaml.

Until then I see two workarounds:

• Don't use spack load and use normale environment modules / lmod: Let spack generate them for you, then use module load foo.
• Patch lib/spack/spack/user_environment.py:prefix_inspections() in your local spack.

I might look into implementing #17457 (comment) sometime.

[ChristianTackeGSI]

Another question that lately came up concerning this topic:

Let's assume one disables LD_LIBRARY_PATH globally (as described above in various ways).
Let's further assume doing so is officially supported.

How does the package.py for a specific package that really needs LD_LIBRARY_PATH to be set (for whatever reason), and knows, that it will likely not break too much, go at setting it?

• Would it simply do that in setup_*_environment?
• Do we have another method for this?

[haampie]

Since #18689 was closed as a duplicate, let me add the tl;dr here:

In the last two weeks two people reported issues on Slack where spack load broke system executables as a result of spack paths in LD_LIBRARY_PATH.

In particular: spack load cmake causes system git to fail on CentOS 8, because openssl gets pirated.

Another instance is where my terminal/shell became completely unusable after spack load [dependent of ncurses] as system lua started using an in compatible ncurses provided by spack, resulting in a warning message at the start of every line, as I'm using z.lua in my shell. So no spack load for me :(

---

I would suggest to not modify LD_LIBRARY_PATH upon spack load [...], except when the user explicitly asks for it: spack load --libraries [...] or so.

[adamjstewart]

Just to add another point to this, many Python packages don't support RPATH, and won't work unless (DY)?LD_LIBRARY_PATH is set.

[haampie]

Python packages don't support RPATH

Does Python maybe have a dedicated env variable for search paths it dlopens?

If not, how about adding a prop export_search_paths = false to PackageBase, and set it to true for PythonPackage? Then spack load would only have to add paths to (DY)LD_LIBRARY_PATH whenever export_search_paths is true.

[adamjstewart]

Does Python maybe have a dedicated env variable for search paths it dlopens?

Not that I know of, just (DY)?LD_LIBRARY_PATH.

[michaelkarlcoleman]

Personally I would prefer that LD_LIBRARY_PATH always be prepended with useful shared libs, when available. Yes, it can cause trouble if you're trying to mix spack and non-spack packages. But, if you do that, you're already living in sin--don't do it.

A middle ground would be to add a flag to 'spack load' that would optionally also export libs this way. So, they could be exported when needed, but not by default.

If nothing else, it would be handy to (optionally?) set a package-specific env var so that the lib location in question could be easily derived, rather than having to determine it by magical means.

[BenWibking]

Since #18689 was closed as a duplicate, let me add the tl;dr here:

In the last two weeks two people reported issues on Slack where spack load broke system executables as a result of spack paths in LD_LIBRARY_PATH.

In particular: spack load cmake causes system git to fail on CentOS 8, because openssl gets pirated.

Another instance is where my terminal/shell became completely unusable after spack load [dependent of ncurses] as system lua started using an in compatible ncurses provided by spack, resulting in a warning message at the start of every line, as I'm using z.lua in my shell. So no spack load for me :(

I would suggest to not modify LD_LIBRARY_PATH upon spack load [...], except when the user explicitly asks for it: spack load --libraries [...] or so.

git also does not work after spack load openmpi on CentOS 8 :(

bwibking@avatargpue:~/quokka> ldd /usr/libexec/git-core/git-remote-https
	linux-vdso.so.1 (0x00007ffcb07d3000)
	libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f6fefcfa000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f6fefabf000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f6fef83b000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f6fef624000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6fef404000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f6fef1fc000)
	libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f6feed16000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f6fee951000)
	libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f6fee72a000)
	libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f6fee50c000)
	libssh.so.4 => /lib64/libssh.so.4 (0x00007f6fee29d000)
	libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f6fee08c000)
	libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f6feddf8000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f6fedba3000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f6fed8ba000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f6fed6a3000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f6fed49f000)
	libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f6fed251000)
	liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f6fed041000)
	libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f6fece34000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6ff0385000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f6fecc30000)
	libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f6fec8af000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f6fec69e000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6fec49a000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6fec283000)
	libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f6fec065000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f6febce3000)
	libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f6febac2000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f6feb898000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f6feb66f000)
bwibking@avatargpue:~/quokka> spack load openmpi
bwibking@avatargpue:~/quokka> ldd /usr/libexec/git-core/git-remote-https
	linux-vdso.so.1 (0x00007ffea5ae9000)
	libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f4b7b7d2000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f4b7b597000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f4b7b313000)
	libz.so.1 => /avatar/bwibking/spack/opt/spack/linux-rocky8-skylake_avx512/gcc-8.4.1/zlib-1.2.11-hlaky7kmtfaieud3deytwmi73gkicoby/lib/libz.so.1 (0x00007f4b7b0fc000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4b7aedc000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f4b7acd4000)
	libcrypto.so.1.1 => /avatar/bwibking/spack/opt/spack/linux-rocky8-skylake_avx512/gcc-8.4.1/openssl-1.1.1l-2k7gnwjtptaejuxphbnvwn7s6lm3szp6/lib/libcrypto.so.1.1 (0x00007f4b7a7ee000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f4b7a429000)
	libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f4b7a202000)
	libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f4b79fe4000)
	libssh.so.4 => /lib64/libssh.so.4 (0x00007f4b79d75000)
	libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f4b79b64000)
	libssl.so.1.1 => /avatar/bwibking/spack/opt/spack/linux-rocky8-skylake_avx512/gcc-8.4.1/openssl-1.1.1l-2k7gnwjtptaejuxphbnvwn7s6lm3szp6/lib/libssl.so.1.1 (0x00007f4b798d2000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f4b7967d000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f4b79394000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f4b7917d000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f4b78f79000)
	libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f4b78d2b000)
	liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f4b78b1b000)
	libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f4b7890e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4b7be5d000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f4b7870a000)
	libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f4b78389000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f4b78178000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f4b77f74000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f4b77d5d000)
	libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f4b77b3f000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f4b777bd000)
	libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f4b7759c000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f4b77372000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f4b77149000)

[BenWibking]

Unfortunately, spack load cmake, spack load openmpi and spack load rsync all break git on CentOS / Rocky 8 :(

Any workarounds would be appreciated!

[citibeth]

1. I generally use spack environments not spack load. Either with views or modules. 2. In assembling the environment I generally address issues like this as the come up. In this case I would save LD-Library-Path then source all the module loads, then restore ld library path. That would all happen in the final environment loading script I produce. 3. In general I would use the spack command to build and assemble environments but not to load them.

On Thu, Dec 23, 2021 at 23:56 Ben Wibking ***@***.***> wrote: Unfortunatley, spack load cmake, spack load openmpi and spack load rsync all break git on CentOS / Rocky 8 :( Any workarounds would be appreciated! — Reply to this email directly, view it on GitHub <#3955 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOVY52NKYOZO4GUGQXBV6LUSP4QZANCNFSM4DIUO3HQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Elizabeth Ashley Fischer, she/her University of Alaska, Geophysical Institute cell: 617-308-0436 UA is an AA/EO employer and educational institution and prohibits illegal discrimination against any individual: www.alaska.edu/nondiscrimination. UAF is located at Troth Yeddha', on the traditional homelands of the Lower Tanana Dené People and UAF therefore benefits from the long stewardship of these lands.

[adamjstewart]

@BenWibking Can you try using the following modules.yaml?

modules:
  prefix_inspections::  # notice the double colon
    bin:
      - PATH
    man:
      - MANPATH
    share/man:
      - MANPATH
    share/aclocal:
      - ACLOCAL_PATH
    lib/pkgconfig:
      - PKG_CONFIG_PATH
    lib64/pkgconfig:
      - PKG_CONFIG_PATH
    share/pkgconfig:
      - PKG_CONFIG_PATH
    '':
      - CMAKE_PREFIX_PATH

This will override the default settings and prevent Spack from setting LD_LIBRARY_PATH. I know this works for module file creation. If it doesn't work for spack load, let me know.

[BenWibking]

It works, thanks!

[haampie]

Unfortunately, Python doesn't really have any way of RPATHing dependencies.

I believed this cause I am not a Python user, but I think it's not true. Python packages typically open a wrapper library which has rpaths, right? So, I doubt they rely on LD_LIBRARY_PATH much?

I still believe we should disable LD_LIBRARY_PATH by default.

@giordano reports a system where less is dynamically linked to curses, and pretty much any spack environment includes curses, meaning that loading spack environments break the command line.