[BUG] No RPATH in Python extensions #719

citibeth · 2016-04-01T21:29:33Z

See the dialog below for details. Since Spack does not use hierarchical modules, working around this bug requires that one do spack load on an arbitrary and growing set of modules.

[me@ankeli spack2]$ spack purge
[me@ankeli spack2]$ spack load py-numpy
[me@ankeli spack2]$ python3 -c 'import numpy'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

Uh-oh, what's wrong... let's debug this with ldd:

[me@ankeli spack2]$ ldd `spack location -i py-numpy`/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/core/multiarray.cpython-35m-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007ffc4ef1f000)
    libopenblas.so.0 => not found
    libm.so.6 => /lib64/libm.so.6 (0x00007f6ccbd1d000)
    libpython3.5m.so.1.0 => /home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/python-3.5.1-5q5rsrtjld4u6jiicuvtnx52m7tfhegi/lib/libpython3.5m.so.1.0 (0x00007f6ccb7f7000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6ccb5db000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f6ccb219000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6ccc440000)
    libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f6ccb014000)
    libutil.so.1 => /usr/lib64/libutil.so.1 (0x00007f6ccae11000)

Same problem, it cannot load libopenblas.so. So let's tell Spack to load it explicitly and try again:

[me@ankeli spack2]$ spack load openblas
[me@ankeli spack2]$ ldd `spack location -i py-numpy`/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/core/multiarray.cpython-35m-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007ffc78d2c000)
    libopenblas.so.0 => /home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/openblas-0.2.17-pw6rmlom7apfsnjtzfttyayzc7nx5e7y/lib/libopenblas.so.0 (0x00007fb61f48e000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fb61f175000)
    libpython3.5m.so.1.0 => /home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/python-3.5.1-5q5rsrtjld4u6jiicuvtnx52m7tfhegi/lib/libpython3.5m.so.1.0 (0x00007fb61ec4f000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb61ea32000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fb61e671000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb620878000)
    libgfortran.so.3 => /home/me/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-4.9.3-layphctulnk3omsbjpzftqv6dlxpfe3d/lib64/libgfortran.so.3 (0x00007fb61e352000)
    libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007fb61e14d000)
    libutil.so.1 => /usr/lib64/libutil.so.1 (0x00007fb61df4a000)
    libquadmath.so.0 => /home/me/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-4.9.3-layphctulnk3omsbjpzftqv6dlxpfe3d/lib64/../lib64/libquadmath.so.0 (0x00007fb61dd0c000)
    libgcc_s.so.1 => /home/me/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-4.9.3-layphctulnk3omsbjpzftqv6dlxpfe3d/lib64/../lib64/libgcc_s.so.1 (0x00007fb61daf6000)

OK, now it works with ldd. And try with Python:

[me@ankeli spack2]$ python3 -c 'import numpy'

Remember... LD_LIBRARY_PATH is EVIL!!!

The text was updated successfully, but these errors were encountered:

citibeth · 2016-04-02T02:35:25Z

Another consequence of this phenomenon is that it is not possible to BUILD some Python packages without first LOADING their dependencies. The following does NOT work:

spack build py-numpy
spack build py-netcdf     # setup.py cannot import numpy

At this point, I don't know how to get this working.

glennpj · 2016-04-02T04:16:25Z

Using the develop branch I was able to build py-netcdf, using 'spack
install...', without any modules loaded or activated.

Glenn

On Fri, Apr 1, 2016 at 9:35 PM, Elizabeth Fischer notifications@github.com
wrote:

Another consequence of this phenomenon is that it is not possible to BUILD
some Python packages without first LOADING their dependencies. The
following does NOT work:

spack build py-numpy
spack build py-netcdf # Build cannot import numpy

This modified version DOES work:

spack build py-numpy
spack load py-numpy
spack build py-netcdf # Build cannot import numpy

or:

spack build py-numpy
spack activate py-numpy
spack build py-netcdf # Build cannot import numpy

Should Spack load the modules of its dependencies when setting up a build
environment?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#719 (comment)

citibeth · 2016-04-02T04:36:27Z

Thank you for the hint. Looks like I've got my work cut out for me, digging through git diffs...
I will see if I can replicate your develop branch experience, building for Python3.

citibeth · 2016-04-02T04:59:19Z

I tried the same thing from develop branch and it didn't work for me.

I think that SHA 220c72d (Merge branch 'features/blas-lapack-hardening' into develop) broke things here. Before that commit, OpenBLAS ONLY built static libraries. py-numpy would statically link OpenBLAS, and things would "just work." Now, OpenBLAS builds shared by default. py-numpy links to the shared OpenBLAS library (without RPATH) and things don't work.

An alternative workaround to the code posted in this PR would be to use openblas~shared.

citibeth · 2016-04-04T02:27:15Z

PR #721 helps with this somewhat. But it would be better if the new recursive module feature could list ONLY modules that are Python-related (i.e. that set PYTHONPATH in the module file).

adamjstewart · 2016-07-06T16:17:00Z

I think I'm seeing the same problem as you. When I try to install py-scipy, I get:

ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

This never happened to me before though. I wonder if something changed.

citibeth · 2016-07-06T16:48:55Z

Load the scipy module
On Jul 6, 2016 12:17 PM, "Adam J. Stewart" notifications@github.com wrote:

I think I'm seeing the same problem as you. When I try to install
py-scipy, I get:

ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

This never happened to me before though. I wonder if something changed.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#719 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AB1cdxNxneZODhuFaTG3AIGWm-LHf_dYks5qS9T-gaJpZM4H-DYi
.

adamjstewart · 2016-07-06T17:29:28Z

It's not a runtime problem, I literally can't even install it. I swear this worked a week or two ago. I wonder if some environment variables changed.

alalazo · 2016-07-06T17:40:04Z

@adamjstewart you may want to check if this or this affected your build in any way

glennpj · 2016-07-06T18:08:26Z

Just as a data point, I just successfully built py-scipy with openblas from the current spack devel branch.

adamjstewart · 2016-07-06T18:11:53Z

@alalazo I commented out all of the changes from #1157 but it didn't help. I can't imagine that d687e33 is causing the problem since it only affects the PATH.

adamjstewart · 2016-07-06T18:18:43Z

I also tried adding depends_on('blas') and depends_on('lapack') directly to py-scipy. It's almost like scipy is ignoring the BLAS and LAPACK environment variables we're setting.

adamjstewart · 2016-07-06T18:32:10Z

@glennpj I just tried the latest develop but I'm still seeing the same problem. Here is the full spack-build.out:

==> '/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/python-2.7.11-voyxwv4ijmxs3qxyo5wjwxtps3uj2hhm/bin/python' 'setup.py' 'install' '--prefix=/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-scipy-0.17.0-wq7aypg6ggmfgiklptambydihsyxjqb7'
Traceback (most recent call last):
  File "setup.py", line 265, in <module>
    setup_package()
  File "setup.py", line 253, in setup_package
    from numpy.distutils.core import setup
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

Seems to be exactly what @citibeth was seeing. It's interesting that I'm only seeing it for the first time now though. @citibeth you mentioned that you had an idea of how to get RPATH working for Python modules?

glennpj · 2016-07-06T18:59:49Z

@adamjstewart
I have built packages with gcc-6.1 but I am not sure if I built the python stack with it. The successful build that I referenced was with the CentOS-7 system gcc (4.8.5). I will fire up a build with gcc-6.1 to see what happens.

glennpj · 2016-07-07T14:26:17Z

@adamjstewart

I can replicate what you are seeing with spack compiled gcc-6.1.0 and gcc-5.3.0. Building with the system gcc does work though.

* updated bglibpy version * remove pinned BluePy version on BGLibPy * fixed style: [W293] blank line contains whitespace

citibeth mentioned this issue Apr 2, 2016

[WIP] Workaround (or permanent fix?) for depends_on(py-numpy) #722

Closed

glennpj mentioned this issue May 11, 2016

feature : customization of modules from configuration files #744

Merged

8 tasks

nrichart added a commit to epfl-scitas/spack that referenced this issue Jul 7, 2016

fixes spack#719

4c1b536

nrichart mentioned this issue Jul 7, 2016

RPATH for blas in py numpy #1188

Merged

SteVwonder mentioned this issue Jul 11, 2016

Flux fails to configure/build #1213

Closed

tgamblin closed this as completed in #1188 Jul 18, 2016

nrichart mentioned this issue Dec 1, 2016

py-numpy: rework blas/lapack #2365

Merged

4 tasks

citibeth mentioned this issue Apr 28, 2017

Setting LD_LIBRARY_PATH to be "helpful" considered harmful #3955

Closed

matz-e pushed a commit to matz-e/spack that referenced this issue Apr 27, 2020

Update BGLibPy version to 4.2.14 (spack#719)

34a0cc4

* updated bglibpy version * remove pinned BluePy version on BGLibPy * fixed style: [W293] blank line contains whitespace

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] No RPATH in Python extensions #719

[BUG] No RPATH in Python extensions #719

citibeth commented Apr 1, 2016

citibeth commented Apr 2, 2016

glennpj commented Apr 2, 2016

citibeth commented Apr 2, 2016

citibeth commented Apr 2, 2016

citibeth commented Apr 4, 2016

adamjstewart commented Jul 6, 2016

citibeth commented Jul 6, 2016

adamjstewart commented Jul 6, 2016

alalazo commented Jul 6, 2016

glennpj commented Jul 6, 2016 •

edited

Loading

adamjstewart commented Jul 6, 2016

adamjstewart commented Jul 6, 2016

adamjstewart commented Jul 6, 2016

glennpj commented Jul 6, 2016

glennpj commented Jul 7, 2016

[BUG] No RPATH in Python extensions #719

[BUG] No RPATH in Python extensions #719

Comments

citibeth commented Apr 1, 2016

citibeth commented Apr 2, 2016

glennpj commented Apr 2, 2016

citibeth commented Apr 2, 2016

citibeth commented Apr 2, 2016

citibeth commented Apr 4, 2016

adamjstewart commented Jul 6, 2016

citibeth commented Jul 6, 2016

adamjstewart commented Jul 6, 2016

alalazo commented Jul 6, 2016

glennpj commented Jul 6, 2016 • edited Loading

adamjstewart commented Jul 6, 2016

adamjstewart commented Jul 6, 2016

adamjstewart commented Jul 6, 2016

glennpj commented Jul 6, 2016

glennpj commented Jul 7, 2016

glennpj commented Jul 6, 2016 •

edited

Loading