scipy 0.17.1/0.18/0.19 packages built incorrectly

[stuarteberg]

The latest version (0.17.1 0.18.0) of Anaconda's scipy package for OS X was built to link against /usr/local/lib/libgcc_s.1.dylib, which is bad. The previous version (0.17.0) does not have this problem.

$ conda list -n myenv scipy
# packages in environment at /miniconda2/envs/myenv:
#
scipy                     0.17.1              np111py27_1

$ find /miniconda2/envs/myenv/lib/python2.7/site-packages/scipy/ -name "*.so" \
  | xargs otool -L \
  | grep /usr/local \
  | uniq
    /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

On my machine, the bad links don't cause problems, but on my colleague's machine, it causes an error when he tries to import certain modules.

For example, somehow this scipy problem causes an error when importing sklearn:

  File "/miniconda/envs/myenv/lib/python2.7/site-packages/sklearn/__init__.py", line 56, in <module>
    from . import __check_build
ImportError: cannot import name __check_build

[mingwandroid]

Ouch, the perils of homebrew.

[andykitchen]

Yup can confirm I had this problem as well, the fix for me was to also to downgrade to 0.17.0

conda install --force scipy=0.17.0

EDIT: Turns out this can manifest because of a custom value set for DYLD_FALLBACK_LIBRARY_PATH

You can try:

$ unset DYLD_FALLBACK_LIBRARY_PATH

If scipy works after that, you may choose to add /usr/lib to your DYLD_FALLBACK_LIBRARY_PATH e.g.

$ DYLD_FALLBACK_LIBRARY_PATH="$DYLD_FALLBACK_LIBRARY_PATH:$(HOME)/lib:/usr/local/lib:/lib:/usr/lib"

A note to the anaconda maintainers, I believe that this is still an error, scipy should be linking /usr/lib/libgcc_s.1.dylib not /usr/lib/local/libgcc_s.1.dylib. I believe this accidentally works on some setups because the default DYLD_FALLBACK_LIBRARY_PATH contains /usr/lib

[stuarteberg]

Update: The new latest scipy version (0.18.0) also has the same problem. In case it's somehow useful, the conda-forge package for scipy is not broken in this way.

[ktavabi]

I am getting similar error with virtual environments built around scipy 0.16.x - 0.18 on OSX 10.11.6

ImportError: dlopen(/Users/.../anaconda/lib/python2.7/site-packages/scipy/special/_ufuncs.so, 2): Library not loaded: /usr/local/lib/libgcc_s.1.dylib
Referenced from: /Users/.../anaconda/lib/python2.7/site-packages/scipy/special/_ufuncs.so
Reason: image not found

EDIT: It looks like any import from scipy fails with same error on my MacBook. @andykitchen workaround works

[sensharma]

@andykitchen You saved the day for me.

[ghost]

@ilanschnell Is there any plan to fix this?

[stuarteberg]

FYI, this is still a problem with the latest scipy packages (0.19.0) from the defaults channel.

As a workaround, I've created a custom post-link.sh script that fixes the bad scipy .so files. I added it to a special version of the scipy package (adapted from the one on the defaults) channel, but you can also just run it as a stand-alone script after installing the standard scipy package:

conda install -n myenv scipy
PREFIX=$(conda info --root)/envs/myenv scipy-post-link.sh

Anyway, here's the script:

#!/bin/bash

# scipy-post-link.sh
#
# This is a post-link script for the scipy package, to work around
# a problem with how those binaries are built.
# For details, see:
# https://github.com/ContinuumIO/anaconda-issues/issues/899
#

if [ $(uname) != "Darwin" ];
then
  # Nothing to do
  exit 0
fi

if [ -z "$PREFIX" ];
then
  1>&2 echo "PREFIX is undefined! Exiting."
  exit 1
fi

#
# The .so files in the scipy package link against /usr/local/lib/libgcc_s.1.dylib, which is bad.
# Here, we replace those bad links with /usr/lib/libSystem.B.dylib
#
for f_dylib in $(find ${PREFIX}/lib/python2.7/site-packages/scipy -type f -name "*.so");
do
  if otool -L $f_dylib | grep --quiet '/usr/local/lib/libgcc_s.1.dylib';
  then
    cmd="install_name_tool -change /usr/local/lib/libgcc_s.1.dylib /usr/lib/libSystem.B.dylib $f_dylib"
    echo $cmd
    $cmd
  fi
done

[ilanschnell]

Thanks. I'm going to fix the issue by basically adding the relink step to the build script in the scipy recipe on OSX.

[stuarteberg]

I'm going to fix the issue by basically adding the relink step to the build script in the scipy recipe on OSX.

Excellent.

In case you want to see what the resulting .so files will look like, my adapted version of your scipy package can be found here:

https://anaconda.org/ilastik/scipy/0.19.0/download/osx-64/scipy-0.19.0-np112py27_999_fixed_links.tar.bz2

conda install -c ilastik scipy=0.19.0=np112py27_999_fixed_links

Note: By replacing /usr/local/lib/libgcc_s.1.dylib with /usr/lib/libSystem.B.dylib, the .so files end up with TWO references to libSystem.B.dylib, but it doesn't seem to cause any problems. (The scipy test suite still passes.)

$ otool -L /miniconda2/envs/test-scipy/lib/python2.7/site-packages/scipy/linalg/_flinalg.so
/miniconda2/envs/test-scipy/lib/python2.7/site-packages/scipy/linalg/_flinalg.so:
	@rpath/libmkl_intel_lp64.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_intel_thread.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_core.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1.0.0)

Indeed, Apple's own version of /usr/lib/libgcc_s.1.dylib is merely a link to libSystem.B.dylib:

$ ls -l /usr/lib/libgcc_s.1.dylib
lrwxr-xr-x  1 root  wheel  17 Dec 26 15:45 /usr/lib/libgcc_s.1.dylib@ -> libSystem.B.dylib

... the only problem, for our purposes, is that it lives in /usr/lib, not /usr/local/lib.

[mingwandroid]

This has been fixed since the release of AD5.