Rebuild Python with conda build on OS X

[asmeurer]

I was kind of hoping this would have happened by now. Conda build on OS X adds @rpath to the install name on OS X and adds an LC_RPATH. This makes it possible to build things that link against libraries and have them work without having to call install_name_tool (or build them with conda build, which calls it for you).

An example of this: sympy.utilities.autowrap generates a Fortran extension module on the fly using f2py (part of NumPy). Create a conda environment on OS X with python sympy numpy gcc and run

from sympy import *
x = symbols('x')
from sympy.utilities.autowrap import autowrap
autowrap(sin(x), backend='f2py', tempdir='.')

This will attempt to use f2py to generate an extension module in the current directory using fortran (with gfortran from the gcc package) and import it. On OS X, you get an error like

ImportError: dlopen(/Users/aaronmeurer/Documents/Python/sympy/sympy/wrapper_module_0.cpython-35m-darwin.so, 2): Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/aaronmeurer/Documents/Python/sympy/sympy/wrapper_module_0.cpython-35m-darwin.so
  Reason: image not found

What this means is that the extension module linked against libgfortran from the gcc package which has @rpath/./libgfortran.3.dylib as its install name. The @rpath should reference the LC_RPATH of some loaded library, which would point to the lib/ directory of the environment, but nothing already loaded has an LC_RPATH, in particular, Python itself does not. Effectively, it's impossible to import this extension module without "fixing" it with install_name_tool, which is undesirable for an automated tool like autowrap.

To fix this, Python should be rebuilt with conda-build. This will add the LC_RPATH to libpython3.5m.dylib and the dyld loader will be able to find the extension module.

I tested this myself, by building the python-3.5 recipe in conda-recipe (I added the build string conda_build_0 so you can tell the difference).

You can test with

conda create -n test -c asmeurer python=3.5.0=conda_build_0 sympy numpy gcc

and run the above code. It will work. (note that if you don't pin that Python build in the test environment further conda commands will "update" it to the standard one, since that has a newer build string)

You can see the difference with conda inspect objects python (or just using otool -L and otool -l).

[ccordoba12]

Letting @msarahan know about this one. We'll try to migrate all recipes to be conda-build ones after we release Anaconda 2.4.

[groutr]

@groutr, I want to track this issue as well.

[danielsf]

Just adding a +1 to the desirability of this fix. I work on the software development team for the Large Synoptic Survey Telescope. We have been having trouble making our build system work with anaconda because of the lack of @rpath in the loading address for libpython*.dylib in Anaconda.

[mjuric]

@msarahan et al -- would a short-term fix be possible, in the form of adding a line such as:

install_name_tool -id @rpath/libpython2.7.dylib libpython2.7.dylib

to your existing Python build script? That seems to fix some of our build issues.

[msarahan]

Ping @ilanschnell on this. He needs to make the call on this issue.

[asmeurer]

@mjuric that page can't be viewed without a login.

Just changing the id isn't enough. You also need to set the LC_RPATH on the dylib. It may work anyway for some cases if it can find the LC_RPATH from another dylib in the load chain, but in general it is needed. conda build does this all automatically (and also changes about a dozen other object files that are part of the python package).

As a workaround, I built python with conda build, as noted in the OP (Python 3.5 only). You can use that with

conda create -n conda-build-python -c asmeurer python=3.5.0=conda_build_0 gnureadline 
cat "python 3.5.0 conda_build_0" > ~/anaconda/envs/conda-build-python/conda-meta/pinned

and use the conda-build-python environment. gnureadline is needed because I didn't compile readline support correctly (without it, the interactive interpreter won't have readline history support). The pinning step is necessary to prevent further conda commands in that environment from "updating" python to the one from the default repos.

I HIGHLY recommend doing whatever workarounds you do in a separate conda environment. Don't mess with the Python in the root environment.

[mjuric]

@asmeurer Oops, sorry, I didn't realize that part of our Jira is closed (shouldn't be... argh...) -- here's the PDF dump of the link.

Regarding whether changing the id is enough; while I 100% agree with what you're saying (and that a comprehensive fix is desirable), even just setting the id would fix the particular problem we've been fighting for a while now. So far, we've been able to circumvent the lack of id with DYLD_FALLBACK_LIBRARY_PATH hacks, but now SIP on El Capitan is thwarting those. I suspect it may help other packages encountering similar problems.

PS: If the switch to conda-build is imminent, then this may be a non-issue. But (to be honest :) ), our (totally selfish :) ) worry is that if it takes longer than you guys expect, even having the id set (which would hopefully be easy to add to your existing build scripts?) would give us an Anaconda python that our code can build against on El Capitan.

A bird in the hand, etc, etc :).

[ilanschnell]

I have recently done some changes to our internal Anaconda build system, to
call install_name_tool -add_rpath on shared object files during the post processing step. This has solved similar problems we've been seeing. However, since that change was made, Python has not been rebuild. The switch to conda-build for Python itself is not imminent, but I hope rebuilding Python will solve these issues.

[mjuric]

Brilliant, thanks! So we should expect a rebuild no later than Anaconda 2.5 (some ~3 months or so)?

[danielsf]

I would just like to draw attention back to this issue. I just installed anaconda 2.5 on my OSX machine

bash-3.2$ otool -L libpython2.7.dylib 
libpython2.7.dylib:
        libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 476.0.0)
        /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

the lack of @rpath in the loading path is really killing us.

[rmjarvis]

+1 for requesting a resolution to this sooner rather than later.

For GalSim users who are using Anaconda python, we currently recommend a complicated procedure using install_name_tool to fix your broken python dylib file. cf. this StackOverflow question.

[henryiii]

This is affecting me too, I have a CMake Python extension that I manually have to patch to get around the missing rpath (all the other conda libs seem to have it, besides lib python 3.5). I'd like to make it easier to install than that!

[minrk]

It would be really great to get this patched. I'm working with a library (fenics, dolfin, ffc) that does its own compilation of Python modules at runtime, and they can't load libpython.dylib correctly because of this. That means I have to tell users to set DYLD_FALLBACK_LIBRARY_PATH whenever they activate an env with this package, which isn't tenable. I don't see a workaround that can be applied in my conda package. The only other library I've noticed with the same problem is mkl, which also lacks the rpath, so that should get rebuilt as well.

[ccordoba12]

@msarahan, is there something we can do about this one?

[msarahan]

It is up to @ilanschnell when he wants to do this. I am trying to help the conda-forge effort (conda-forge/staged-recipes#370), so that people will have another choice in this matter. Hopefully that work might also make any transition that Ilan makes easier.

[asmeurer]

@minrk a workaround in your conda package could be a post-link script that rewrites the rpath on libpython2.7.dylib/libpython3.5m.dylib.

[danielsf]

I would just like to say that, for the purposes of the LSST group, @asmeurer 's workaround has been deemed unhelpful, as we want to be able to deploy to shared systems where the users installing our package may not have write-access to libpython2.7.dylib. Fixing this in the conda install itself is really the only option.

[henryiii]

No, you can rewrite the path in your binary with a script/command line. Rewriting the path in the lib python binary is great (that’s how I do it, on EVERY anaconda install on mac :’( ), but it requires write access and manipulation of the base anaconda install. The link command inside your executable can be rewritten instead.

On May 11, 2016, at 11:23 AM, danielsf notifications@github.com wrote:

I would just like to say that, for the purposes of the LSST group, @asmeuer 's workaround has been deemed unhelpful, as we want to be able to deploy to shared systems where the users installing our package may not have write-access to libpython2.7.dylib. Fixing this in the conda install itself is really the only option.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub #498 (comment)

[asmeurer]

Do they have write access to the environment? I should point out that to do the workaround I suggested properly, you should break the hardlink for libpython2.7.dylib in the environment first. You should always break hardlinks before editing installed files in-place in conda packages, so that you don't inadvertently break other environments.

[minrk]

@asmeurer cool, thanks, I'll look into the post-link script. I made patch-conda-rpaths as a quick-and-dirty reusable script for doing this. It might even work!

[katyhuff]

+1 just ran into this. The patch from @minrk worked for me.

[ghost]

I have the same issue with libgfortran.3.dylib and self written Fortran code but also with scikits.bvp1lg (a BVP solver bases on Fortran). I tried @minrk's patch using

patch-conda-rpaths ~/anacaonda2/lib/,

however, this gives me a fatal error:

fatal error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: input file: anaconda2/lib/libgcc_ext.10.4.dylib is Mach-O dynamic shared library stub file and can't be changed.

Am I using the patch correctly? Or will there be a solution for OS X soon? Thanks!

[asmeurer]

@lkiewidt, @minrk needs to make his script ignore stub files (conda-build does this).

[minrk]

Thanks, @asmeurer! I didn't know about stub files (still don't, really), but I've pushed a new release of patch-conda-rpaths with the same check as conda-build, so maybe it'll work now.

[asmeurer]

Yeah I don't know anything about them either, but as far as I can tell, ignoring them is the right thing to do.

[mjuric]

@asmeurer: I ran into a problem today that made me think whether (some?) libraries in conda's lib/ directory should have absolute, rather than @rpath-relative install names. Coincidentally, the issue has to do with libgfortran as well.

I tried to build OpenOrb with conda-supplied gfortran from the gcc package. As a part of the build OpenOrb builds a tiny utility to convert files from one format to another, and then runs it. But when it tries to run it, it aborts with the following:

dyld: Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/lynnej/lsstRepos/conda-lsst/miniconda/conda-bld/work/data/JPL_ephemeris/./asc2eph
  Reason: image not found
/bin/sh: line 1: 22267 Trace/BPT trap: 5       nice ./asc2eph --eph-type=405 < ascii.405
make: *** [eph] Error 133

This is not surprising -- running otool -L on asc2eph above shows that it's looking for @rpath/./libgfortran.3.dylib. But as it doesn't have the necessary LC_RPATH set, it doesn't find it (remember, this code is run while the build is still in progress).

The usual remedy for this is to set DYLD_LIBRARY_PATH (or it's FALLBACK sibling). But I'm running on El Capitan, the build system is make (from /usr/bin/make). Therefore, SIP strips out all *DYLD* variables. I did find a horrible way to work around this, but it's not something to be emulated, IMHO.

But this made me thinking... I think having the compiler runtimes (libgfortran, libgcc, etc.) have relative install_names may be the wrong thing to do. This will always lead to problems at build time, while the RPATHs haven't been set by the conda-build's afterburner. And with *DYLD* variables made unreliable by SIP, there's no real workaround. More broadly, this issue goes beyond just the compiler runtimes -- e.g., if I link against openssl that has a @rpath-relative name, and try to run that code before conda-build has fixed it up, I'll have the same problem. This may happen to developers who use anaconda as a part of their developer toolchain who don't build/distribute using conda-build.

I therefore worry whether the strategy to have all libraries have @rpath-relative names is fundamentally problematic. Would it be better to have absolute paths in install_names instead (that's set at package install time)?

[asmeurer]

The rpath trick may require you to add an LC_RPATH (pointing to the lib/ directory) to and changing the install names of your own executables, meaning either building them as conda packages with conda-build, using conda develop, or manually doing it with install_name_tool or with a compilation flag. Actually, if you link against the conda libgfortran, the dependent install name with @rpath will be pulled in automatically.

Using absolute install names is problematic, because it adds complexity to conda install (or to the package, via pre-link script, which has its own issues). By using relative rpaths, the complexity is offloaded to conda build, and the install is simple (just linking the files into place).

[msarahan]

CC @mingwandroid - you're also interested in this terrible game on Mac.

[mjuric]

@asmeurer Right, but the way the gcc package is currently build, you get unexpected results such as:

$ cat hello.f90
program main
  implicit none

  write ( *, '(a)' ) '  Hello, world!'

  stop
end

$ gfortran -o hello hello.f90

$ ./hello
dyld: Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/mjuric/projects/conda_gcc_test/./hello
  Reason: image not found
Trace/BPT trap: 5

This is unexpected/conuterintuitive. The same thing would actually happen if you built a C++ program, except that /usr/lib/libstdc++.6.dylib is found and saves the day (at least until one hits a symbol that isn't found in the older libstdc+++ that OS X ships with, or some other incompatibility).

While I understand the simplicity that @rpath brings elsewhere, maybe a middle ground could be to consider the compilers as a special case? It's sort of weird when a compiler isn't generating functional binaries in simple cases such as the above.

[rmjarvis]

+1 for absolute library paths on Macs.

Anaconda python is by far the hardest python for us to support for GalSim. 90+% of our user installation problems come from Anaconda users on Macs (and they don't make up nearly 90% of our user base). It's almost always a combination of the @rpath stuff, the fact that libpython2.7.dylib doesn't even have that (i.e. this issue), and now the new SIP on El Capitan removing the DYLD_LIBRARY_PATH solution that used to make this doable.

I think RPATH is a great solution on linux (where Anaconda shines, imho), but the Mac OS developers really seem to be steering people toward absolute paths in library files, and bucking that often just leads to problems.

[manodeep]

Came by this thread while trying to sort out my solve my conda build issues. Just in case this proves useful to someone, I have noted my solution for anaconda python on MAC. I simply fix the paths after creating the shared extension. This requires compiling with -headerpad_max_install_names and setting a flag like so:

 ifeq ($(UNAME), Darwin)
      PATH_TO_PYTHON := $(shell which python)
      ifeq (conda, $(findstring conda, $(PATH_TO_PYTHON)))
          FIX_PYTHON_LINK := 1
      endif
endif

And then after building the shared library,

sharedlib: $(PROJECT).so
ifeq ($(FIX_PYTHON_LINK), 1)
      @{ \
              CURRENT_PYTHON_LIB=`otool -L $(PYTHON_LIBRARY) | grep -i python | cut -d " " -f1 | xargs` ; \
              PYTHON_LIB_NAME=$(PYTHON_LIB_BASE).dylib ; \
              LINK_PYTHON_LIB=$(PYTHON_LIBDIR)/$$PYTHON_LIB_NAME ;\
              if [[ "$$CURRENT_PYTHON_LIB" != "$$LINK_PYTHON_LIB" ]] ; then \
                  install_name_tool -change $$CURRENT_PYTHON_LIB $$LINK_PYTHON_LIB $(PYTHON_LIBRARY); \
              fi ;\
      }
endif

where PYTHON_LIBRARY is the C/C++ extension filename PYTHON_LIBRARY := $(PROJECT).so.$(MAJOR).$(MINOR).$(PATCH). The other variables are set from the Makefile using python(3)-config.

 ifeq ($(PYTHON_VERSION_MAJOR), 2)
      PYTHON_CONFIG_EXE:=python-config
 else
      PYTHON_CONFIG_EXE:=python3-config
 endif
 PYTHON_LIBDIR := $(shell $(PYTHON_CONFIG_EXE) --prefix)/lib
 PYTHON_LIBS   := $(shell $(PYTHON_CONFIG_EXE) --libs)
 PYTHON_LINK   := -L$(PYTHON_LIBDIR) $(PYTHON_LIBS) -Xlinker -rpath -Xlinker $(PYTHON_LIBDIR)
 PYTHON_LIB_BASE := $(strip $(subst -l,lib, $(filter -lpython%,$(PYTHON_LIBS))))

These code blocks (taken from https://github.com/manodeep/Corrfunc/) should solve the build/run problems with conda python on OSX. There can be some additional issues involving links with /path/./file.dylib -- these can be solved in a similar manner with install_name_tool by simply removing the extraneous ./

[IrinaMax]

How to uninstall anaconda from my MacBook? i want to install Tensorflow to work from R, but pip automaticly put it in anaconda directory and its not tensorFlow not working in R.

[henryiii]

This is probably not the very best place for that question, but I do happen to know the answer :-) You don't need to uninstall anaconda, all you need to do is remove it from your path environment variable. I believe, on my Mac it set in the .profile file in your home directory. Remove or comment that line out to Use the system Python. Add back to your path, to get anaconda back.