Rebuild Python with conda build on OS X #498

Closed
asmeurer opened this Issue Oct 29, 2015 · 33 comments

Comments

Projects
None yet
@asmeurer
Contributor

asmeurer commented Oct 29, 2015

I was kind of hoping this would have happened by now. Conda build on OS X adds @rpath to the install name on OS X and adds an LC_RPATH. This makes it possible to build things that link against libraries and have them work without having to call install_name_tool (or build them with conda build, which calls it for you).

An example of this: sympy.utilities.autowrap generates a Fortran extension module on the fly using f2py (part of NumPy). Create a conda environment on OS X with python sympy numpy gcc and run

from sympy import *
x = symbols('x')
from sympy.utilities.autowrap import autowrap
autowrap(sin(x), backend='f2py', tempdir='.')

This will attempt to use f2py to generate an extension module in the current directory using fortran (with gfortran from the gcc package) and import it. On OS X, you get an error like

ImportError: dlopen(/Users/aaronmeurer/Documents/Python/sympy/sympy/wrapper_module_0.cpython-35m-darwin.so, 2): Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/aaronmeurer/Documents/Python/sympy/sympy/wrapper_module_0.cpython-35m-darwin.so
  Reason: image not found

What this means is that the extension module linked against libgfortran from the gcc package which has @rpath/./libgfortran.3.dylib as its install name. The @rpath should reference the LC_RPATH of some loaded library, which would point to the lib/ directory of the environment, but nothing already loaded has an LC_RPATH, in particular, Python itself does not. Effectively, it's impossible to import this extension module without "fixing" it with install_name_tool, which is undesirable for an automated tool like autowrap.

To fix this, Python should be rebuilt with conda-build. This will add the LC_RPATH to libpython3.5m.dylib and the dyld loader will be able to find the extension module.

I tested this myself, by building the python-3.5 recipe in conda-recipe (I added the build string conda_build_0 so you can tell the difference).

You can test with

conda create -n test -c asmeurer python=3.5.0=conda_build_0 sympy numpy gcc

and run the above code. It will work. (note that if you don't pin that Python build in the test environment further conda commands will "update" it to the standard one, since that has a newer build string)

You can see the difference with conda inspect objects python (or just using otool -L and otool -l).

@ccordoba12

This comment has been minimized.

Show comment
Hide comment
@ccordoba12

ccordoba12 Oct 29, 2015

Letting @msarahan know about this one. We'll try to migrate all recipes to be conda-build ones after we release Anaconda 2.4.

Letting @msarahan know about this one. We'll try to migrate all recipes to be conda-build ones after we release Anaconda 2.4.

@groutr

This comment has been minimized.

Show comment
Hide comment
@groutr

groutr Oct 30, 2015

@groutr, I want to track this issue as well.

groutr commented Oct 30, 2015

@groutr, I want to track this issue as well.

@danielsf

This comment has been minimized.

Show comment
Hide comment
@danielsf

danielsf Dec 2, 2015

Just adding a +1 to the desirability of this fix. I work on the software development team for the Large Synoptic Survey Telescope. We have been having trouble making our build system work with anaconda because of the lack of @rpath in the loading address for libpython*.dylib in Anaconda.

danielsf commented Dec 2, 2015

Just adding a +1 to the desirability of this fix. I work on the software development team for the Large Synoptic Survey Telescope. We have been having trouble making our build system work with anaconda because of the lack of @rpath in the loading address for libpython*.dylib in Anaconda.

@mjuric

This comment has been minimized.

Show comment
Hide comment
@mjuric

mjuric Dec 2, 2015

@msarahan et al -- would a short-term fix be possible, in the form of adding a line such as:

install_name_tool -id @rpath/libpython2.7.dylib libpython2.7.dylib

to your existing Python build script? That seems to fix some of our build issues.

mjuric commented Dec 2, 2015

@msarahan et al -- would a short-term fix be possible, in the form of adding a line such as:

install_name_tool -id @rpath/libpython2.7.dylib libpython2.7.dylib

to your existing Python build script? That seems to fix some of our build issues.

@msarahan

This comment has been minimized.

Show comment
Hide comment
@msarahan

msarahan Dec 2, 2015

Member

Ping @ilanschnell on this. He needs to make the call on this issue.

Member

msarahan commented Dec 2, 2015

Ping @ilanschnell on this. He needs to make the call on this issue.

@asmeurer

This comment has been minimized.

Show comment
Hide comment
@asmeurer

asmeurer Dec 2, 2015

Contributor

@mjuric that page can't be viewed without a login.

Just changing the id isn't enough. You also need to set the LC_RPATH on the dylib. It may work anyway for some cases if it can find the LC_RPATH from another dylib in the load chain, but in general it is needed. conda build does this all automatically (and also changes about a dozen other object files that are part of the python package).

As a workaround, I built python with conda build, as noted in the OP (Python 3.5 only). You can use that with

conda create -n conda-build-python -c asmeurer python=3.5.0=conda_build_0 gnureadline 
cat "python 3.5.0 conda_build_0" > ~/anaconda/envs/conda-build-python/conda-meta/pinned

and use the conda-build-python environment. gnureadline is needed because I didn't compile readline support correctly (without it, the interactive interpreter won't have readline history support). The pinning step is necessary to prevent further conda commands in that environment from "updating" python to the one from the default repos.

I HIGHLY recommend doing whatever workarounds you do in a separate conda environment. Don't mess with the Python in the root environment.

Contributor

asmeurer commented Dec 2, 2015

@mjuric that page can't be viewed without a login.

Just changing the id isn't enough. You also need to set the LC_RPATH on the dylib. It may work anyway for some cases if it can find the LC_RPATH from another dylib in the load chain, but in general it is needed. conda build does this all automatically (and also changes about a dozen other object files that are part of the python package).

As a workaround, I built python with conda build, as noted in the OP (Python 3.5 only). You can use that with

conda create -n conda-build-python -c asmeurer python=3.5.0=conda_build_0 gnureadline 
cat "python 3.5.0 conda_build_0" > ~/anaconda/envs/conda-build-python/conda-meta/pinned

and use the conda-build-python environment. gnureadline is needed because I didn't compile readline support correctly (without it, the interactive interpreter won't have readline history support). The pinning step is necessary to prevent further conda commands in that environment from "updating" python to the one from the default repos.

I HIGHLY recommend doing whatever workarounds you do in a separate conda environment. Don't mess with the Python in the root environment.

@mjuric

This comment has been minimized.

Show comment
Hide comment
@mjuric

mjuric Dec 3, 2015

@asmeurer Oops, sorry, I didn't realize that part of our Jira is closed (shouldn't be... argh...) -- here's the PDF dump of the link.

Regarding whether changing the id is enough; while I 100% agree with what you're saying (and that a comprehensive fix is desirable), even just setting the id would fix the particular problem we've been fighting for a while now. So far, we've been able to circumvent the lack of id with DYLD_FALLBACK_LIBRARY_PATH hacks, but now SIP on El Capitan is thwarting those. I suspect it may help other packages encountering similar problems.

PS: If the switch to conda-build is imminent, then this may be a non-issue. But (to be honest :) ), our (totally selfish :) ) worry is that if it takes longer than you guys expect, even having the id set (which would hopefully be easy to add to your existing build scripts?) would give us an Anaconda python that our code can build against on El Capitan.

A bird in the hand, etc, etc :).

mjuric commented Dec 3, 2015

@asmeurer Oops, sorry, I didn't realize that part of our Jira is closed (shouldn't be... argh...) -- here's the PDF dump of the link.

Regarding whether changing the id is enough; while I 100% agree with what you're saying (and that a comprehensive fix is desirable), even just setting the id would fix the particular problem we've been fighting for a while now. So far, we've been able to circumvent the lack of id with DYLD_FALLBACK_LIBRARY_PATH hacks, but now SIP on El Capitan is thwarting those. I suspect it may help other packages encountering similar problems.

PS: If the switch to conda-build is imminent, then this may be a non-issue. But (to be honest :) ), our (totally selfish :) ) worry is that if it takes longer than you guys expect, even having the id set (which would hopefully be easy to add to your existing build scripts?) would give us an Anaconda python that our code can build against on El Capitan.

A bird in the hand, etc, etc :).

@ilanschnell

This comment has been minimized.

Show comment
Hide comment
@ilanschnell

ilanschnell Dec 3, 2015

Member

I have recently done some changes to our internal Anaconda build system, to
call install_name_tool -add_rpath on shared object files during the post processing step. This has solved similar problems we've been seeing. However, since that change was made, Python has not been rebuild. The switch to conda-build for Python itself is not imminent, but I hope rebuilding Python will solve these issues.

Member

ilanschnell commented Dec 3, 2015

I have recently done some changes to our internal Anaconda build system, to
call install_name_tool -add_rpath on shared object files during the post processing step. This has solved similar problems we've been seeing. However, since that change was made, Python has not been rebuild. The switch to conda-build for Python itself is not imminent, but I hope rebuilding Python will solve these issues.

@mjuric

This comment has been minimized.

Show comment
Hide comment
@mjuric

mjuric Dec 3, 2015

Brilliant, thanks! So we should expect a rebuild no later than Anaconda 2.5 (some ~3 months or so)?

mjuric commented Dec 3, 2015

Brilliant, thanks! So we should expect a rebuild no later than Anaconda 2.5 (some ~3 months or so)?

@daler daler referenced this issue in bioconda/bioconda-recipes Dec 18, 2015

Closed

Help with bioconductor-rhdf5 on OSX #476

@connie connie referenced this issue in ReactionMechanismGenerator/PyDAS Dec 31, 2015

Open

Build problems with Anaconda on Mac OSX #8

@danielsf

This comment has been minimized.

Show comment
Hide comment
@danielsf

danielsf Feb 5, 2016

I would just like to draw attention back to this issue. I just installed anaconda 2.5 on my OSX machine

bash-3.2$ otool -L libpython2.7.dylib 
libpython2.7.dylib:
        libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 476.0.0)
        /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

the lack of @rpath in the loading path is really killing us.

danielsf commented Feb 5, 2016

I would just like to draw attention back to this issue. I just installed anaconda 2.5 on my OSX machine

bash-3.2$ otool -L libpython2.7.dylib 
libpython2.7.dylib:
        libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 476.0.0)
        /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

the lack of @rpath in the loading path is really killing us.

@mjuric mjuric referenced this issue in lsst-dm/spie-2016-dm-astropy Mar 15, 2016

Open

Comments from DESC Pundits #1

@rmjarvis

This comment has been minimized.

Show comment
Hide comment
@rmjarvis

rmjarvis Mar 15, 2016

+1 for requesting a resolution to this sooner rather than later.

For GalSim users who are using Anaconda python, we currently recommend a complicated procedure using install_name_tool to fix your broken python dylib file. cf. this StackOverflow question.

+1 for requesting a resolution to this sooner rather than later.

For GalSim users who are using Anaconda python, we currently recommend a complicated procedure using install_name_tool to fix your broken python dylib file. cf. this StackOverflow question.

@henryiii

This comment has been minimized.

Show comment
Hide comment
@henryiii

henryiii Apr 7, 2016

This is affecting me too, I have a CMake Python extension that I manually have to patch to get around the missing rpath (all the other conda libs seem to have it, besides lib python 3.5). I'd like to make it easier to install than that!

henryiii commented Apr 7, 2016

This is affecting me too, I have a CMake Python extension that I manually have to patch to get around the missing rpath (all the other conda libs seem to have it, besides lib python 3.5). I'd like to make it easier to install than that!

mjuric added a commit to lsst/galsim that referenced this issue Apr 10, 2016

Non-invasive fix for building on El Capitan with Anaconda python
Detect if the libpython*.dylib library's install_name is incorrect
(relative), and:

 * pass DYLD_FALLBACK_LIBRARY_PATH to GalSim's scons, to enable the build to proceed
 * fix the install_name in _galsim.so after the build succeeds.

This enables us to work around Anaconda Python's bug described in:

	ContinuumIO/anaconda-issues#498

and successfully build+run GalSim.

mjuric added a commit to mjuric/GalSim that referenced this issue Apr 10, 2016

Pass DYLD_FALLBACK_LIBRARY_PATH to scons
This makes it easier to build GalSim against buggy Python builds (such as
the one shipping with Anaconda; see
ContinuumIO/anaconda-issues#498).

@mjuric mjuric referenced this issue in GalSim-developers/GalSim Apr 10, 2016

Merged

Pass DYLD_FALLBACK_LIBRARY_PATH to scons #737

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk May 11, 2016

It would be really great to get this patched. I'm working with a library (fenics, dolfin, ffc) that does its own compilation of Python modules at runtime, and they can't load libpython.dylib correctly because of this. That means I have to tell users to set DYLD_FALLBACK_LIBRARY_PATH whenever they activate an env with this package, which isn't tenable. I don't see a workaround that can be applied in my conda package. The only other library I've noticed with the same problem is mkl, which also lacks the rpath, so that should get rebuilt as well.

minrk commented May 11, 2016

It would be really great to get this patched. I'm working with a library (fenics, dolfin, ffc) that does its own compilation of Python modules at runtime, and they can't load libpython.dylib correctly because of this. That means I have to tell users to set DYLD_FALLBACK_LIBRARY_PATH whenever they activate an env with this package, which isn't tenable. I don't see a workaround that can be applied in my conda package. The only other library I've noticed with the same problem is mkl, which also lacks the rpath, so that should get rebuilt as well.

@ccordoba12

This comment has been minimized.

Show comment
Hide comment
@ccordoba12

ccordoba12 May 11, 2016

@msarahan, is there something we can do about this one?

@msarahan, is there something we can do about this one?

@msarahan

This comment has been minimized.

Show comment
Hide comment
@msarahan

msarahan May 11, 2016

Member

It is up to @ilanschnell when he wants to do this. I am trying to help the conda-forge effort (conda-forge/staged-recipes#370), so that people will have another choice in this matter. Hopefully that work might also make any transition that Ilan makes easier.

Member

msarahan commented May 11, 2016

It is up to @ilanschnell when he wants to do this. I am trying to help the conda-forge effort (conda-forge/staged-recipes#370), so that people will have another choice in this matter. Hopefully that work might also make any transition that Ilan makes easier.

@asmeurer

This comment has been minimized.

Show comment
Hide comment
@asmeurer

asmeurer May 11, 2016

Contributor

@minrk a workaround in your conda package could be a post-link script that rewrites the rpath on libpython2.7.dylib/libpython3.5m.dylib.

Contributor

asmeurer commented May 11, 2016

@minrk a workaround in your conda package could be a post-link script that rewrites the rpath on libpython2.7.dylib/libpython3.5m.dylib.

@danielsf

This comment has been minimized.

Show comment
Hide comment
@danielsf

danielsf May 11, 2016

I would just like to say that, for the purposes of the LSST group, @asmeurer 's workaround has been deemed unhelpful, as we want to be able to deploy to shared systems where the users installing our package may not have write-access to libpython2.7.dylib. Fixing this in the conda install itself is really the only option.

danielsf commented May 11, 2016

I would just like to say that, for the purposes of the LSST group, @asmeurer 's workaround has been deemed unhelpful, as we want to be able to deploy to shared systems where the users installing our package may not have write-access to libpython2.7.dylib. Fixing this in the conda install itself is really the only option.

@henryiii

This comment has been minimized.

Show comment
Hide comment
@henryiii

henryiii May 11, 2016

No, you can rewrite the path in your binary with a script/command line. Rewriting the path in the lib python binary is great (that’s how I do it, on EVERY anaconda install on mac :’( ), but it requires write access and manipulation of the base anaconda install. The link command inside your executable can be rewritten instead.

On May 11, 2016, at 11:23 AM, danielsf notifications@github.com wrote:

I would just like to say that, for the purposes of the LSST group, @asmeuer 's workaround has been deemed unhelpful, as we want to be able to deploy to shared systems where the users installing our package may not have write-access to libpython2.7.dylib. Fixing this in the conda install itself is really the only option.


You are receiving this because you commented.
Reply to this email directly or view it on GitHub #498 (comment)

No, you can rewrite the path in your binary with a script/command line. Rewriting the path in the lib python binary is great (that’s how I do it, on EVERY anaconda install on mac :’( ), but it requires write access and manipulation of the base anaconda install. The link command inside your executable can be rewritten instead.

On May 11, 2016, at 11:23 AM, danielsf notifications@github.com wrote:

I would just like to say that, for the purposes of the LSST group, @asmeuer 's workaround has been deemed unhelpful, as we want to be able to deploy to shared systems where the users installing our package may not have write-access to libpython2.7.dylib. Fixing this in the conda install itself is really the only option.


You are receiving this because you commented.
Reply to this email directly or view it on GitHub #498 (comment)

@asmeurer

This comment has been minimized.

Show comment
Hide comment
@asmeurer

asmeurer May 11, 2016

Contributor

Do they have write access to the environment? I should point out that to do the workaround I suggested properly, you should break the hardlink for libpython2.7.dylib in the environment first. You should always break hardlinks before editing installed files in-place in conda packages, so that you don't inadvertently break other environments.

Contributor

asmeurer commented May 11, 2016

Do they have write access to the environment? I should point out that to do the workaround I suggested properly, you should break the hardlink for libpython2.7.dylib in the environment first. You should always break hardlinks before editing installed files in-place in conda packages, so that you don't inadvertently break other environments.

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk May 12, 2016

@asmeurer cool, thanks, I'll look into the post-link script. I made patch-conda-rpaths as a quick-and-dirty reusable script for doing this. It might even work!

minrk commented May 12, 2016

@asmeurer cool, thanks, I'll look into the post-link script. I made patch-conda-rpaths as a quick-and-dirty reusable script for doing this. It might even work!

@katyhuff

This comment has been minimized.

Show comment
Hide comment
@katyhuff

katyhuff May 13, 2016

+1 just ran into this. The patch from @minrk worked for me.

+1 just ran into this. The patch from @minrk worked for me.

@lkiewidt

This comment has been minimized.

Show comment
Hide comment
@lkiewidt

lkiewidt Jun 14, 2016

I have the same issue with libgfortran.3.dylib and self written Fortran code but also with scikits.bvp1lg (a BVP solver bases on Fortran). I tried @minrk's patch using

patch-conda-rpaths ~/anacaonda2/lib/,

however, this gives me a fatal error:

fatal error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: input file: anaconda2/lib/libgcc_ext.10.4.dylib is Mach-O dynamic shared library stub file and can't be changed.

Am I using the patch correctly? Or will there be a solution for OS X soon? Thanks!

I have the same issue with libgfortran.3.dylib and self written Fortran code but also with scikits.bvp1lg (a BVP solver bases on Fortran). I tried @minrk's patch using

patch-conda-rpaths ~/anacaonda2/lib/,

however, this gives me a fatal error:

fatal error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: input file: anaconda2/lib/libgcc_ext.10.4.dylib is Mach-O dynamic shared library stub file and can't be changed.

Am I using the patch correctly? Or will there be a solution for OS X soon? Thanks!

@asmeurer

This comment has been minimized.

Show comment
Hide comment
@asmeurer

asmeurer Jun 14, 2016

Contributor

@lkiewidt, @minrk needs to make his script ignore stub files (conda-build does this).

Contributor

asmeurer commented Jun 14, 2016

@lkiewidt, @minrk needs to make his script ignore stub files (conda-build does this).

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Jun 14, 2016

Thanks, @asmeurer! I didn't know about stub files (still don't, really), but I've pushed a new release of patch-conda-rpaths with the same check as conda-build, so maybe it'll work now.

minrk commented Jun 14, 2016

Thanks, @asmeurer! I didn't know about stub files (still don't, really), but I've pushed a new release of patch-conda-rpaths with the same check as conda-build, so maybe it'll work now.

@asmeurer

This comment has been minimized.

Show comment
Hide comment
@asmeurer

asmeurer Jun 14, 2016

Contributor

Yeah I don't know anything about them either, but as far as I can tell, ignoring them is the right thing to do.

Contributor

asmeurer commented Jun 14, 2016

Yeah I don't know anything about them either, but as far as I can tell, ignoring them is the right thing to do.

@mjuric

This comment has been minimized.

Show comment
Hide comment
@mjuric

mjuric Jun 15, 2016

@asmeurer: I ran into a problem today that made me think whether (some?) libraries in conda's lib/ directory should have absolute, rather than @rpath-relative install names. Coincidentally, the issue has to do with libgfortran as well.

I tried to build OpenOrb with conda-supplied gfortran from the gcc package. As a part of the build OpenOrb builds a tiny utility to convert files from one format to another, and then runs it. But when it tries to run it, it aborts with the following:

dyld: Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/lynnej/lsstRepos/conda-lsst/miniconda/conda-bld/work/data/JPL_ephemeris/./asc2eph
  Reason: image not found
/bin/sh: line 1: 22267 Trace/BPT trap: 5       nice ./asc2eph --eph-type=405 < ascii.405
make: *** [eph] Error 133

This is not surprising -- running otool -L on asc2eph above shows that it's looking for @rpath/./libgfortran.3.dylib. But as it doesn't have the necessary LC_RPATH set, it doesn't find it (remember, this code is run while the build is still in progress).

The usual remedy for this is to set DYLD_LIBRARY_PATH (or it's FALLBACK sibling). But I'm running on El Capitan, the build system is make (from /usr/bin/make). Therefore, SIP strips out all *DYLD* variables. I did find a horrible way to work around this, but it's not something to be emulated, IMHO.

But this made me thinking... I think having the compiler runtimes (libgfortran, libgcc, etc.) have relative install_names may be the wrong thing to do. This will always lead to problems at build time, while the RPATHs haven't been set by the conda-build's afterburner. And with *DYLD* variables made unreliable by SIP, there's no real workaround. More broadly, this issue goes beyond just the compiler runtimes -- e.g., if I link against openssl that has a @rpath-relative name, and try to run that code before conda-build has fixed it up, I'll have the same problem. This may happen to developers who use anaconda as a part of their developer toolchain who don't build/distribute using conda-build.

I therefore worry whether the strategy to have all libraries have @rpath-relative names is fundamentally problematic. Would it be better to have absolute paths in install_names instead (that's set at package install time)?

mjuric commented Jun 15, 2016

@asmeurer: I ran into a problem today that made me think whether (some?) libraries in conda's lib/ directory should have absolute, rather than @rpath-relative install names. Coincidentally, the issue has to do with libgfortran as well.

I tried to build OpenOrb with conda-supplied gfortran from the gcc package. As a part of the build OpenOrb builds a tiny utility to convert files from one format to another, and then runs it. But when it tries to run it, it aborts with the following:

dyld: Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/lynnej/lsstRepos/conda-lsst/miniconda/conda-bld/work/data/JPL_ephemeris/./asc2eph
  Reason: image not found
/bin/sh: line 1: 22267 Trace/BPT trap: 5       nice ./asc2eph --eph-type=405 < ascii.405
make: *** [eph] Error 133

This is not surprising -- running otool -L on asc2eph above shows that it's looking for @rpath/./libgfortran.3.dylib. But as it doesn't have the necessary LC_RPATH set, it doesn't find it (remember, this code is run while the build is still in progress).

The usual remedy for this is to set DYLD_LIBRARY_PATH (or it's FALLBACK sibling). But I'm running on El Capitan, the build system is make (from /usr/bin/make). Therefore, SIP strips out all *DYLD* variables. I did find a horrible way to work around this, but it's not something to be emulated, IMHO.

But this made me thinking... I think having the compiler runtimes (libgfortran, libgcc, etc.) have relative install_names may be the wrong thing to do. This will always lead to problems at build time, while the RPATHs haven't been set by the conda-build's afterburner. And with *DYLD* variables made unreliable by SIP, there's no real workaround. More broadly, this issue goes beyond just the compiler runtimes -- e.g., if I link against openssl that has a @rpath-relative name, and try to run that code before conda-build has fixed it up, I'll have the same problem. This may happen to developers who use anaconda as a part of their developer toolchain who don't build/distribute using conda-build.

I therefore worry whether the strategy to have all libraries have @rpath-relative names is fundamentally problematic. Would it be better to have absolute paths in install_names instead (that's set at package install time)?

@asmeurer

This comment has been minimized.

Show comment
Hide comment
@asmeurer

asmeurer Jun 15, 2016

Contributor

The rpath trick may require you to add an LC_RPATH (pointing to the lib/ directory) to and changing the install names of your own executables, meaning either building them as conda packages with conda-build, using conda develop, or manually doing it with install_name_tool or with a compilation flag. Actually, if you link against the conda libgfortran, the dependent install name with @rpath will be pulled in automatically.

Using absolute install names is problematic, because it adds complexity to conda install (or to the package, via pre-link script, which has its own issues). By using relative rpaths, the complexity is offloaded to conda build, and the install is simple (just linking the files into place).

Contributor

asmeurer commented Jun 15, 2016

The rpath trick may require you to add an LC_RPATH (pointing to the lib/ directory) to and changing the install names of your own executables, meaning either building them as conda packages with conda-build, using conda develop, or manually doing it with install_name_tool or with a compilation flag. Actually, if you link against the conda libgfortran, the dependent install name with @rpath will be pulled in automatically.

Using absolute install names is problematic, because it adds complexity to conda install (or to the package, via pre-link script, which has its own issues). By using relative rpaths, the complexity is offloaded to conda build, and the install is simple (just linking the files into place).

@msarahan

This comment has been minimized.

Show comment
Hide comment
@msarahan

msarahan Jun 15, 2016

Member

CC @mingwandroid - you're also interested in this terrible game on Mac.

Member

msarahan commented Jun 15, 2016

CC @mingwandroid - you're also interested in this terrible game on Mac.

@mjuric

This comment has been minimized.

Show comment
Hide comment
@mjuric

mjuric Jun 15, 2016

@asmeurer Right, but the way the gcc package is currently build, you get unexpected results such as:

$ cat hello.f90
program main
  implicit none

  write ( *, '(a)' ) '  Hello, world!'

  stop
end

$ gfortran -o hello hello.f90

$ ./hello
dyld: Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/mjuric/projects/conda_gcc_test/./hello
  Reason: image not found
Trace/BPT trap: 5

This is unexpected/conuterintuitive. The same thing would actually happen if you built a C++ program, except that /usr/lib/libstdc++.6.dylib is found and saves the day (at least until one hits a symbol that isn't found in the older libstdc+++ that OS X ships with, or some other incompatibility).

While I understand the simplicity that @rpath brings elsewhere, maybe a middle ground could be to consider the compilers as a special case? It's sort of weird when a compiler isn't generating functional binaries in simple cases such as the above.

mjuric commented Jun 15, 2016

@asmeurer Right, but the way the gcc package is currently build, you get unexpected results such as:

$ cat hello.f90
program main
  implicit none

  write ( *, '(a)' ) '  Hello, world!'

  stop
end

$ gfortran -o hello hello.f90

$ ./hello
dyld: Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/mjuric/projects/conda_gcc_test/./hello
  Reason: image not found
Trace/BPT trap: 5

This is unexpected/conuterintuitive. The same thing would actually happen if you built a C++ program, except that /usr/lib/libstdc++.6.dylib is found and saves the day (at least until one hits a symbol that isn't found in the older libstdc+++ that OS X ships with, or some other incompatibility).

While I understand the simplicity that @rpath brings elsewhere, maybe a middle ground could be to consider the compilers as a special case? It's sort of weird when a compiler isn't generating functional binaries in simple cases such as the above.

@rmjarvis

This comment has been minimized.

Show comment
Hide comment
@rmjarvis

rmjarvis Jun 15, 2016

+1 for absolute library paths on Macs.

Anaconda python is by far the hardest python for us to support for GalSim. 90+% of our user installation problems come from Anaconda users on Macs (and they don't make up nearly 90% of our user base). It's almost always a combination of the @rpath stuff, the fact that libpython2.7.dylib doesn't even have that (i.e. this issue), and now the new SIP on El Capitan removing the DYLD_LIBRARY_PATH solution that used to make this doable.

I think RPATH is a great solution on linux (where Anaconda shines, imho), but the Mac OS developers really seem to be steering people toward absolute paths in library files, and bucking that often just leads to problems.

+1 for absolute library paths on Macs.

Anaconda python is by far the hardest python for us to support for GalSim. 90+% of our user installation problems come from Anaconda users on Macs (and they don't make up nearly 90% of our user base). It's almost always a combination of the @rpath stuff, the fact that libpython2.7.dylib doesn't even have that (i.e. this issue), and now the new SIP on El Capitan removing the DYLD_LIBRARY_PATH solution that used to make this doable.

I think RPATH is a great solution on linux (where Anaconda shines, imho), but the Mac OS developers really seem to be steering people toward absolute paths in library files, and bucking that often just leads to problems.

@kalefranz kalefranz referenced this issue in conda-forge/staged-recipes Aug 19, 2016

Closed

Patch and add virtualenv #1139

@asmeurer asmeurer referenced this issue in conda-forge/staged-recipes Aug 19, 2016

Merged

Moar Psycopg2 #1162

@ocefpaf ocefpaf referenced this issue in conda-forge/pyferret-feedstock Sep 13, 2016

Closed

OS X binaries are failing #15

@manodeep

This comment has been minimized.

Show comment
Hide comment
@manodeep

manodeep Sep 21, 2016

Came by this thread while trying to sort out my solve my conda build issues. Just in case this proves useful to someone, I have noted my solution for anaconda python on MAC. I simply fix the paths after creating the shared extension. This requires compiling with -headerpad_max_install_names and setting a flag like so:

 ifeq ($(UNAME), Darwin)
      PATH_TO_PYTHON := $(shell which python)
      ifeq (conda, $(findstring conda, $(PATH_TO_PYTHON)))
          FIX_PYTHON_LINK := 1
      endif
endif

And then after building the shared library,

sharedlib: $(PROJECT).so
ifeq ($(FIX_PYTHON_LINK), 1)
      @{ \
              CURRENT_PYTHON_LIB=`otool -L $(PYTHON_LIBRARY) | grep -i python | cut -d " " -f1 | xargs` ; \
              PYTHON_LIB_NAME=$(PYTHON_LIB_BASE).dylib ; \
              LINK_PYTHON_LIB=$(PYTHON_LIBDIR)/$$PYTHON_LIB_NAME ;\
              if [[ "$$CURRENT_PYTHON_LIB" != "$$LINK_PYTHON_LIB" ]] ; then \
                  install_name_tool -change $$CURRENT_PYTHON_LIB $$LINK_PYTHON_LIB $(PYTHON_LIBRARY); \
              fi ;\
      }
endif

where PYTHON_LIBRARY is the C/C++ extension filename PYTHON_LIBRARY := $(PROJECT).so.$(MAJOR).$(MINOR).$(PATCH). The other variables are set from the Makefile using python(3)-config.

 ifeq ($(PYTHON_VERSION_MAJOR), 2)
      PYTHON_CONFIG_EXE:=python-config
 else
      PYTHON_CONFIG_EXE:=python3-config
 endif
 PYTHON_LIBDIR := $(shell $(PYTHON_CONFIG_EXE) --prefix)/lib
 PYTHON_LIBS   := $(shell $(PYTHON_CONFIG_EXE) --libs)
 PYTHON_LINK   := -L$(PYTHON_LIBDIR) $(PYTHON_LIBS) -Xlinker -rpath -Xlinker $(PYTHON_LIBDIR)
 PYTHON_LIB_BASE := $(strip $(subst -l,lib, $(filter -lpython%,$(PYTHON_LIBS))))

These code blocks (taken from https://github.com/manodeep/Corrfunc/) should solve the build/run problems with conda python on OSX. There can be some additional issues involving links with /path/./file.dylib -- these can be solved in a similar manner with install_name_tool by simply removing the extraneous ./

Came by this thread while trying to sort out my solve my conda build issues. Just in case this proves useful to someone, I have noted my solution for anaconda python on MAC. I simply fix the paths after creating the shared extension. This requires compiling with -headerpad_max_install_names and setting a flag like so:

 ifeq ($(UNAME), Darwin)
      PATH_TO_PYTHON := $(shell which python)
      ifeq (conda, $(findstring conda, $(PATH_TO_PYTHON)))
          FIX_PYTHON_LINK := 1
      endif
endif

And then after building the shared library,

sharedlib: $(PROJECT).so
ifeq ($(FIX_PYTHON_LINK), 1)
      @{ \
              CURRENT_PYTHON_LIB=`otool -L $(PYTHON_LIBRARY) | grep -i python | cut -d " " -f1 | xargs` ; \
              PYTHON_LIB_NAME=$(PYTHON_LIB_BASE).dylib ; \
              LINK_PYTHON_LIB=$(PYTHON_LIBDIR)/$$PYTHON_LIB_NAME ;\
              if [[ "$$CURRENT_PYTHON_LIB" != "$$LINK_PYTHON_LIB" ]] ; then \
                  install_name_tool -change $$CURRENT_PYTHON_LIB $$LINK_PYTHON_LIB $(PYTHON_LIBRARY); \
              fi ;\
      }
endif

where PYTHON_LIBRARY is the C/C++ extension filename PYTHON_LIBRARY := $(PROJECT).so.$(MAJOR).$(MINOR).$(PATCH). The other variables are set from the Makefile using python(3)-config.

 ifeq ($(PYTHON_VERSION_MAJOR), 2)
      PYTHON_CONFIG_EXE:=python-config
 else
      PYTHON_CONFIG_EXE:=python3-config
 endif
 PYTHON_LIBDIR := $(shell $(PYTHON_CONFIG_EXE) --prefix)/lib
 PYTHON_LIBS   := $(shell $(PYTHON_CONFIG_EXE) --libs)
 PYTHON_LINK   := -L$(PYTHON_LIBDIR) $(PYTHON_LIBS) -Xlinker -rpath -Xlinker $(PYTHON_LIBDIR)
 PYTHON_LIB_BASE := $(strip $(subst -l,lib, $(filter -lpython%,$(PYTHON_LIBS))))

These code blocks (taken from https://github.com/manodeep/Corrfunc/) should solve the build/run problems with conda python on OSX. There can be some additional issues involving links with /path/./file.dylib -- these can be solved in a similar manner with install_name_tool by simply removing the extraneous ./

@jjallaire jjallaire referenced this issue in rstudio/tensorflow Sep 29, 2016

Closed

installation issue with Anaconda on Mac #8

@IrinaMax

This comment has been minimized.

Show comment
Hide comment
@IrinaMax

IrinaMax Jan 11, 2017

How to uninstall anaconda from my MacBook? i want to install Tensorflow to work from R, but pip automaticly put it in anaconda directory and its not tensorFlow not working in R.

How to uninstall anaconda from my MacBook? i want to install Tensorflow to work from R, but pip automaticly put it in anaconda directory and its not tensorFlow not working in R.

@henryiii

This comment has been minimized.

Show comment
Hide comment
@henryiii

henryiii Jan 11, 2017

This is probably not the very best place for that question, but I do happen to know the answer :-) You don't need to uninstall anaconda, all you need to do is remove it from your path environment variable. I believe, on my Mac it set in the .profile file in your home directory. Remove or comment that line out to Use the system Python. Add back to your path, to get anaconda back.

This is probably not the very best place for that question, but I do happen to know the answer :-) You don't need to uninstall anaconda, all you need to do is remove it from your path environment variable. I believe, on my Mac it set in the .profile file in your home directory. Remove or comment that line out to Use the system Python. Add back to your path, to get anaconda back.

@acaprez acaprez referenced this issue in bioconda/bioconda-recipes Jan 24, 2017

Merged

Add OSX build for bioconductor-rhtslib. #3379

2 of 4 tasks complete

@csoja csoja closed this Nov 13, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment