Parallel build causing problems with egg installs overwriting easy-install.pth

[durack1]

I've just noticed that a module that installs using an egg (gsw-3.0.3-py2.7.egg):

[me@ocean ~]$ more /usr/local/uvcdat/2015-12-21_all/lib/python2.7/site-packages/easy-install.pth 
import sys; sys.__plen = len(sys.path)
./setuptools-17.1.1-py2.7.egg
./pip-7.1.0-py2.7.egg
./six-1.9.0-py2.7.egg
./singledispatch-3.4.0.3-py2.7.egg
./cffi-0.8.2-py2.7-linux-x86_64.egg
./cryptography-0.4-py2.7-linux-x86_64.egg
./pyOpenSSL-0.14-py2.7.egg
./MyProxyClient-1.3.0-py2.7.egg
./numexpr-2.2.2-py2.7-linux-x86_64.egg
./matplotlib-1.5.0-py2.7-linux-x86_64.egg
./rpy2-2.6.0-py2.7-linux-x86_64.egg
./Jinja2-2.7-py2.7.egg
./MarkupSafe-0.18-py2.7-linux-x86_64.egg
./Sphinx-1.2.2-py2.7.egg
./Pygments-1.6-py2.7.egg
./tornado-3.1-py2.7.egg
./LEPL-5.1.3-py2.7.egg
./windspharm-1.3.x-py2.7.egg
./gsw-3.0.3-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p
+len(new)

Was not correctly installed, as duplicate writes to lib/python2.7/site-packages/easy-install.pth were occurring, and as gsw-3.0.3-py2.7.egg wasn't last it's path wasn't added permanently to easy-install.pth (I added it back in and I can now load it as an installed package)

@aashish24 @doutriaux1 this needs some thinking and attention to solve these intermittent build conflicts.. Can setup.py installations be forced to not build an egg? And therefore no concurrently be trying to write to easy-install.pth?

[doutriaux1]

@durack1 I believe that it is the old_and_unmanageable function, I think I already setup the pip installer to accet this argument OLD has to be set and then we're good to go, it might be only in the ocgis branch though.

[durack1]

@doutriaux1 are there any other options to force a non-egg install? "old and unmanageable" seems to be something that might disappear any new release around the corner..

[durack1]

@aashish24 this needs your input

[doutriaux1]

Conda solves this

[durack1]

@doutriaux1 not sure this is still required but just in case:

From: <owner-uvcdat-devel@listserv.llnl.gov> on behalf of "Doutriaux, Charles"
Date: Thursday, May 12, 2016 at 9:41 AM
To: uvcdat-devel
Subject: FW: [conda] gcc / libgomp and @rpath on OS X

As long we did not fully switched to anaconda, we should definitely use this in our build.

C.

From: <conda@continuum.io> on behalf of Stuart Berg
Date: Thursday, May 12, 2016 at 8:57 AM
To: Dougal Sutherland
Cc: conda - Public <conda@continuum.io>
Subject: Re: [conda] gcc / libgomp and @rpath on OS X

Hi Dougal,
The problem you're seeing isn't related to OpenMP specifically.  You could see the same issue for any .dylib file that is linked with a relative path.

First, here's the "tl;dr" solution:
In your build.sh script, pass --single-version-externally-managed to setup.py:

$PYTHON setup.py --single-version-externally-managed install

Explanation:

The issue here is that zipped python eggs aren't really well suited for python extension modules.  As a workaround, eggs extract their contents to an "egg cache":
http://stackoverflow.com/questions/2192323/what-is-the-python-egg-cache-python-egg-cache
...but that's basically a hacky workaround that doesn't work nicely in all cases.  Instead of creating a zipped .egg file, you can just tell setup.py to install the *unzipped* egg contents by using the option shown above.
So, in your case, _mmd.so thinks it lives in your conda environment's lib folder, and therefore it has a relative path to locate libgomp.dylib.  But in reality, it doesn't live there -- it has been copied to the egg cache (/Users/dsutherl/.python-eggs).  Now the relative paths in that _mmd.so make no sense any more.  Using the option shown above, the egg cache is avoided entirely, and therefore all the necessary files are still in the correct locations relative to each other.

Best,
Stuart


On Mon, May 9, 2016 at 4:43 AM, Dougal Sutherland wrote:
Hi, 

I'm having some trouble getting a conda package built properly using OpenMP on OS X.

My setup is basically:

My build requirements include gcc; install requirements include libgcc.
In build.sh, I set CC=gcc.
The python extension (via cython) that uses OpenMP is then built with gcc, and it all builds seemingly successfully.
When conda-build then goes to import the module in testing, though, I get:

ImportError: dlopen(/Users/dsutherl/.python-eggs/mmd-0.1.1-py2.7-macosx-10.5-x86_64.egg-tmp/mmd/_mmd.so, 2): Library not loaded: @rpath/./libgomp.1.dylib
  Referenced from: /Users/dsutherl/.python-eggs/mmd-0.1.1-py2.7-macosx-10.5-x86_64.egg-tmp/mmd/_mmd.so
  Reason: image not found

libgomp.1.dylib is in the expected place (anaconda/lib/), but @rpath seems to not be pointing there.

(I see the same behavior if I try to install with a homebrew-installed GCC with anaconda; it works with homebrew-installed GCC against system python, and also in anaconda on Linux.)

Is there something else I need to do to get this to work? I'm finding the documentation about @rpath in general on OSX to be quite cryptic.

The current version of the package + recipe, which is quite small and so nearly a minimal example, is here.

Thanks,
Dougal
-- 
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+unsubscribe@continuum.io.
To post to this group, send email to conda@continuum.io.
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.

-- 
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+unsubscribe@continuum.io.
To post to this group, send email to conda@continuum.io.
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.