Rebuild conda packages #556

twiecki · 2014-07-01T09:46:12Z

The conda packages for pymc 2.3 are not compatible with current anaconda versions. We thus need to rebuild and reupload. I can do linux 64bit, @fonnesbeck can you do osx 64bit?

fonnesbeck · 2014-07-01T12:11:46Z

Yes. Will release 2.3.3 today, so hold tight.

aflaxman · 2014-07-01T14:03:19Z

It would be great to have x64 windows version as well. :)

twiecki · 2014-07-01T14:04:28Z

I think 64bit windows has problems with fortran. I can do 32 bit though.

aflaxman · 2014-07-01T14:06:23Z

If you are up for experimenting, I saw some hints on 64bit windows and fortran here: https://twitter.com/ContinuumIO/status/481908466047803392

fonnesbeck · 2014-07-01T16:22:54Z

@twiecki Branch 2.3 contains my fixes and updates for 2.3.3. It passes locally and on Travis, so have a peek and I will tag it if you think it looks okay.

twiecki · 2014-07-01T17:28:07Z

Looks good to me 👍. Let me know once you updated pypi.

fonnesbeck · 2014-07-01T17:40:01Z

Done

twiecki · 2014-07-01T17:48:28Z

RuntimeError: MD5 mismatch: '99645bf558a41376be0e687ccf90a39f' !=
'ea95fd6186a5c08937744f38f85390b9'

On Tue, Jul 1, 2014 at 7:40 PM, Chris Fonnesbeck notifications@github.com
wrote:

Done
https://pypi.python.org/pypi?:action=display&name=pymc&version=2.3.3

—
Reply to this email directly or view it on GitHub
#556 (comment).

Thomas Wiecki
PhD candidate, Brown University
Quantitative Researcher, Quantopian Inc, Boston

fonnesbeck · 2014-07-01T17:49:46Z

PyPI created the MD5 automatically, so not sure what's up.

twiecki · 2014-07-01T17:57:15Z

linux 64 updated https://binstar.org/pymc/pymc/2.3.3/files

On Tue, Jul 1, 2014 at 7:49 PM, Chris Fonnesbeck notifications@github.com
wrote:

PyPI created the MD5 automatically, so not sure what's up.

—
Reply to this email directly or view it on GitHub
#556 (comment).

Thomas Wiecki
PhD candidate, Brown University
Quantitative Researcher, Quantopian Inc, Boston

terhardt · 2014-07-02T06:46:27Z

I have the same problem with the md5 checksum. The last time something was wrong with one of the various caches that conda uses, didn't fix it this time.

Maybe the problem is, that the md5 changes as soon as we put the md5 in the build recipe. In that way they will never match up.

I've pushed a good build to binstar:
https://binstar.org/pymc/pymc/2.3.3/download/osx-64/pymc-2.3.3-np18py27_0.tar.bz2

twiecki · 2014-07-02T08:01:03Z

@tobeplugged Yeah, I also just manually changed the md5 string. Should probably just replace that. I don't think we need to update the pypi version for that.

Since you're set up to do this and have pymc installed successfully, would you be willing to help me upload a hddm conda for osx? Build files etc are all in place and it works well for linux so should just be a conda build and binstar upload. Of course no worries if not.

terhardt · 2014-07-02T11:29:38Z

I've just build 3.3 and 3.4 packages for OSX as well:

https://binstar.org/pymc/pymc/2.3.3/download/osx-64/pymc-2.3.3-np18py33_0.tar.bz2
https://binstar.org/pymc/pymc/2.3.3/download/osx-64/pymc-2.3.3-np18py34_0.tar.bz2

Sure building hddm should be no problem. But I cannot find the recipe in the repository, or am I missing something?

twiecki · 2014-07-02T11:44:42Z

I just uploaded windows 32 bit for py2.

Building hddm should be the same, just checkout master, and then from the root conda build conda should do the trick.

terhardt · 2014-07-02T11:48:45Z

I wasn't looking in the right repo...

voila:

Removed download link

build like a charm. I did not check, but if the package does not have any compiled code, than it should be possible to build cross platform as well.

I'll remove the file as soon as its up on binstar.

twiecki · 2014-07-02T11:54:17Z

Actually, you could just upload it to the pymc repo as well. It has cython
so probably not cross-platform.

On Wed, Jul 2, 2014 at 1:48 PM, Tobias notifications@github.com wrote:

I wasn't looking in the right repo...

voila:
https://www.dropbox.com/s/pwq5ibvakpedqm8/hddm-0.5.4-np18py27_0.tar.bz2
build like a charm. I did not check, but if the package does not have any
compiled code, than it should be possible to build cross platform as well.

I'll remove the file as soon as its up on binstar.

—
Reply to this email directly or view it on GitHub
#556 (comment).

Thomas Wiecki
PhD candidate, Brown University
Quantitative Researcher, Quantopian Inc, Boston

terhardt · 2014-07-02T12:02:15Z

Okay, I'll do that. I thought you might want to keep the channels apart and make a new channel for hddm

twiecki · 2014-07-02T12:04:25Z

I had that for a while but the installation is easier that way. I don't think it could cause any problems.

terhardt · 2014-07-02T12:04:43Z

Turns out that I cannot upload to pymc I always get an error that the file was not found. Maybe I do not have the permissions.

See here:
anaconda/anaconda-client#63

Edit:
for hddm that is.

I've build and uploaded 3.3 and 3.4 versions of pymc for 64 bit Linux as well.

twiecki · 2014-07-02T12:37:03Z

OK, I uploaded the dropbox file. Unfortunately it also needs kabuki which
is a source package. Could I bother you to also create an osx upload for
hddm-devs/kabuki?

On Wed, Jul 2, 2014 at 2:04 PM, Tobias notifications@github.com wrote:

Turns out that I cannot upload to pymc I always get an error that the file
was not found. Maybe I do not have the permissions.

—
Reply to this email directly or view it on GitHub
#556 (comment).

Thomas Wiecki
PhD candidate, Brown University
Quantitative Researcher, Quantopian Inc, Boston

terhardt · 2014-07-02T12:49:30Z

No problem:
you can find it here:

removed download link

I did not build against Python 3 because kabuki did not seem to be compatible.

I have a script that builds a package against all possible combinations of Python (and numpy) so doing the builds for the Pythons is not a big deal.

twiecki · 2014-07-02T12:55:29Z

Many thanks, I really appreciate it.

A user just tried to install but gets a pymc-related error (on osx):

ImportError: dlopen(//anaconda/lib/python2.7/site-packages/pymc/flib.so, 2): Library not loaded: /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib

  Referenced from: //anaconda/lib/python2.7/site-packages/pymc/flib.so

  Reason: image not found

Any ideas? Is a separate fortran library required even with anaconda?

terhardt · 2014-07-02T13:03:05Z

Okay, that is weird. I can load and run pymc with no problem on my machine after installing from binstar.

twiecki · 2014-07-02T13:05:47Z

Do you have that directory?

On Wed, Jul 2, 2014 at 3:03 PM, Tobias notifications@github.com wrote:

Okay, that is weird. I can load and run pymc with no problem on my machine
after installing from binstar.

—
Reply to this email directly or view it on GitHub
#556 (comment).

Thomas Wiecki
PhD candidate, Brown University
Quantitative Researcher, Quantopian Inc, Boston

terhardt · 2014-07-02T13:10:26Z

Yes I do have that directory. But the even with a fresh environment, I have no problems running PyMC.

And the flib.so file is part of the binary packages that I've uploaded.

twiecki · 2014-07-02T13:25:14Z

Right, but it seems to be linked against the other lib dynamically which
doesn't exist on the other system. Do you know where Cellar might be coming
from?

On Wed, Jul 2, 2014 at 3:10 PM, Tobias notifications@github.com wrote:

Yes I do have that directory. But the even with a fresh environment, I
have no problems running PyMC.

And the flib.so file is part of the binary packages that I've uploaded.

—
Reply to this email directly or view it on GitHub
#556 (comment).

Thomas Wiecki
PhD candidate, Brown University
Quantitative Researcher, Quantopian Inc, Boston

terhardt · 2014-07-02T13:35:08Z

That is from homebrew. The library it is linked against is gfortran which I use to build it in the first place.

twiecki · 2014-07-02T13:36:24Z

I see. We might want to statically link in that case or provide the library with pymc together.

twiecki · 2014-11-20T06:32:59Z

@asmeurer Where you able to give this a try with recipe we have on this repo?

twiecki · 2014-11-24T10:03:13Z

Seems like gfortran is installed by the conda gcc package:

conda install gcc

conda$ which gfortran

/Users/jelleferinga/miniconda/bin/gfortran


conda$ find ~/miniconda/ -name "libgfortran.dylib"

/Users/jelleferinga/miniconda//lib/libgfortran.dylib

/Users/jelleferinga/miniconda//pkgs/gcc-4.8.2-4/lib/libgfortran.dylib

asmeurer · 2014-11-25T01:22:31Z

conda package -w will also tell you that.

I plan to split out the libs from my gcc package soon, at least for Linux.

twiecki · 2014-12-16T13:30:31Z

@asmeurer Did you have a chance to give this another try (compiling the conda recipe that's in this repo) or is there some progress on the fortran libraries in conda? Would be great to make this available!

iDrDex · 2014-12-16T18:57:04Z

@asmeurer I agree with @twiecki that it would be great for me to easily import pymc on OS X Yosemite.

asmeurer · 2014-12-16T18:59:04Z

I think the thing that needs to be done is to take my gcc package and extract the dylibs that pymc links to into a separate libgcc package, which would be a runtime dependency. We would ideally include this package in the Continuum repos.

iDrDex · 2014-12-16T19:10:38Z

I can confirm that I got it to install on Anaconda Yosemite with:

conda install -c asmeurer pymc

Thx @asmeurer!

On Dec 16, 2014, at 10:59 AM, Aaron Meurer notifications@github.com wrote:

I think the thing that needs to be done is to take my gcc package and extract the dylibs that pymc links to into a separate libgcc package, which would be a runtime dependency. We would ideally include this package in the Continuum repos.

—
Reply to this email directly or view it on GitHub #556 (comment).

asmeurer · 2014-12-16T19:43:58Z

Yes, that should work. The issue is that it will be a very large download because it currently depends on the entire gcc package.

jonsedar · 2014-12-23T11:06:59Z

Thought I should chip in with my 2p:

I'm running OSX Yosemite (10.10) and have an Anaconda Python 3.4.2 env that I'm intending to use for some work in pymc 2.3.

My first attempt was to install and build gcc-4.9.2_1 via brew install gcc, and then install pymc 2.3.4-np18py34_0 fonnesbeck using conda install -c fonnesbeck pymc

In a ipython session I then import pymc and receive an error:

ImportError: dlopen(/Users/jon/anaconda/envs/price_modelling/lib/python3.4/site-packages/pymc/flib.so, 2): Library not loaded: /usr/local/lib/gcc/x86_64-apple-darwin13.3.0/4.9.1/libquadmath.0.dylib Referenced from: /Users/jon/anaconda/envs/price_modelling/lib/python3.4/site-packages/pymc/flib.so Reason: image not found

So before diving into some potential fun with gcc etc, I thought I'd uninstall pymc and try asmeurer's build conda install -c asmeurer pymc, which seems to install and import happily, and I've run pymc.tests.test() getting 1 minor failure out of 187 runs.

I'll report back with any subtle issues, but so far, thanks for all your help!

twiecki · 2014-12-23T11:18:46Z

Yeah, I think @asmeurer's package works fine. @asmeurer any time line on when this could get included in anaconda?

twiecki · 2014-12-23T11:19:18Z

I think a package with a larger download is better than no package.

erwindassen · 2015-01-05T14:16:36Z

Humm... I've read the whole thread (quite a learning!) and in the end I've tried @asmeurer package. It installed gcc and pymc but after the first import I get the now all to familiar dlopen error: I'm running on Yosemite with Python 3.4.2 (Anaconda 3.7.3).

In [1]: import pymc
/Applications/anaconda/lib/python3.4/importlib/_bootstrap.py:321: RuntimeWarning: compiletime version 3.3 of module 'pymc.Container_values' does not match runtime version 3.4
  return f(*args, **kwds)
/Applications/anaconda/lib/python3.4/importlib/_bootstrap.py:321: RuntimeWarning: compiletime version 3.3 of module 'pymc.LazyFunction' does not match runtime version 3.4
  return f(*args, **kwds)
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-5f262cfcb99b> in <module>()
----> 1 import pymc

/Applications/anaconda/lib/python3.4/site-packages/pymc-2.3.4-py3.3-macosx-10.5-x86_64.egg/pymc/__init__.py in <module>()
     28 from .PyMCObjects import *
     29 from .InstantiationDecorators import *
---> 30 from .CommonDeterministics import *
     31 from .NumpyDeterministics import *
     32 from .distributions import *

/Applications/anaconda/lib/python3.4/site-packages/pymc-2.3.4-py3.3-macosx-10.5-x86_64.egg/pymc/CommonDeterministics.py in <module>()
     19 import inspect
     20 import types
---> 21 from .utils import safe_len, stukel_logit, stukel_invlogit, logit, invlogit, value, find_element
     22 from copy import copy
     23 import sys

/Applications/anaconda/lib/python3.4/site-packages/pymc-2.3.4-py3.3-macosx-10.5-x86_64.egg/pymc/utils.py in <module>()
     12 from copy import copy
     13 from .PyMCObjects import Variable
---> 14 from . import flib
     15 import pdb
     16 from numpy.linalg.linalg import LinAlgError

ImportError: dlopen(/Applications/anaconda/lib/python3.4/site-packages/pymc-2.3.4-py3.3-macosx-10.5-x86_64.egg/pymc/flib.so, 2): Library not loaded: /usr/local/lib/gcc/x86_64-apple-darwin13.3.0/4.9.1/libgfortran.3.dylib
  Referenced from: /Applications/anaconda/lib/python3.4/site-packages/pymc-2.3.4-py3.3-macosx-10.5-x86_64.egg/pymc/flib.so
  Reason: image not found

As you can see it is trying to load the inexistent local flib.so. Something else I'm missing?

asmeurer · 2015-01-06T07:49:23Z

Can you show me the output of otool -L /Applications/anaconda/lib/python3.4/site-packages/pymc-2.3.4-py3.3-macosx-10.5-x86_64.egg/pymc/flib.so?

erwindassen · 2015-01-07T10:13:16Z

Below is the output. By the way did you consider linking it against the Accelerate framework in OSX? It has a tuned implementation of the BLAS/Linpack (no need for GCC) but for libquadmath I'm clueless except for this: http://stackoverflow.com/questions/17590525/correct-way-to-statically-link-in-gfortran-libraries-on-osx.

Thanks for the support!

Output:

/Applications/anaconda/lib/python3.4/site-packages/pymc-2.3.4-py3.3-macosx-10.5-x86_64.egg/pymc/flib.so:
    /usr/local/lib/gcc/x86_64-apple-darwin13.3.0/4.9.1/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 283.0.0)
    /usr/local/Cellar/gcc/4.9.1/lib/gcc/x86_64-apple-darwin13.3.0/4.9.1/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/lib/gcc/x86_64-apple-darwin13.3.0/4.9.1/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

Notice that my /usr/local dir only has bin and lib subdirs and both are empty! I once installed gfortran there but since trying to use asmeurer's package I removed it (following gfortran's site own instructions which amounted to rm -r the binary and the libs in the aforementioned dirs).

asmeurer · 2015-01-08T20:31:29Z

And this is the latest build of my package (what is conda list -e)?

As for using Accelerate, I don't know how to do it, but I agree that it's better. We build the Continuum numpy/scipy against the Accelerate framework on OS X.

erwindassen · 2015-01-08T22:11:49Z

I assumed it is the latest as I installed with as shown in the previous post by jonsedar. Here are the versions:

gcc=4.8.2=4
pymc=2.3.3=np19py34_2.

asmeurer · 2015-01-08T23:11:09Z

That looks right. I'm not sure what's going on. Does conda install -f gcc pymc fix it?

erwindassen · 2015-01-09T07:26:07Z

No, I get the same error but this time I also got this:

/Applications/anaconda/lib/python3.4/importlib/_bootstrap.py:321: RuntimeWarning: compiletime version 3.3 of module 'pymc.Container_values' does not match runtime version 3.4
  return f(*args, **kwds)
/Applications/anaconda/lib/python3.4/importlib/_bootstrap.py:321: RuntimeWarning: compiletime version 3.3 of module 'pymc.LazyFunction' does not match runtime version 3.4
  return f(*args, **kwds)

By the way, does the output of my otool look right?

Cheers,

ilanschnell · 2015-01-26T23:28:50Z

I managed to link pymc statically against libgfortran on MacOSX. Basically by adding the following to the conda recipe:

FCFLAGS=$CFLAGS
cp /usr/local/lib/libgfortran*.*a .
LDFLAGS="-undefined dynamic_lookup -bundle -Wl,-search_paths_first -L$(pwd) $LDFLAGS"

PyMC is now officially supported by Anaconda on Linux and Mac OS X! Conda packages are available for Python 2.

cdagnino · 2015-01-26T23:58:24Z

Great! I just installed pymc 2.3.4 with conda install pymc and run the tests without issue on a Mac OS

twiecki · 2015-01-27T11:54:45Z

@ilanschnell Thanks so much for looking into that!

What about Python 3 and Windows? Those should be pretty easy to add I think.

ilanschnell · 2015-01-27T14:08:31Z

Python 3 packages are also already available for Linux and MacOSX. We haven't looked into Windows yet, but it's on our list.

twiecki · 2015-01-27T14:10:49Z

Cool! FWIW I think the recipe we have works pretty well:
https://github.com/pymc-devs/pymc/blob/2.3/conda/bld.bat

Not sure if it successfully links in gfortran though.

twiecki · 2015-02-09T08:32:59Z

@ilanschnell Any update on a windows version? I think then we can close this issue and just advertise conda as the preferred way to install pymc!

ilanschnell · 2015-02-09T19:23:29Z

I've just created Windows conda packages for pymc, such that we now support pymc on all platforms and for all Python versions!

twiecki · 2015-02-09T21:07:34Z

@ilanschnell @asmeurer Thank you guys so much! 👍

twiecki · 2015-02-09T21:09:20Z

Can the recipe to build them all be found anywhere?

twiecki · 2015-05-15T07:40:32Z

@tobeplugged You were extremely helpful in building OSX conda packages for HDDM before. Do you think I could ask you to help with this once more? Specifically, kabuki and hddm would need to be pulled and then conda build conda. Of course, no worries if you can't do that.

existeundelta · 2015-06-29T10:59:58Z

Hi, i'm getting this error:

 1 import pymc as pm

/anaconda/lib/python2.7/site-packages/pymc-2.3.4-py2.7-macosx-10.9-x86_64.egg/pymc/__init__.pyc in <module>()
     28 from .PyMCObjects import *
     29 from .InstantiationDecorators import *
---> 30 from .CommonDeterministics import *
     31 from .NumpyDeterministics import *
     32 from .distributions import *

anaconda/lib/python2.7/site-packages/pymc-2.3.4-py2.7-macosx-10.9-x86_64.egg/pymc/CommonDeterministics.py in <module>()
     11 __docformat__ = 'reStructuredText'
     12 
---> 13 from . import PyMCObjects as pm
     14 from .Node import Variable
     15 from .Container import Container

ImportError: cannot import name PyMCObjects

I think I've tried all, but may be I did something wrong, can you help me? thanks

twiecki added v.2 labels Jul 1, 2014

twiecki closed this as completed Feb 9, 2015

Rebuild conda packages #556

Rebuild conda packages #556

Comments

twiecki commented Jul 1, 2014

fonnesbeck commented Jul 1, 2014

aflaxman commented Jul 1, 2014

twiecki commented Jul 1, 2014

aflaxman commented Jul 1, 2014

fonnesbeck commented Jul 1, 2014

twiecki commented Jul 1, 2014

fonnesbeck commented Jul 1, 2014

twiecki commented Jul 1, 2014

fonnesbeck commented Jul 1, 2014

twiecki commented Jul 1, 2014

terhardt commented Jul 2, 2014

twiecki commented Jul 2, 2014

terhardt commented Jul 2, 2014

twiecki commented Jul 2, 2014

terhardt commented Jul 2, 2014

twiecki commented Jul 2, 2014

terhardt commented Jul 2, 2014

twiecki commented Jul 2, 2014

terhardt commented Jul 2, 2014

twiecki commented Jul 2, 2014

terhardt commented Jul 2, 2014

twiecki commented Jul 2, 2014

terhardt commented Jul 2, 2014

twiecki commented Jul 2, 2014

terhardt commented Jul 2, 2014

twiecki commented Jul 2, 2014

terhardt commented Jul 2, 2014

twiecki commented Jul 2, 2014

twiecki commented Nov 20, 2014

twiecki commented Nov 24, 2014

asmeurer commented Nov 25, 2014

twiecki commented Dec 16, 2014

iDrDex commented Dec 16, 2014

asmeurer commented Dec 16, 2014

iDrDex commented Dec 16, 2014

asmeurer commented Dec 16, 2014

jonsedar commented Dec 23, 2014

twiecki commented Dec 23, 2014

twiecki commented Dec 23, 2014

erwindassen commented Jan 5, 2015

asmeurer commented Jan 6, 2015

erwindassen commented Jan 7, 2015

asmeurer commented Jan 8, 2015

erwindassen commented Jan 8, 2015

asmeurer commented Jan 8, 2015

erwindassen commented Jan 9, 2015

ilanschnell commented Jan 26, 2015

cdagnino commented Jan 26, 2015

twiecki commented Jan 27, 2015

ilanschnell commented Jan 27, 2015

twiecki commented Jan 27, 2015

twiecki commented Feb 9, 2015

ilanschnell commented Feb 9, 2015

twiecki commented Feb 9, 2015

twiecki commented Feb 9, 2015

twiecki commented May 15, 2015

existeundelta commented Jun 29, 2015