provide conda package

[davidslac]

Can you provide a PyAbel conda package? A user at LCLS/SLAC has requested PyAbel. We have central miniconda installations for our users, I'm interested in putting PyAbel in the central install (this is for linux - rhel7 - python 2.7 primarily) however we are trying to keep pip installs out of the central installation, preferring conda packages - otherwise we get special cases where we have to manage the package dependencies more carefully. User's can clone the central install and use pip (which I'll tell this user how to do), but our in house software for parsing LCLS data is best managed centrally.

[rth]

@davidslac Thanks for the feedback! Do you use packages from conda-forge or regular conda? Adding PyAbel to conda-forge would indeed be useful (and would remove all the compiler / Cython requirements issues).

[davidslac]

Hi Roman, we use conda-forge as well as defaults, but I don't mind using a custom channel hosted at anaconda, like I also install guppy from https://anaconda.org/kalefranzso if you are able to start with a channel like that, that would be great! best, David

…

On 01/04/17 12:27, Roman Yurchak wrote: @davidslac <https://github.com/davidslac> Thanks for the feedback! Do you use packages from conda-forge <https://conda-forge.github.io/> or regular conda? Adding PyAbel to conda-forge would indeed be useful (and would remove all the compiler / Cython requirements issues). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#182 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQAThaDtTAumMsEbtmADrKymnqYqqMLcks5rPACvgaJpZM4LbAQL>.

[rth]

Hi David,
OK, we will look into it. Not sure how fast this could be done though, so using pip might be a more immediate temporary solution.
Also great to hear that somebody is using PyAbel at LCLS!

[DanHickstein]

Hi @davidslac, We are excited that LCLS is using PyAbel! We would be happy to work with you in whatever ways that we can to make PyAbel easier for your to install and maintain.

I don't think that any of the PyAbel developers have much experience with making conda packages (we are mostly experimentalists :) ). We would be happy to look into how to do this, but if you have recommendations on the best practices or would like to spearhead this, please let us know.

EDIT: conda-forge looks very cool. Seems like @rth is on the right track, as usual.

[rth]

@DanHickstein Yes, I agree that conda-forge looks really great, though I have also not tried submitting a package there yet. The fact that it's community based means that we might also get some feedback about the packaging (as opposed to a standalone conda channel), and automatically building conda packages on Linux, MacOS and Windows would be worth the effort I think...

[davidslac]

I'm not the best person for building conda packages, I haven't figured out how to build C/C++ in an OS independent way. That seems to get hairy. For example conda-forge doesn't support rhel5, and some of there packages introduced a dependency on glibc, I guess to run on older OS's. However you might be fine since you are only using Cython, maybe that will all get taken care of. An example of creating a conda package from a wheel file (which you already produce, I believe) is tensorflow on conda-forge. However that wheel file doesn't run on rhel6 or rhel5 - but if you produce a conda package that only runs on rhel7 that would be fine.

[rth]

@davidslac I agree that packaging of C/C++ (or of Python code that has C/Cython) in OS independent way is definitely not simple and it takes a lot of time to figure things out. Interesting that conda-forge doesn't work on rhel5 (saw your issue at numpy-feedstock) but I imagine it's all the difficulty of keeping things working in a legacy environment while also supporting all the latest features/updates. Generally there is an issue of fast deprecation in the python ecosystem, unlike for some Fortran code from 20 years ago that could be still compiled and used without issues...

[DanHickstein]

I forked the conda-forge/staged-recipes repo to PyAbel: https://github.com/PyAbel/staged-recipes/tree/pyabelrecipe

I am a little confused about how things work, but I think that I just needed to edit the recipes/meta.yml file and then we can submit a PR to conda-forge and they will review it.

I'll submit the PR in a few hours, so check out the meta.yml file that I made and see if it looks okay: https://github.com/PyAbel/staged-recipes/blob/pyabelrecipe/recipes/pyabel/meta.yaml

[stggh]

@DanHickstein what about this section in meta.yaml?

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  fn: {{ name }}-{{ version }}.tar.gz
  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz
  md5: {{ md5 }}

Is the .tar.bz2 file built using this tutorial Building conda packages with conda skeleton or is it already available on PyPi somewhere?

[DanHickstein]

The tar.bz2 is on PyPi: https://pypi.python.org/pypi/PyAbel. If I understand correctly, conda-forge is basically going to be calling conda skeleton on the PyAbel package that is on PyPi and making the generated package available for Mac, Win, and Linux.

I'm going to go ahead and submit a PR to conda-forge and see what they say...

[rth]

@DanHickstein Thanks for making it happen! it looks great!

[DanHickstein]

I released a new version of PyAbel (0.7.5) on PyPi. Barring any hiccups, the package should be available on conda-forge sometime in the near future (whenever they merge the PR).

All that said, @davidslac has instigated a renaissance in PyAbel development, and may want to recommend that his user fork the latest GitHub version of PyAbel to stay up-to-date with some exciting new features and get involved with the development fun as well :)

[davidslac]

Sounds great, I cc-ed the user (Gregor Hartmann) so he can be informed. For the time being I gave Gregor instructions on creating his own conda environment and using pip to get PyAbel - when PyAbel gets on conda-forge I'll look into adding it to the multi-user conda environments we maintain. best, David

…

On 01/10/17 13:05, Danhickstein wrote: I released a new version of PyAbel (0.7.5) on PyPi. Barring any hiccups, the package should be available on conda-forge sometime in the near future (whenever they merge the PR). All that said, @davidslac <https://github.com/davidslac> has instigated a renaissance in PyAbel development, and may want to recommend that his user fork the latest GitHub version of PyAbel to stay up-to-date with some exciting new features and get involved with the development fun as well :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#182 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQAThQS38aKBrR4vIeo8Vj8fDRCKoWDiks5rQ_J8gaJpZM4LbAQL>.

[DanHickstein]

Thanks David! Sorry for taking a while to get the packaging done. We research scientists are not always that savvy when it comes to actually getting our software packaged for public consumption :)

[stggh]

conda installs PyAbel, but the "old" version runs. How do I import the conda (auto correct thinks this word is condo) version?

[SteveMBP:~]$ conda config --add channels conda-forge
[SteveMBP:~]$ conda install pyabel
Fetching package metadata .........
Solving package specifications: ..........

Package plan for installation in environment /Users/stg121/anaconda:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pyabel-0.7.6               |      np111py35_0         677 KB  conda-forge

The following NEW packages will be INSTALLED:

    pyabel: 0.7.6-np111py35_0 conda-forge

Proceed ([y]/n)? y

Fetching packages ...
pyabel-0.7.6-n 100% |################################| Time: 0:00:03 191.89 kB/s
Extracting packages ...
[      COMPLETE      ]|###################################################| 100%
Linking packages ...
[      COMPLETE      ]|###################################################| 100%
[SteveMBP:~]$ ipython3
Python 3.5.2 |Anaconda 4.2.0 (x86_64)| (default, Jul  2 2016, 17:52:12) 
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import abel

In [2]: abel.__version__
Out[2]: '0.7.5'

In [3]: quit

[rth]

@stggh What's the value of abel.__file__ ?

[stggh]

The path to the cloned PyAbel that was installed using (conda) python3 setup.py develop --user

[rth]

@stggh Well at least on Linux, the Python packages you install with the --user flag into ~/.local/lib/python.. will be imported before any Conda or system wide installs (I think). I think running pip uninstall pyabel then reinstalling pyabel via conda (to be safe) should fix it,

The long term solution to this is to use virual environments, for instance,

conda create --name pyabel-dev-env numpy scipy nose cython python=3.5
source activate pyabel-dev-env
python3 setup.py develop   # without the user flag

then the conda install could be tested in a separate environment

conda create --name pyabel-binary-env pyabel  python=2.7
source activate pyabel-binary-env

which gets you the right version,

(pyabel-binary-env) rth@byzance /tmp % python                           
Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:09:15) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
>>> import abel
>>> abel.__file__
'/home/rth/.miniconda/envs/pyabel-binary-env/lib/python2.7/site-packages/abel/__init__.pyc'
>>> abel.__version__
'0.7.6'

this is also practical for testing different python versions in different environments..

Congrats with the sucessfull conda-forge setup!
For now it only includes a PY2.7 build; we would also need to add the PY3 ones (opened a new issue at conda-forge/pyabel-feedstock#1). Fixed, PY2.7, PY3.5, PY3.6 are now available.

[stggh]

@rth - you know your stuff! Thanks, I will test out py3 later.

Forgot to add, my installation, above, is on a MacBook Pro. Linux is at work.

[rth]

Correction: the end of the first paragraph in my previous message should read "then reinstalling pyabel via conda" not "then reinstalling conda" (made the correction on Github, but email notifications would still not be correct).

[rth]

@stggh No problem. Conda is cross platform, so the process should be mostly the same on Mac / Windows, I think..

[DanHickstein]

Closing this issue, and declaring victory, since PyAbel is now on conda-forge!

https://github.com/conda-forge/pyabel-feedstock