[Python] pypi packages compatible with setuptools

[asfimport]

setuptools is internally using easy_install for pulling packages from pypi. easy_install doesn't support wheel so since pyarrow is in pypi distributed only as wheels it is not possible to package a product depending on pyarrow using setuptools.

see attached setup.py:

$ python setup.py test
running test
Searching for pyarrow==0.5.0.post2
Reading https://pypi.python.org/simple/pyarrow/
No local packages or working download links found for pyarrow==0.5.0.post2
error: Could not find suitable distribution for Requirement.parse('pyarrow==0.5.0.post2')

It's a shame setuptools don't support wheels. unfortunately it supports only eggs or raw source packages (see distro_for_location). I am not suggesting providing eggs but perhaps publishing raw tar.gz should be considered so that setuptools can at least build the dependency itself.

Reporter: Antony Mayi / @antonymayi

Original Issue Attachments:

• setup.py

_{Note: This issue was originally created as ARROW-1346. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
What platform are you on? There are only 0.5.0.post2 binaries for Linux, for the other platforms it is still 0.5.0

https://pypi.python.org/pypi/pyarrow

[asfimport]

Wes McKinney / @wesm:

setuptools is internally using easy_install for pulling packages from pypi. easy_install doesn't support wheel so since pyarrow is in pypi distributed only as wheels it is not possible to package a product depending on pyarrow using setuptools.

What version of setuptools? I'm pretty surprised by this. In production deployments I usually see pip install -r requirements.txt, and pip knows how to install wheels

[asfimport]

Antony Mayi / @antonymayi:
I am on linux.

Yes, pip works fine, but setuptools is not using pip. Any version of setuptools (I am on the latest 36.2.7).

The attached setup.py demonstrates the problem - ie if I am trying to distribute my python project that depends on pyarrow as per the setup.py it won't pull pyarrow out of the box (running the typical python setup.py install fails as per the error shown above due to the setuptools inability to pull wheels from pypi).

[asfimport]

Wes McKinney / @wesm:
My understanding is that installing packages via setup.py (i.e. easy_install) is effectively deprecated, and so it's unlikely we'll support it here. See associated discussion in the TensorFlow issue tracker tensorflow/tensorflow#6540 ("Building and distributing an egg file is not planned for tensorflow, as it is the old deprecated standard. I will close this issue now.") . I would recommend changing your software deployment process to use pip and requirements.txt. You can also use conda to achieve the same result.

[asfimport]

Antony Mayi / @antonymayi:
Right, that's why said I am not suggesting to publish eggs. No probs. It's just odd that setuptools is still the main python packaging concept that can nicely define dependencies (even in pyarrow iself) and is unable to deploy such packages itself (in case of wheel deps) but needs another tools to resolve its dependencies. But I understand this is not a problem of arrow...

[asfimport]

Wes McKinney / @wesm:
Yeah, I agree that the state of Python packaging continues to be very sad. 26 years into the Python programming language I'm not optimistic that things will get much better