Make Rados ready for PyPI

[onyb]

Test package is hosted on http://testpypi.python.org/pypi/rados

Steps to build from source:

• First install the development headers and libraries of Python and Rados.

  $ sudo apt-get install python-dev librados-dev
  

• Install Cython

  $ sudo pip install cython
  

• Install Rados bindings

  $ cd src/pybind/rados
  $ sudo python setup.py install
  

• (optional) Install Rados bindings from pre-cythonized source
  You don't need to have Cython installed in this case.

  $ cd src/pybind/rados
  $ sudo python setup.py install --without-cython
  

• Building Rados bindings from Ceph top-level

  $ PYTHON=python3 ./configure
  $ make
  

Note that virtualenv and pyvenv are also supported.

Steps to run the test package:

$ sudo pip install -i https://testpypi.python.org/pypi rados

[ktdreyer]

Want to link to http://tracker.ceph.com/issues/5900 ?

[jcsp]

Sorry if this has already been discussed, but why is it useful to have our rados modules on pypi? They are not versioned independently of librados.so, so unless someone happens to have exactly the same versions of both, things are liable to not work.

Given that someone will always need native (librados) packages when using the python modules, what is the advantage of getting the python modules from pypi instead of getting the python modules from the packages that we provide?

[onyb]

@jcsp: I am not very sure how the Debian package python-rados has been packaged, but I think it doesn't contain the new Cython bindings, and installs only for Python 2.7.

Advantage of using pip is that we can ship Cython extensions which will be cythonized and compiled according to the Python version pip was launched with. This is also important because bindings compiled on Python 2 won't work on Python 3, and vice-versa. Another reason to go for pip is that it allows installing rados inside virtual environments.

Maybe @jdurgin will have something to add.

[onyb]

@ktdreyer Updated issue #5900 with a link to this PR.

[jdurgin]

@jcsp the version requirement isn't exact, as-is older python bindings will continue to work with new libraries, they'll only fail when newer python bindings are used with older libraries which don't have new functions that the python bindings use.

The drive for this is to play nicer with python packaging for developers, and like @onyb mentioned using different versions of python or the bindings is one aspect. It allows virtualenvs with no system site packages. Compiling just the c generated by cython isn't expensive, so it seems like a good tradeoff to me, rather than trying to bundle librados or compile it in the python package.

[jdurgin]

Is it necessary to commit the generated .c file? Could that be created by the usual build process before creating the python package?

[jdurgin]

can leave off 2.6 - no distros ceph supports use it, and this setup.py won't work with it (subprocess.check_output() isn't there until 2.7).

Is it necessary to specify all the minor versions rather than just python 2 and python 3?

[onyb]

I idea was to be explicit about the versions supported. I kept Python 2.6+ and 3.2+ since it was supported by the bindings, but it is seems logical to drop 2.6, 3.2, and 3.3.

[jdurgin]

On 05/27/2016 02:50 AM, Anirudha Bose wrote:

In src/pybind/rados/setup.py
#9332 (comment):

         )

• ], build_dir=os.environ.get("CYTHON_BUILD_DIR", None)),
• cmdclass={

•    "egg_info": EggInfoCommand,
  

• },

•    ], build_dir=os.environ.get("CYTHON_BUILD_DIR", None)
  

• ),
• classifiers=[

•    'Intended Audience :: Developers',
  

•    'Intended Audience :: System Administrators',
  

•    'License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)',
  

•    'Operating System :: POSIX :: Linux',
  

•    'Programming Language :: Cython',
  

•    'Programming Language :: Python :: 2.6',
  

I idea was to be explicit about the versions supported. I kept Python
2.6+ and 3.2+ since it was supported by the bindings, but it is seems
logical to drop 2.6, 3.2, and 3.3.

Sounds good

[jdurgin]

@alfredodeza perhaps you'd like to take a look at this too

[onyb]

@jdurgin The motivation behind including the cythonized file is to allow users to do a python setup.py install --without-cython if they don't want to have Cython as a dependency. Said that, not committing the file is an option, but will only work for pip install, and not python setup.py install.

I can push a commit to exclude rados.c from source tree, and only include it in the PyPI source distribution. I will also drop the --without-cython option, and instead use the already cythonized .c file only if Cython is not available.

[jdurgin]

@onyb that sounds good wrt to rados.c

[jdurgin]

built nicely, and it's qutie easy to reconfigure + rebuild for a new python version. I pushed this as wip-rados-pypi so we can see how it builds on different platforms (http://ceph.com/gitbuilder.cgi)

[jdurgin]

We will want to make this work with cmake too (I think that could be a later PR though)
@alimaredia @tchaikov any comments on this?

[onyb]

@jdurgin I see builds in Gitbuilder fail with an error: option --single-version-externally-managed not recognized. The reason is that the setup.py which uses only distutils (not setuptools), doesn't accept that option. Switching back to using setuptools should fix the problem.

[alimaredia]

@tchaikov will you take care of the ramifications to CMake of this PR?

[tchaikov]

@alimaredia i will.

[onyb]

Please head over to pull request #9833