Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Python setuptools/distutils packaging #1455

Closed
wants to merge 7 commits into from

4 participants

Axel Haustant Mathieu Le Marec - Pasquet Dane Springmeyer Sean Gillies
Axel Haustant

I ported the pymapnik2 packaging into mapnik.

As Python distribution is tied to mapnik release, I don't see the interest of having a separate project.

  • No more manual synchronization and versioning
  • Publish on PyPI as mapnik instead of mapnik2
  • Works in a virtualenv

It has been tested on Ubuntu with Mapnik 2.1.0 PPA.

You just need to have libmapnik, libmapnik-dev and mapnik-utils installed.

It works the classic Python way:

  • Install:
$ python setup.py install
  • Run tests:
$ python setup.py test
  • Make a source distribution and a binary distribution:
$ python setup.py sdist bdist
  • Publish on PyPI
$ python setup.py sdist register upload

It may need a little bit of polish for Mac OSX and Windows (I can't test it).

Axel Haustant

I forgot to ask: It could be very nce to have it backported on 2.1.0 and to release it on PyPI.

Right now there is no 2.1.0 release on PyPI so it's not virtualenv/pip/easy_install installable.

Mathieu Le Marec - Pasquet

A big -10000000 on this pull request.

Historically, @springmeyer didn't want at first to split out the python bindings from the source tree of mapnik.
Naturally, we have comed to the point that experiencing of repackaging the python bindings as a python egg (in source distribution term) may need to be "semi official" and then having it living on the mapnik organization straightforward to make it semi official, but also to tell users "use at your own risks".

Concerning the packaging of the python bindings, for me :

  • bindings of library for any language must be packaged using the targeted language distributions tool (rake, setuptools, maven, pear, whatever the packaging it is, it s not the point).
    • Obviously the bindings are not the library itself, the library become a dependency of the current bindings externalization. Thus, as any other dependency this bindings package may need.
    • This is just then evident that the package only need to contains files concerned by the packaging itself or the source code of only the bindings.

From where i started & speaking as a long time python user & developer for deploying in lot of bunch systems, at first mapnik2 bindings were all but practical & standards to install.

  • Linking to only one system python linked to a specific system version of mapnik, configured and installed at the same time of the library itself never match one of my real cases.
  • I practically never use the system python but one in a virtualenv, nowadays lot of people do
  • Generally, the virtualenv comes from a locally installed python, not /usr/bin/python
  • 99% of the time, i'm using zc.buildout, nevertheless you ll know that nowadays lot of people use zc.buildout. to install python packages.
  • The common point of those eco systems is the package must be be easy_installable. After that, the package is python friendly.
  • It is not that rare to have multiple pythons on the same box, i dont want to have to compile a new whole mapnik library just to have the python bindings linked of a specific python. I want to have the lib, and then to install the bindings of each specific project linked to that UNIQUE lib.

And, hey, i use a python binding, i just want to run the python setup.py dance (that s what pip/easy_install/zc.buildout do under the hood).

Coming from all that unhapiness, the solution i found was naturally to let the bindings look like any other python source distribution:

  • we have the setup.py,
  • the python bindings
  • an extension module to the Compiled c++ mapnik2 lib.
  • some tests
  • some doc
  • The data files We then just have to make love to the extension module described in the setup.py to be compiled/linked with the appropriate compilation flags. As for now the official bindings live in mapnik2 source code itself, i have made an helper to port this code to be externalized and help the developer to make a new release of the python bindings.

This is not ideal, this is just waiting for mapnik2 developers to decide to one day use only the python bindings as their base to work on mapnik/python and meanwhile to enable users to have an easy_install version of those bindings, and finally remove the python code from their mapnik2 repo.

  • On a subside note, i understand why mapnik developers want to keep the bindings inside the source tree, its just more simpler for them to release ... Maybe the solution is for them just to grab the bindings inside of the source directory as an extra step of their build system and wrap the construct construct the bindings from there but using under the hood a call to the python sdist dance and add a python sdist register to upload to pypi. For binary packages, they can use the same scheme to run a bdist_egg on the targeted platform.

The externalized bindings are working really well and this releasing scheme also works, the only thing we have to do is to release bindings where tests work which is not the case of the actual release, that's why i didnt do a release yet.
I will see what i have locally and push it to the master branch of the python bindings even if it is not working for you to work on it if you are in the mood of.
Ping me if you see nothing monday (03/09).

Mathieu Le Marec - Pasquet

I wasnt clear on the fact that i am VERY pleased of your implication toward mapnik python bindings, but your efforts would be better targeted on making tests of the actual python code working with the actual mapnik2 lib.

Axel Haustant

I totally agree with you on Python packaging but I submitted this pull request because I think it's half a solution to have setuptools packaging outside the repository.

Like you said:

the bindings look like any other python source distribution:

  • we have the setup.py,
  • the python bindings
  • an extension module to the Compiled c++ mapnik2 lib.
  • some tests
  • some doc
  • ...

Look at the setup.py file, it's still the case.
It just moved it into mapnik repository and tuned it a little bit.

If you make the source release you will see that I don't package the entire mapnik sources but only the python bindings.
Installing it or building a binary release does not compile mapnik itself, only the bindings.
They are still compiling on the installed libmapnik, not the source in the tree.

I really don't think syncing manually the 2 repositories is a decent solution:

  • no sources history
  • huge commits due to the sync makes history unreadable
  • backporting Python fixes is too complicated
  • it makes setuptools distribution a second level distribution (still no 2.1.0 release)
  • it clutters the workflow (as I said backporting, releasing...)

I just want to keep bindings and packaging together because they work together.
It allows you to tests latest bindings without syncing every time.:

  1. compile/install mapnik with scons
  2. work on the packaging
  3. install or package it if needed

Right now, I think this solution doesn't remove any benefits from your works (still a standard distutils/setuptools packaging) but add the benefits of simplicity and maintainability.
As long you have libmapnik installed on your system, you can install it like any other python souce distribution and you are able:

  • publish it on PyPI
  • install it with pip or easy_install:
    • from PyPI
    • directly from github
    • from the sources
    • from a standalone source distribution
    • from a standalone binary distribution
  • launch the test using the setup.py
  • ...

If tomorrow the bindings are externalized, you already have the setuptools packaging in the repository.
The only task left will be to renamed the folders.

I agree, externalizing is the preferred solution, but I think it has to be all or nothing.
Maybe a git submodule can provide a transitionnal state.

For the test point, they are still in the mapnik repository so it has to be fixed in there.
Like you I prefer a release where all tests pass, but it's not the case and I still need to be able to work with the last python mapnik version.
Why should it be released in homebrew and PPA and not in PyPI: it's exactly the same code and tests fail for all three distributions.

Mathieu Le Marec - Pasquet

looks like the stuff for 2.0.2 was commited in 82dc7b15d254da6c70a164f6d76d463a7b592de0, looking to fix what were my issues.

Axel Haustant

Link to the commit is not working :(
Is it mapnik/pymapnik2@82dc7b1 ?

Why not submitting it directly to the mapnik codebase ?

Mathieu Le Marec - Pasquet

This workaround is just a test, it was to test escaping the exeption.

But my final conclusion is that anyway it just indicate i have a problem in my local mapnik installation.

I'm on my way to setup a new cleanup test sandbox.

Mathieu Le Marec - Pasquet

Concerning your history issues, as soon as @springmeyer decide to let the python bindings living outside of mapnik source tree we can:

  • clone the mapnik bindings, delete all not python stuff, make some rebase to swap the history clean and have the python code history saved.
  • backport all my releasing stuff in this new layout and adapt it.
  • push --force to this repository to replace it.

Until such a decision is done and until i'm not dead or find another any other not yet found decent solution, i will only support those externalized bindings.

Mathieu Le Marec - Pasquet

See http://pypi.python.org/pypi/mapnik2/2.1.0
and the github code, there is lot of improvments in the distutils port of the build from 2.0.x

Dane Springmeyer
Owner

Hey guys. Just seeing this - I will take a closer look and advise. I generally like the idea of starting to use distutils more, even if the bindings are kept in core. Could be a good foundation for moving them out at some point - which I'm open to, but only at a a major release, like 3.x.

Mathieu Le Marec - Pasquet

That what is said now for years without any progress.
I will just continue to maintain my 'pymapnik2', going back to silence.

Axel Haustant

I agree, it will ease you the work.

Split bindings for 3.0 seams OK for me.

Mathieu Le Marec - Pasquet

In the idea its fine, in the facts not.
A long time ago, pymapnik2 born for that purpose to be integrated in a very near future, some years later im just maintaining again & again this fork ;)

Dane Springmeyer
Owner

@noirbizarre - would it be possible to move these files from the root into bindings/python/, as they only pertain to the python bindings (which are an optional part of mapnik)?

Axel Haustant

Sorry for answering late.
Having the setup.py into the root allwos installing directly from git repository.
pip doesn't support yet installing from subfolder (but a work is in progress: pypa/pip#526 ).

If you want me to move the files into bindings/python, I need to modify them before merging.

Dane Springmeyer
Owner

What do you think about moving all related files except setup.py into bindings/python?

Axel Haustant

I moved rst files into bindings/python and removed the setup.cfg (using environment variables instead to configure nose).

The MANIFEST.in and the setup.py are mandatory so I left them into the root.

Sean Gillies sgillies self-assigned this
Sean Gillies sgillies removed their assignment
Sean Gillies
Collaborator

Moving this to #2408.

Sean Gillies sgillies closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
6 .gitignore
View
@@ -52,3 +52,9 @@ demo/viewer/ui_about.h
demo/viewer/ui_info.h
demo/viewer/ui_layer_info.h
tests/cpp_tests/*-bin
+
+# Setuptools artifacts
+dist/
+build/
+*.egg-info
+*.egg
15 MANIFEST.in
View
@@ -0,0 +1,15 @@
+global-exclude *
+
+include setup.py
+include COPYING
+include AUTHORS.md
+include MANIFEST.in
+include bindings/python/*.rst
+
+recursive-include bindings/python *.cpp *.hpp
+recursive-include bindings/python/mapnik *.py
+recursive-include bindings/python/mapnik2 *.py
+recursive-include deps/agg/include *.h
+recursive-include tests/python_tests *.py
+recursive-include tests/python_tests/images *.png
+recursive-include bindings/python *.cpp
38 bindings/python/CHANGELOG.rst
View
@@ -0,0 +1,38 @@
+Changelog
+=========
+This the PyPI Python bindings only changelog.
+For a full changelog see `the official Mapnik changelog <https://github.com/mapnik/mapnik/blob/master/CHANGELOG.md>`_.
+
+2.1.0
+-----
+
+- Updated for mapnik 2.1.0
+- No more separate bindings repository.
+- New Pypi package name: mapnik instead of mapnik2
+
+
+2.0.1.3 (2012-08-05)
+--------------------
+
+- Nothing changed yet.
+
+
+2.0.1.1 (2012-08-05)
+--------------------
+
+- renaming release
+
+
+2.0.2 (2012-08-04)
+------------------
+
+- Multi Arch Support, thx to noirbizarre. [kiorky]
+ See https://github.com/mapnik/pymapnik2/pull/4
+
+
+2.0.1 (2012-05-06)
+------------------
+
+- First public release of mapnik2 eggified python bindings
+
+
137 bindings/python/README.rst
View
@@ -0,0 +1,137 @@
+======================
+Mapnik Python Bindings
+======================
+
+Official `Mapnik`_ python bindings.
+
+.. image:: https://secure.travis-ci.org/mapnik/mapnik.png
+ :target: http://travis-ci.org/mapnik/mapnik
+
+.. contents::
+
+
+Installation
+============
+
+Prerequisites
+-------------
+
+Before installing, you need:
+
+* the ``mapnik-config`` utility in your ``$PATH``
+* ``Boost.Python`` linked to your python interpreter (see `Boost notes`_)
+* ``cairo`` / ``cairomm`` (optionnal but enabled if you compiled mapnik with cairo support)
+* mapnik
+* The current python interpreter
+
+
+Version notes
+-------------
+
+The python bindings are tied to the mapnik library version.
+
+======================== ====================
+ Mapnik library version Python package
+======================== ====================
+2.0.x mapnik2 (==2.0.1.3)
+2.1.0 mapnik (==2.1.0)
+======================== ====================
+
+You can find your installed version with:
+
+.. code-block:: bash
+
+ $ mapnik-config -v
+
+
+Boost notes
+-----------
+
+To specify which ``boost_python`` lib to link against, you can use:
+
+.. code-block:: bash
+
+ $ export MAPNIK2_BOOST_PYTHON="libboost_python.so.1"
+
+Where you have on your filesystem
+
+.. code-block:: bash
+
+ /usr/lib/libboost_python.so.1
+
+Don't forget that you can play with ``LDFLAGS``/``CFLAGS``/``LD_LIBRARY_PATH`` to indicate non standart locations for the requirements if it applies.
+
+
+Standard installation
+---------------------
+You can use either ``easy_install`` or ``pip``.
+Choose the one you prefer into:
+
+.. code-block:: bash
+
+ $ easy_install mapnik # to use the last known version
+ $ easy_install mapnik==2.1.0 # to pin the version
+ $ pip install mapnik # to use the last known version
+ $ pip install mapnik==2.1.0 # to pin the version
+
+Standard installation works with virtualenv too.
+
+If an error occurs, please read carefully `Prerequisites`_, `Version notes`_ and `Boost notes`_ sections before submitting an issue to the `Mapnik tracker`_.
+
+
+Buildout
+--------
+
+Some developers use buildout_ to ease deployments.
+
+* Add ``mapnik`` to the list of eggs to install, e.g.
+
+.. code-block:: ini
+
+ [buildout]
+ parts = somepart
+
+ [somepart]
+ recipe = minitage.recipe.scripts
+ ...
+ # (options like include dirs)
+ ...
+ eggs =
+ ...
+ mapnik
+
+* Re-run buildout, e.g. with:
+
+.. code-block:: bash
+
+ $ ./bin/buildout
+
+
+Credits
+=======
+
+Companies
+---------
+
+|makinacom|_
+
+* `Planet Makina Corpus <http://www.makina-corpus.org>`_
+* `Contact us <mailto:python@makina-corpus.org>`_
+
+.. |makinacom| image:: http://depot.makina-corpus.org/public/logo.gif
+.. _makinacom: http://www.makina-corpus.com
+
+
+Authors
+-------
+
+Contributors
+------------
+
+* kiorky <kiorky@cryptelium.net>
+* noirbizarre <noirbizarre+mapnik@gmail.com>
+
+
+.. _buildout: http://buildout.org
+.. _Mapnik: http://www.mapnik.org
+.. _Mapnik tracker: https://github.com/mapnik/mapnik/issues
144 setup.py
View
@@ -0,0 +1,144 @@
+import os
+import re
+import sys
+import glob
+
+from setuptools import setup, find_packages, Extension
+from subprocess import Popen, PIPE, check_output
+from ctypes import CDLL
+
+
+def rst(filename):
+ '''
+ Load an rst file and sanitize it for PyPI.
+ Remove unsupported github tags:
+ - code-block directive
+ '''
+ content = open(filename).read()
+ return re.sub(r'\.\.\s? code-block::\s*\w+', '::', content)
+
+
+def mapnik_config(key):
+ cmd = 'mapnik-config --%s' % key
+ try:
+ ret = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
+ except OSError as e:
+ if e.errno == os.errno.ENOENT:
+ raise Exception('mapnik-config not found') # handle file not found error.
+ else:
+ raise
+ if ret.wait() != 0:
+ raise Exception(
+ '%s error: %s\n%s' % (
+ cmd,
+ ret.stdout.read(),
+ ret.stderr.read(),
+ )
+ )
+ return ret.stdout.read().strip()
+
+
+bindings_dir = os.path.join('bindings', 'python')
+include_dirs = [
+ bindings_dir,
+ os.path.join('deps', 'agg', 'include'),
+]
+libraries = ['jpeg', 'png', 'boost_thread', 'python%s.%s' % sys.version_info[:2]]
+extra_compile_args = mapnik_config('cflags').split()
+extra_link_args = mapnik_config('libs').split()
+
+
+if sys.platform.startswith("linux"):
+ # Multiarch support
+ try:
+ arch = check_output(['dpkg-architecture', '-qDEB_HOST_MULTIARCH']).strip()
+ extra_compile_args.append('-I/usr/lib/%s/sigc++-2.0/include' % arch)
+ extra_link_args.append('-L/usr/lib/%s' % arch)
+ except:
+ pass
+
+
+if sys.platform == "win32":
+ libraries.extend(
+ "boost_python-mgw",
+ )
+else:
+ prefix = 'cyg' if sys.platform == 'cygwin' else 'lib'
+ if sys.platform == 'darwin':
+ suffix = 'dylib'
+ elif os.name == 'nt':
+ suffix = 'dll'
+ else:
+ suffix = 'so'
+
+ for lib in ["boost_python", "boost_python-gcc"]:
+ try:
+ if ".%s" % suffix in lib:
+ lib = re.sub(".%s.*" % suffix, '', lib)
+ if lib.startswith(prefix):
+ lib = lib[len(prefix):]
+ CDLL('%s%s.%s' % (prefix, lib, suffix))
+ libraries.append(lib)
+ break
+ except OSError:
+ pass
+ except IndexError:
+ raise Exception('Cant find boost_python lib!')
+
+
+long_description = '\n'.join((
+ rst(os.path.join(bindings_dir, 'README.rst')),
+ rst(os.path.join(bindings_dir, 'CHANGELOG.rst')),
+ ''
+))
+
+files = glob.glob(os.path.join(bindings_dir, '*.cpp'))
+
+install_requires = ['setuptools', ]
+for lib in extra_link_args:
+ if 'cairo' in lib:
+ dep = 'pycairo'
+ if sys.version_info[0] < 3:
+ dep = 'py2cairo'
+ install_requires.append(dep)
+ break
+
+version = mapnik_config('version')
+
+# nosetests configuration
+os.environ['NOSE_WHERE'] = 'tests/python_tests'
+
+setup(
+ name='mapnik',
+ version=version,
+ description="Python bindings for mapnik",
+ long_description=long_description,
+ keywords='',
+ author='Mathieu Le Marec - Pasquet & the mapnik community',
+ author_email='kiorky@cryptelium.net',
+ url='http://pypi.python.org/pypi/mapnik',
+ license='LGPL',
+ include_package_data=True,
+ zip_safe=False,
+ packages=find_packages(bindings_dir),
+ package_dir={'': bindings_dir},
+ ext_modules=[
+ Extension(
+ "_mapnik", files,
+ include_dirs=include_dirs,
+ extra_compile_args=extra_compile_args,
+ libraries=libraries,
+ extra_link_args=extra_link_args
+ ),
+ ],
+ install_requires=install_requires,
+ tests_require=['nose>=1.0'],
+ test_suite='nose.collector',
+ entry_points={},
+ classifiers=[
+ "Programming Language :: Python",
+ "Topic :: Software Development :: Libraries :: Python Modules",
+ "Topic :: Scientific/Engineering :: GIS",
+ "License :: OSI Approved :: GNU Lesser General Public License v2 (LGPLv2)",
+ ],
+)
Something went wrong with that request. Please try again.