Python setuptools/distutils packaging #1455

wants to merge 7 commits into


None yet
4 participants

noirbizarre commented Sep 1, 2012

I ported the pymapnik2 packaging into mapnik.

As Python distribution is tied to mapnik release, I don't see the interest of having a separate project.

  • No more manual synchronization and versioning
  • Publish on PyPI as mapnik instead of mapnik2
  • Works in a virtualenv

It has been tested on Ubuntu with Mapnik 2.1.0 PPA.

You just need to have libmapnik, libmapnik-dev and mapnik-utils installed.

It works the classic Python way:

  • Install:
$ python install
  • Run tests:
$ python test
  • Make a source distribution and a binary distribution:
$ python sdist bdist
  • Publish on PyPI
$ python sdist register upload

It may need a little bit of polish for Mac OSX and Windows (I can't test it).


noirbizarre commented Sep 1, 2012

I forgot to ask: It could be very nce to have it backported on 2.1.0 and to release it on PyPI.

Right now there is no 2.1.0 release on PyPI so it's not virtualenv/pip/easy_install installable.

kiorky commented Sep 1, 2012

A big -10000000 on this pull request.

Historically, @springmeyer didn't want at first to split out the python bindings from the source tree of mapnik.
Naturally, we have comed to the point that experiencing of repackaging the python bindings as a python egg (in source distribution term) may need to be "semi official" and then having it living on the mapnik organization straightforward to make it semi official, but also to tell users "use at your own risks".

Concerning the packaging of the python bindings, for me :

  • bindings of library for any language must be packaged using the targeted language distributions tool (rake, setuptools, maven, pear, whatever the packaging it is, it s not the point).
    • Obviously the bindings are not the library itself, the library become a dependency of the current bindings externalization. Thus, as any other dependency this bindings package may need.
    • This is just then evident that the package only need to contains files concerned by the packaging itself or the source code of only the bindings.

From where i started & speaking as a long time python user & developer for deploying in lot of bunch systems, at first mapnik2 bindings were all but practical & standards to install.

  • Linking to only one system python linked to a specific system version of mapnik, configured and installed at the same time of the library itself never match one of my real cases.
  • I practically never use the system python but one in a virtualenv, nowadays lot of people do
  • Generally, the virtualenv comes from a locally installed python, not /usr/bin/python
  • 99% of the time, i'm using zc.buildout, nevertheless you ll know that nowadays lot of people use zc.buildout. to install python packages.
  • The common point of those eco systems is the package must be be easy_installable. After that, the package is python friendly.
  • It is not that rare to have multiple pythons on the same box, i dont want to have to compile a new whole mapnik library just to have the python bindings linked of a specific python. I want to have the lib, and then to install the bindings of each specific project linked to that UNIQUE lib.

And, hey, i use a python binding, i just want to run the python dance (that s what pip/easy_install/zc.buildout do under the hood).

Coming from all that unhapiness, the solution i found was naturally to let the bindings look like any other python source distribution:

  • we have the,
  • the python bindings
  • an extension module to the Compiled c++ mapnik2 lib.
  • some tests
  • some doc
  • The data files
    We then just have to make love to the extension module described in the to be compiled/linked with the appropriate compilation flags.
    As for now the official bindings live in mapnik2 source code itself, i have made an helper to port this code to be externalized and help the developer to make a new release of the python bindings.

This is not ideal, this is just waiting for mapnik2 developers to decide to one day use only the python bindings as their base to work on mapnik/python and meanwhile to enable users to have an easy_install version of those bindings, and finally remove the python code from their mapnik2 repo.

  • On a subside note, i understand why mapnik developers want to keep the bindings inside the source tree, its just more simpler for them to release ...
    Maybe the solution is for them just to grab the bindings inside of the source directory as an extra step of their build system and wrap the construct construct the bindings from there but using under the hood a call to the python sdist dance and add a python sdist register to upload to pypi. For binary packages, they can use the same scheme to run a bdist_egg on the targeted platform.

The externalized bindings are working really well and this releasing scheme also works, the only thing we have to do is to release bindings where tests work which is not the case of the actual release, that's why i didnt do a release yet.
I will see what i have locally and push it to the master branch of the python bindings even if it is not working for you to work on it if you are in the mood of.
Ping me if you see nothing monday (03/09).

kiorky commented Sep 1, 2012

I wasnt clear on the fact that i am VERY pleased of your implication toward mapnik python bindings, but your efforts would be better targeted on making tests of the actual python code working with the actual mapnik2 lib.


noirbizarre commented Sep 1, 2012

I totally agree with you on Python packaging but I submitted this pull request because I think it's half a solution to have setuptools packaging outside the repository.

Like you said:

the bindings look like any other python source distribution:

  • we have the,
  • the python bindings
  • an extension module to the Compiled c++ mapnik2 lib.
  • some tests
  • some doc
  • ...

Look at the file, it's still the case.
It just moved it into mapnik repository and tuned it a little bit.

If you make the source release you will see that I don't package the entire mapnik sources but only the python bindings.
Installing it or building a binary release does not compile mapnik itself, only the bindings.
They are still compiling on the installed libmapnik, not the source in the tree.

I really don't think syncing manually the 2 repositories is a decent solution:

  • no sources history
  • huge commits due to the sync makes history unreadable
  • backporting Python fixes is too complicated
  • it makes setuptools distribution a second level distribution (still no 2.1.0 release)
  • it clutters the workflow (as I said backporting, releasing...)

I just want to keep bindings and packaging together because they work together.
It allows you to tests latest bindings without syncing every time.:

  1. compile/install mapnik with scons
  2. work on the packaging
  3. install or package it if needed

Right now, I think this solution doesn't remove any benefits from your works (still a standard distutils/setuptools packaging) but add the benefits of simplicity and maintainability.
As long you have libmapnik installed on your system, you can install it like any other python souce distribution and you are able:

  • publish it on PyPI
  • install it with pip or easy_install:
    • from PyPI
    • directly from github
    • from the sources
    • from a standalone source distribution
    • from a standalone binary distribution
  • launch the test using the
  • ...

If tomorrow the bindings are externalized, you already have the setuptools packaging in the repository.
The only task left will be to renamed the folders.

I agree, externalizing is the preferred solution, but I think it has to be all or nothing.
Maybe a git submodule can provide a transitionnal state.

For the test point, they are still in the mapnik repository so it has to be fixed in there.
Like you I prefer a release where all tests pass, but it's not the case and I still need to be able to work with the last python mapnik version.
Why should it be released in homebrew and PPA and not in PyPI: it's exactly the same code and tests fail for all three distributions.

kiorky commented Sep 1, 2012

looks like the stuff for 2.0.2 was commited in 82dc7b15d254da6c70a164f6d76d463a7b592de0, looking to fix what were my issues.


noirbizarre commented Sep 1, 2012

Link to the commit is not working :(
Is it mapnik/pymapnik2@82dc7b1 ?

Why not submitting it directly to the mapnik codebase ?

kiorky commented Sep 1, 2012

This workaround is just a test, it was to test escaping the exeption.

But my final conclusion is that anyway it just indicate i have a problem in my local mapnik installation.

I'm on my way to setup a new cleanup test sandbox.

kiorky commented Sep 1, 2012

Concerning your history issues, as soon as @springmeyer decide to let the python bindings living outside of mapnik source tree we can:

  • clone the mapnik bindings, delete all not python stuff, make some rebase to swap the history clean and have the python code history saved.
  • backport all my releasing stuff in this new layout and adapt it.
  • push --force to this repository to replace it.

Until such a decision is done and until i'm not dead or find another any other not yet found decent solution, i will only support those externalized bindings.

kiorky commented Sep 2, 2012

and the github code, there is lot of improvments in the distutils port of the build from 2.0.x


springmeyer commented Sep 3, 2012

Hey guys. Just seeing this - I will take a closer look and advise. I generally like the idea of starting to use distutils more, even if the bindings are kept in core. Could be a good foundation for moving them out at some point - which I'm open to, but only at a a major release, like 3.x.

kiorky commented Sep 3, 2012

That what is said now for years without any progress.
I will just continue to maintain my 'pymapnik2', going back to silence.


noirbizarre commented Sep 4, 2012

I agree, it will ease you the work.

Split bindings for 3.0 seams OK for me.

kiorky commented Sep 4, 2012

In the idea its fine, in the facts not.
A long time ago, pymapnik2 born for that purpose to be integrated in a very near future, some years later im just maintaining again & again this fork ;)


springmeyer commented Oct 3, 2012

@noirbizarre - would it be possible to move these files from the root into bindings/python/, as they only pertain to the python bindings (which are an optional part of mapnik)?


noirbizarre commented Oct 22, 2012

Sorry for answering late.
Having the into the root allwos installing directly from git repository.
pip doesn't support yet installing from subfolder (but a work is in progress: pypa/pip#526 ).

If you want me to move the files into bindings/python, I need to modify them before merging.


springmeyer commented Oct 22, 2012

What do you think about moving all related files except into bindings/python?


noirbizarre commented Oct 23, 2012

I moved rst files into bindings/python and removed the setup.cfg (using environment variables instead to configure nose).

The and the are mandatory so I left them into the root.

@sgillies sgillies assigned sgillies and unassigned sgillies Sep 4, 2014


sgillies commented Sep 4, 2014

Moving this to #2408.

@sgillies sgillies closed this Sep 4, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment