Pure Python (2.*) fallback #42

Merged
merged 13 commits into from Jan 29, 2013

Projects

None yet

2 participants

@bwesterb
Contributor

There are several cases where the Cython module cannot be used.

  • When using PyPy.
  • On Google App Engine or another restricted environment.

For those cases it is useful to have a pure python implementation of msgpack.

I could release this as a separate project, but I prefer it merged upstream: it will be easier to keep the API the same.

There is another project which implements msgpack in pure Python, ( keisukefukuda/msgpack-python-pure ) but that is not a drop-in replacement.

Notes

  • Setup will automatically skip the module if it fails to build. Maybe this is not desired: it could be moved behind a commandline flag.
  • msgpack will first try to import the native module. If none is available, it will use the fallback.
  • The fallback module passes all current tests (on Python 2.*)
  • The fallback module does not yet work on Python 3. (That is why Travis complains.) I will look into that.
  • The fallback module has been written with compatibility as first priority. There is a lot opportunity to improve performance.

A benchmark

As a very rough benchmark, I timed the pack and unpack time of a 30MB msgpack file from one of my applications.

  1. Cython module 0.34; 0.24
  2. Python 15.4; 10.77
  3. PyPy 1.9 1.37; 0.43
  4. PyPy dev-version 1.26; 0.40
bwesterb added some commits Jan 28, 2013
@bwesterb bwesterb Add pure Python fallback module
Signed-off-by: Bas Westerbaan <bas@westerbaan.name>
6a28b28
@bwesterb bwesterb setup: automatically fallback to pure Python module
Signed-off-by: Bas Westerbaan <bas@westerbaan.name>
2627b6a
@bwesterb bwesterb fallback: use __pypy__.builders.StringBuilder when available
This increases performance *a lot* on PyPy.

Signed-off-by: Bas Westerbaan <bas@westerbaan.name>
69ba3c9
@bwesterb bwesterb setup: remove Python 2 only syntax 6fa0f46
@methane
Member
methane commented Jan 28, 2013

Very thanks to this PR!
I'll check, merge and release this in this week.

@methane methane was assigned Jan 29, 2013
@methane methane merged commit 4cde7f0 into msgpack:master Jan 29, 2013

1 check passed

default The Travis build passed
Details
@bwesterb
Contributor

Thanks for the improvements!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment