Add support for MessagePack

[scls19fr]

Hello,

Related #4200
Supporting an other binary serializer/deserializer such as MessagePack could be a nice feature also.

I noticed
https://github.com/msgpack/msgpack-python
and https://github.com/vsergeev/u-msgpack-python

but I'm not sure if it's currently available for micropython.

Is there unit tests framework such as pytest on micropython because u-msgpack-python uses it? ...

Not sure pytest is available on micropython according construct/construct#224 (comment)

Kind regards

[EcmaXp]

https://github.com/EcmaXp/OpenPie/blob/master/oprom/msgpack.c

[EcmaXp]

mpack library is good for embedding to micropython.
https://github.com/ludocode/mpack

[kingsjl]

mpack library is good for embedding to micropython.
https://github.com/ludocode/mpack

how to use?

[stinos]

Don't know about mpack, but the pure Python implementation of the 'official' MessagePack Python implementation (i.e. what you get when installing via pip or conda, all .py files from https://github.com/msgpack/msgpack-python/tree/master/msgpack) almost works out of the box for basic usecases. The only problem is it uses multiple inheritance for it's exceptions, which MicroPython doesn't support, but is fairly easy to fix: modify 2 lines and make FormatError and StackError derive only from UnpackException.

[smurfix]

Well, the "official" msgpack implementation is inefficient as heck. Encoding a single three-element dict takes ~1k memory. Owch.

I'm working on simplifying and rewriting it so that it uses (small) buffers, instead of copying data all over the place. Will be part of my microfuse project (https://github.com/M-o-a-T/microfuse) shortly.

[stinos]

Well, the "official" msgpack implementation is inefficient as heck. Encoding a single three-element dict takes ~1k memory. Owch.

Which implementation, one of the Python implementations? Or the C one?

[peterhinch]

my microfuse project (https://github.com/M-o-a-T/microfuse)

404

[smurfix]

Well, the "official" msgpack implementation is inefficient as heck. Encoding a single three-element dict takes ~1k memory. Owch.

Which implementation, one of the Python implementations? Or the C one?

Python. I haven't looked at details of the C implementation because I have no free week to write a MicroPython interface for it. Mostly-fixing the Python code is childs play by comparison.

my microfuse project (https://github.com/M-o-a-T/microfuse)

404

The original mail had a typo (github.om). Since fixed, please try again.

[stinos]

Python. I haven't looked at details of the C implementation because I have no free week to write a MicroPython interface for it. Mostly-fixing the Python code is childs play by comparison.

Assuming you mean the one linked above, did you also check https://github.com/vsergeev/u-msgpack-python ?

[smurfix]

Assuming you mean the one linked above, did you also check https://github.com/vsergeev/u-msgpack-python ?

I just looked at it. Same inefficiency problems. Worse, the author doesn't even seem to know that you can write

if py2:
    def proc():
        …  # py2 version
else:
    def proc():
        …  # py3 version

. Even worse it doesn't support unpacking a stream of messages, which is a required exercise with msgpack because the protocol is self-delimiting.

One use case is OTA updates. If you can't even send a 4k bytestring to or from the device without running into intermittent errors, even though gc says you have >30k of free memory, you have a problem. Thus you need to return large byte objects as memoryview into the decoder's input buffer, which must itself be a static bytearray.

On the plus side my rewrite is almost done and all I need to do is test (and document …) it.

On the minus side there's a memory error in MicroPython somewhere in the bytearray slicing/assignment code. I haven't been able to reproduce it yet, but I frequently get this:

assertion "ATB_GET_KIND(block) == AT_HEAD" failed: file "/home/smurf/src/micropython/py/gc.c", line 591, function: gc_free

[stinos]

The original mail had a typo (github.om). Since fixed, please try again.

Still 404

Worse, the author doesn't even seem to know that you can write if py2

What is py2?

On the minus side there's a memory error in MicroPython somewhere in the bytearray slicing/assignment code.

Is this the same as #7261 and if so did you try with a version which includes the fix for that? Otherwise please create an issue with code to reproduce this.

[smurfix]

The original mail had a typo (github.om). Since fixed, please try again.

Still 404

I have no idea which link you're trying, but https://github.com/M-o-a-T/microfuse works perfectly for me.

Worse, the author doesn't even seem to know that you can write if py2

What is py2?

Some check whether you're running on python2 or python3. Sorry, I thought that'd be obvious.

On the minus side there's a memory error in MicroPython somewhere in the bytearray slicing/assignment code.

Is this the same as #7261 and if so did you try with a version which includes the fix for that? Otherwise please create an issue with code to reproduce this.

Unfortunately not. I will as soon as I can actually reproduce the thing.

[stinos]

but https://github.com/M-o-a-T/microfuse works perfectly for me

Test in a private browser window or when not logged in into github, it has probably non-public accessibility.

Some check whether you're running on python2 or python3. Sorry, I thought that'd be obvious.

That part is obvious, what is not obvious is where you get it from, and/or what you really mean with your rant :) But it's off-topic anyway.

[dpgeorge]

See this new library by @peterhinch : https://github.com/peterhinch/micropython-msgpack