Kwargs default/object_hook ignored for tuple/list?

[h-vetinari]

I understand that msgpack has a single array-type, that both tuple and list both get mapped to (and that this is the reason for the use_list-kwarg).

However, the list/tuple distinction is essential in Python, and one can't fully emulate the other (e.g. tuple is hashable, list isn't; list has an .append-method, tuple doesn't). For this reason, I need to reconstruct tuples and lists correctly, e.g. [(1, 'a'), (2, 'b'), (3, 'c')] -- actually, arbitrary nestings of tuple/list/dict.

Currently (v0.5.6), this does not work:

orig = [(1, 'a'), (2, 'b'), (3, 'c')]
packed = msgpack.packb(orig, use_bin_type=True)
packed
# b'\x93\x92\x01\xa1a\x92\x02\xa1b\x92\x03\xa1c'

unpacked = msgpack.unpackb(packed, raw=False)
unpacked
# [[1, 'a'], [2, 'b'], [3, 'c']]
unpacked == orig
# False

unpacked = msgpack.unpackb(packed, use_list=False, raw=False)
unpacked
# ((1, 'a'), (2, 'b'), (3, 'c'))
unpacked == orig
# False

So now to the actual question - I thought I could use the default/object_hook keywords to help me out, but this does not work for list/tuple, even though it does work for sets (and all the types covered by pandas.io.packer.encode/decode).

def encode(obj):
    if isinstance(obj, (list, set, tuple)):
        return {'t': type(obj).__name__, # type
                'v': tuple(obj)} # values
    return obj

def decode(obj):
    otype = obj.get('t', None)
    if otype is None:
        return obj
    elif otype == 'tuple':
        return obj['v']
    elif otype == 'list':
        return list(obj['v'])
    elif otype == 'set':
        return set(obj['v'])
    else:
        return obj

### works for set
orig = [{1, 'a'}, {2, 'b'}, {3, 'c'}]
packed = msgpack.packb(orig, default=encode, use_bin_type=True)
packed
# b'\x93\x82\xa1t\xa3set\xa1v\x92\x01\xa1a\x82\xa1t\xa3set\xa1v\x92\xa1b\x02\x82\xa1t\xa3set\xa1v\x92\x03\xa1c'

unpacked = msgpack.unpackb(packed, object_hook=decode, raw=False)
unpacked == orig
# True

### but does not work for tuple/list
orig = [(1, 'a'), (2, 'b'), (3, 'c')]
packed = msgpack.packb(orig, default=encode, use_bin_type=True)
packed # no change!
# b'\x93\x92\x01\xa1a\x92\x02\xa1b\x92\x03\xa1c'

unpacked = msgpack.unpackb(packed, object_hook=decode, raw=False)
unpacked
# [[1, 'a'], [2, 'b'], [3, 'c']]
unpacked == orig
# False

[methane]

Like JSON, it's not goal of msgpack.

You can use strict_types option to distinguish tuple and list when packing, and
pack them to custom ExtType. Then you can unpack ExtType manually.

I won't add any more features for Python-only usages.
msgpack is cross-language, portable format.

[h-vetinari]

I understand that it's cross-language, but it's not clear to me why default/object_hook don't seem to process list/tuple - then I can deal with it myself.

Also, it's not clear to me what the difference is between using these encode/decode hooks and an ExtType...

[methane]

I understand that it's cross-language, but it's not clear to me why default/object_hook don't seem to process list/tuple - then I can deal with it myself.

It's for compatibility with JSON library.

>>> import json
>>> json.dumps([(1,2)])
'[[1, 2]]'
>>> json.dumps([(1,2)], default=lambda x: "xxx")
'[[1, 2]]'

And I teach you about strict_types option. Why don't you try it before reply?

Also, it's not clear to me what the difference is between using these encode/decode hooks and an ExtType...

I'm not free tech support.

[h-vetinari]

And I teach you about strict_keys option. Why don't you try it before reply?

I'm on it.

Also, it's not clear to me what the difference is between using these encode/decode hooks and an ExtType...

I'm not free tech support.

No need to be like that. I've been trying to read up all the documentation I could find, but neither https://msgpack.org/ nor https://github.com/msgpack/msgpack-python does not mention strict_types explain the ExtTypes concept in depth (which codes are available/reserved? Restrictions? Nesting?), much less the differences to default/object_hook.

I started with the encode/decode because that's how pandas does it too. The barrier for ExtType was a fair bit higher without much documentation, but I'll try it now.

Finally, you haven't responded either why the default/object_hook kwargs work for so many types, but not for list/tuple...

[methane]

No need to be like that.

So you must ask question on Stackoverflow.
Posting question on here means you send spam notification to me.

and https://github.com/msgpack/msgpack-python does not mention strict_types or explain the ExtTypes concept in depth (which codes are available/reserved? Restrictions? Nesting?).

README is very short entry point for new users, not full document. You must read docstring and https://msgpack-python.readthedocs.io/en/latest/api.html

I'm bad English writer, contribution on document is welcome.

Finally, you haven't responded either why the default/object_hook kwargs work for so many types, but not for list/tuple...

I respond already. It's compatibility with JSON and strict_types allows you to use default.
"object" in object_hook is JSON's object, you must understand it if you read JSON API document.

You must use your time before respond so quickly.
Or you should use Stackoverflow to get reply from kind volunteer for you. I'm not.

[h-vetinari]

README is very short entry point for new users, not full document. You must read docstring and https://msgpack-python.readthedocs.io/en/latest/api.html

May I suggest to put this link into README.rst? I couldn't find the API documentation before, and it's not like I didn't try. It's only for v.0.4 instead of 0.5.6, but it's already more detailed than README.rst.

I'm bad English writer, contribution on document is welcome.

To the contrary, it does not appear like participation/contribution is welcome.

However, to document the solution of my problem if someone else encounters it as well -- setting strict_types=True was enough (and adapting the code a little because tuple does not work for strict_types). No extension types are necessary, although I understand now that these are independent of the encoding/deconding, but would help to save some bytes.

def encode(obj):
    if isinstance(obj, (tuple, set)):
        return {'_t': type(obj).__name__, # type
                '_v': list(obj)} # values
    return obj

def decode(obj):
    otype = obj.get('_t', None)
    if otype is None:
        return obj
    elif otype == 'tuple':
        return tuple(obj['_v'])
    elif otype == 'set':
        return set(obj['_v'])
    else:
        return obj

orig = [(1, 'a'), (2, 'b'), (3, 'c')]
packed = msgpack.packb(orig, default=encode, use_bin_type=True, strict_types=True)
packed
# b'\x93\x82\xa2_t\xa5tuple\xa2_v\x92\x01\xa1a\x82\xa2_t\xa5tuple\xa2_v\x92\x02\xa1b\x82\xa2_t\xa5tuple\xa2_v\x92\x03\xa1c'

unpacked = msgpack.unpackb(packed, object_hook=decode, raw=False)
unpacked == orig
# True

[methane]

May I suggest to put this link into README.rst? I couldn't find the API documentation before, and it's not like I didn't try.

It's only for v.0.4 instead of 0.5.6,

Uh, "0.4" is a bug. It's document for latest version.

To the contrary, it does not appear like participation/contribution is welcome.

Because it seems you didn't use your time enough to read code, past issues, etc.
Only contributors who can survey themselves can contribute.
If I need to teach everything, it's not contribution. It just bothering me.