Supporting Unicode values.

[buriy]

In [1]: import msgpack

In [13]: msgpack.version
Out[13]: (0, 4, 7)

In [3]: msgpack.dumps(u'asd', encoding=None)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-11-a642531abdda> in <module>()
----> 1 msgpack.dumps(u'asd', encoding=None)

/usr/local/lib/python2.7/dist-packages/msgpack/__init__.pyc in packb(o, **kwargs)
     45     See :class:`Packer` for options.
     46     """
---> 47     return Packer(**kwargs).pack(o)
     48 
     49 # alias for compatibility to simplejson/marshal/pickle.

msgpack/_packer.pyx in msgpack._packer.Packer.pack (msgpack/_packer.cpp:231)()

msgpack/_packer.pyx in msgpack._packer.Packer.pack (msgpack/_packer.cpp:233)()

msgpack/_packer.pyx in msgpack._packer.Packer._pack (msgpack/_packer.cpp:173)()

TypeError: Can't encode unicode string: no encoding is specified

[methane]

In [3]: msgpack.dumps(u'asd', encoding=None)

Why encoding=None ?????

[methane]

Have you read README before filing issue?
https://github.com/msgpack/msgpack-python#string-and-binary-type

[buriy]

In [11]: msgpack.unpackb(msgpack.packb(u'Hi',use_bin_type=True))
Out[11]: 'Hi'
Doesn't work with Unicode either -- instead encoding it to str.
Please suggest a way to send Unicode strings without converting them to str.

[jfolz]

It's a bit unfortunate, but you also need to tell the unpacker what encoding to use. Adding that bit of info to the serialized data would have been grand (or generally just saying that all strings are UTF-8 and be done with it), but that's not how the format works. You could always define an extension type that includes an encoding hint.

Regarding Unicode: You cannot directly serialize it. Unicode is just a specification that maps symbols to code-points of variable width. If you want to do anything with Unicode, you need to encode these code-points using an encoding of your liking like the ever popular UTF-8, which includes hints on the number of bytes in encoded code-points so they can be decoded again.

[methane]

@jfolz Thank you for explain.

[buriy]

It seems I was wrong, and the library handles it correctly.

What I missed was "encoding='utf-8'" in unpacker:

In [2]: msgpack.unpackb(msgpack.packb(u'Hi',use_bin_type=True), encoding='utf-8')
Out[2]: u'Hi'

Compare this to

In [2]: msgpack.unpackb(msgpack.packb(u'Hi',use_bin_type=True))
Out[2]: 'Hi'

I understand that you can't set unpackb default option encoding='utf-8' like you did for packb, but that is a complication which breaks arguments symmetry and it's easier to misunderstand what's going on.

Probably, this docs section needs rewriting, it's very hard to understand what's going on inside the library from several reads, and I got it wrong from experimentation because of the complication above.

[jfolz]

@buriy If you set encoding='UTF-8' in unpackb you will get Unicode objects. As a side-effect unpackb will now convert all your strings to Unicode even it they were byte strings before, which may become an issue. use_bin_type=True in packb prevents that from happening. This may break with old implementations of msgpack as the binary type was added later.
If you (understandably) do not want to wrap all your strings with extension objects your only option is to enforce UTF-8 as encoding for all Unicode strings in your interface contract.

I'm sorry if that is not a satisfactory solution, but it is the only one. Character encoding is nasty business that's easy to get wrong, but it's a necessary evil. It is literally impossible to even store unencoded Unicode. Your interpreter may pretend that your strings are Unicode, but they are not. CPython 2.x internally represents Unicode as UCS2 or UCS4 encoded strings, depending on whether it was built with wide character support. That means your bog-standard pre-built CPython most likely pretends to support Unicode but can only handle the most common characters that fit into 2 bytes. CPython3.3+ will choose one of ASCII, Latin-1 (UCS1), UCS2, or UCS4 depending on which gives the most compact representation. Jython, PyPy, etc. might use others. That is the reason why it is impossible to create a memoryview of a Unicode object. What the physical memory behind a Unicode looks like entirely depends on what interpreter you are using and the options it was built with, and for CPython3.3+ even the contents of the string. I.e., adding one character can transparently change the encoding.

[methane]

@buriy I'm not native English speaker and writing English is too hard to me.
Contribution is welcome.

[buriy]

@jfolz thanks a lot, actually my problem was that I was using mprpc library which uses msgpack under cover, and the lib doesn't provide a set of sensible defaults for strings passing.
And then a lot of leaky abstractions happened!
Of course, I understand that Unicode is not a format, but a specification for a lot of formats, and e.g. UTF8 or UCS2 are real formats. What I wanted though is not to lose information that this was a Python Unicode string -- which is a completely different point from Unicode encodings. What I missed was that it wasn't lost after use_bin_type=True because then python str is packed into a different (binary) format. This is exactly what a Python3 user would expect when working with file encodings.
I feel that docs just need to be improved, providing internal data on how this is implemented internally and why it is so -- would save me hours of learning its internal structure and reading rants on all this.
I'm sure that I'm not the first person coming into this. Some users might even stop considering adoption of this library for their needs because of getting Unicode handling wrong.

[buriy]

@methane and what if you would introduce use_bin_type parameter for unpacker?
Internally that could just set encoding='utf-8' , but this will add API symmetry.

Before:

In [2]: msgpack.unpackb(msgpack.packb(u'Hi',use_bin_type=True), encoding='utf-8')
Out[2]: u'Hi'

After:

In [2]: msgpack.unpackb(msgpack.packb(u'Hi',use_bin_type=True), use_bin_type=True)
Out[2]: u'Hi'

[methane]

@buriy json.loads has neither ensure_ascii or sort_keys options.
Deserializer should not have option to control how to serialize only for API symmetry.

So I don't want to add use_bin_type option (which means "Use new bin msgpack type for serializing bytes") to unpacker.

Regardless of your suggestion, I know current API is not ideal.
I want to use new spec by default from long ago. And API change should be made when changing default.
But msgpack is cross language format. I choose waiting many libraries supports new spec.
I think this year is the time to change.

I wrote down my ideas on #191.