memoryview objects are not fully supported

[himikof]

memoryview is the python3 type for non-owning memory buffer objects, also backported to python 2.7. unpackb and Unpacker.feed should unpack them without copying, and packing functions should handle them the same as bytes.
The current support is quite limited due to multiple issues:

Python 3.3

>>> msgpack._unpacker.unpackb(memoryview(b'\x91\xc3'))
[True]
>>> msgpack.fallback.unpackb(memoryview(b'\x91\xc3'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nofitserov/.local/lib64/python3.3/site-packages/msgpack/fallback.py", line 93, in unpackb
    ret = unpacker._fb_unpack()
  File "/home/nofitserov/.local/lib64/python3.3/site-packages/msgpack/fallback.py", line 383, in _fb_unpack
    typ, n, obj = self._read_header(execute, write_bytes)
  File "/home/nofitserov/.local/lib64/python3.3/site-packages/msgpack/fallback.py", line 274, in _read_header
    b = ord(c)
TypeError: ord() expected string of length 1, but memoryview found
>>> msgpack._packer.Packer().pack(memoryview(b'abc'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "_packer.pyx", line 224, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:224)
  File "_packer.pyx", line 226, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:226)
  File "_packer.pyx", line 221, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:221)
TypeError: can't serialize <memory at 0x7f37c3e41460>
>>> msgpack.fallback.Packer().pack(memoryview(b'abc'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nofitserov/.local/lib64/python3.3/site-packages/msgpack/fallback.py", line 618, in pack
    self._pack(obj)
  File "/home/nofitserov/.local/lib64/python3.3/site-packages/msgpack/fallback.py", line 615, in _pack
    raise TypeError("Cannot serialize %r" % obj)
TypeError: Cannot serialize <memory at 0x7f37c3e41390>

Python 2.7

>>> msgpack._unpacker.unpackb(memoryview(b'\x91\xc3'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "_unpacker.pyx", line 105, in msgpack._unpacker.unpackb (msgpack/_unpacker.cpp:105)
TypeError: expected a readable buffer object
>>> msgpack.fallback.unpackb(memoryview(b'\x91\xc3'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nofitserov/.local/lib64/python2.7/site-packages/msgpack/fallback.py", line 93, in unpackb
    ret = unpacker._fb_unpack()
  File "/home/nofitserov/.local/lib64/python2.7/site-packages/msgpack/fallback.py", line 381, in _fb_unpack
    typ, n, obj = self._read_header(execute, write_bytes)
  File "/home/nofitserov/.local/lib64/python2.7/site-packages/msgpack/fallback.py", line 272, in _read_header
    b = ord(c)
TypeError: ord() expected string of length 1, but memoryview found
>>> msgpack._packer.Packer().pack(memoryview(b'abc'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "_packer.pyx", line 206, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:206)
  File "_packer.pyx", line 208, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:208)
  File "_packer.pyx", line 203, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:203)
TypeError: can't serialize <memory at 0x7f506e40ddf8>
>>> msgpack.fallback.Packer().pack(memoryview(b'abc'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nofitserov/.local/lib64/python2.7/site-packages/msgpack/fallback.py", line 616, in pack
    self._pack(obj)
  File "/home/nofitserov/.local/lib64/python2.7/site-packages/msgpack/fallback.py", line 613, in _pack
    raise TypeError("Cannot serialize %r" % obj)
TypeError: Cannot serialize <memory at 0x7f506e40ddf8>

The only available workaround right now is to explicitly convert memoryview objects to bytes, needlessly copying the contents, which degrades performance, especially for unpacking large objects.

[ThomasWaldmann]

+1 for memoryviews.

[platz]

Hello,
i found this issue and would like to ask whether it made some progress.
On a Ubuntu 14.04 machine i also ran into this problem but i could fix it quite fast. Unfortunately i didn't make notes on how i fixed it.
At the moment i'm working on a Windows 7 machine and so far i can't solve this problem.
"TypeError: ord() expected string of length 1, but memoryview found"

[platz]

On the Windows 7 machine i downgraded to Python2.7.5 and with this version i don't get the error.

[jfolz]

Packing memoryviews is in master, see: #161
Please try master branch to see if it does what you want. I've probably packed billions of memoryviews by now and have yet to encounter a single problem.

[himikof]

True, packing memoryview objects works in master now. Unpacking from/feeding to unpacker still does not work for memoryviews, so this issue is not resolved fully yet.

[jfolz]

The Cython unpacker uses the old buffer protocol. Which is ironic because it seems to work fine in Python 3 where the old protocol is deprecated, and breaks in Python 2 where it's fine to use it.
For Cython typed memoryviews might be a good idea. The pure Python fallback can use memoryview as well now since 2.6 is no longer supported.
Might take a look at that later. Looking at the code now I also see some potential problems with memoryview packing in fallback, so I guess I'll have to look at that as well.

[nxsofsys]

I'm using unpacker, and I've got serialization error with memoryviews. So problem exists.
EDIT:
unpacker does not work correctly with memoryviews only in MSGPACK_PUREPYTHON mode. Cython version seems is OK.
EDIT 2:
I've found that python version uses "len(obj)" to compute size of memoryview, but obj.nbytes should be used instead, more info https://docs.python.org/3/library/stdtypes.html#memoryview.

[jfolz]

@nxsofsys PR #182 should fix that.

[methane]

Can I close this issue for now?

[nightsailer]

It cannot work on PyPy (master latest commit)

msgpack.unpackb(memoryview(b'\x91\xc3'))
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/bak/vendors/msgpack-python/msgpack/fallback.py", line 106, in unpackb
    ret = unpacker._fb_unpack()
  File "/bak/vendors/msgpack-python/msgpack/fallback.py", line 508, in _fb_unpack
    typ, n, obj = self._read_header(execute, write_bytes)
  File "/bak/vendors/msgpack-python/msgpack/fallback.py", line 357, in _read_header
    b = ord(c)
TypeError: ord() expected string of length 1, but memoryview found

[nxsofsys]

msgpack.unpackb(memoryview(b'\x91\xc3'))

I think memoryview should be packed before unpack. Can you try following code:

msgpack.unpackb(msgpack.packb(memoryview(b'\x91\xc3')))

[methane]

Sorry, #182 fixed only Packer.

[methane]

From CPython 3.4, bytearray is good fifo buffer.
Anyone knows that latest PyPy's bytearray is good fifo buffer?
If it's efficient, I want to move from byte of array to bytearray.