Skip to content
This repository

PyZMQ's use of memoryviews breaks reconstruction of numpy arrays #478

Closed
minrk opened this Issue · 3 comments

2 participants

Min RK Fernando Perez
Min RK
Owner
minrk commented

Until 9e418fa, IPython would use the Message.buffer attribute for noncopying sends, which would be a memoryview on Python >= 2.7. Since pyzmq does not preserve the format and itemsize attributes across the network, arrays were not being reconstructed properly, resulting in improperly shaped arrays, and the error message:
ValueError: total size of new array must be unchanged

The above commit uses buffer(msg) in place of msg.buffer, to ensure that it is a buffer, rather than a memoryview, so we are safe on Python 2.7, but figuring out the memoryview issue is necessary for Python 3, where buffer objects are no longer available.

Min RK
Owner
minrk commented

See here for related Python Issue (still open in 3.3) for changing itemsize in memoryviews. I don't see a way to create a new memoryview of the same memory with a different itemsize.

Since this is specific to numpy arrays, the real question is this:
Is it possible to create a numpy array from a memoryview ignoring the memoryview's metadata (itemsize, format), and let the dtype dictate interpretation of the memory?

Fernando Perez
Owner
fperez commented

This might be worth pinging the numpy list for, I'm not sure anyone there has even noticed this problem...

Fernando Perez
Owner
fperez commented

I made a new py3k label

Min RK minrk referenced this issue from a commit in minrk/ipython
Min RK don't special case for py3k+numpy
py3k+numpy non-copying recv works fine now, with released pyzmq.  There was no need to make any changes in pyzmq.

closes gh-478
22b4e57
Min RK minrk closed this issue from a commit
Min RK don't special case for py3k+numpy
py3k+numpy non-copying recv works fine now, with released pyzmq.  There was no need to make any changes in pyzmq.

closes gh-478, closes gh-587 (rebased)
487466d
Min RK minrk closed this in 487466d
Fernando Perez fperez referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Damián Avila damianavila referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.