Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

PyZMQ's use of memoryviews breaks reconstruction of numpy arrays #478

Closed
minrk opened this Issue · 3 comments

2 participants

Min RK Fernando Perez
Min RK
Owner

Until 9e418fa, IPython would use the Message.buffer attribute for noncopying sends, which would be a memoryview on Python >= 2.7. Since pyzmq does not preserve the format and itemsize attributes across the network, arrays were not being reconstructed properly, resulting in improperly shaped arrays, and the error message:
ValueError: total size of new array must be unchanged

The above commit uses buffer(msg) in place of msg.buffer, to ensure that it is a buffer, rather than a memoryview, so we are safe on Python 2.7, but figuring out the memoryview issue is necessary for Python 3, where buffer objects are no longer available.

Min RK
Owner

See here for related Python Issue (still open in 3.3) for changing itemsize in memoryviews. I don't see a way to create a new memoryview of the same memory with a different itemsize.

Since this is specific to numpy arrays, the real question is this:
Is it possible to create a numpy array from a memoryview ignoring the memoryview's metadata (itemsize, format), and let the dtype dictate interpretation of the memory?

Fernando Perez
Owner

This might be worth pinging the numpy list for, I'm not sure anyone there has even noticed this problem...

Fernando Perez
Owner

I made a new py3k label

Min RK minrk referenced this issue from a commit in minrk/ipython
Min RK minrk don't special case for py3k+numpy
py3k+numpy non-copying recv works fine now, with released pyzmq.  There was no need to make any changes in pyzmq.

closes gh-478
22b4e57
Min RK minrk closed this issue from a commit
Min RK minrk don't special case for py3k+numpy
py3k+numpy non-copying recv works fine now, with released pyzmq.  There was no need to make any changes in pyzmq.

closes gh-478, closes gh-587 (rebased)
487466d
Min RK minrk closed this in 487466d
Fernando Perez fperez referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Damián Avila damianavila referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
matthew von rocketstein mattvonrocketstein referenced this issue from a commit in mattvonrocketstein/ipython
Min RK minrk don't special case for py3k+numpy
py3k+numpy non-copying recv works fine now, with released pyzmq.  There was no need to make any changes in pyzmq.

closes gh-478, closes gh-587 (rebased)
398ed43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.