Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

PyZMQ's use of memoryviews breaks reconstruction of numpy arrays #478

Closed
minrk opened this Issue · 3 comments

2 participants

@minrk
Owner

Until 9e418fa, IPython would use the Message.buffer attribute for noncopying sends, which would be a memoryview on Python >= 2.7. Since pyzmq does not preserve the format and itemsize attributes across the network, arrays were not being reconstructed properly, resulting in improperly shaped arrays, and the error message:
ValueError: total size of new array must be unchanged

The above commit uses buffer(msg) in place of msg.buffer, to ensure that it is a buffer, rather than a memoryview, so we are safe on Python 2.7, but figuring out the memoryview issue is necessary for Python 3, where buffer objects are no longer available.

@minrk
Owner

See here for related Python Issue (still open in 3.3) for changing itemsize in memoryviews. I don't see a way to create a new memoryview of the same memory with a different itemsize.

Since this is specific to numpy arrays, the real question is this:
Is it possible to create a numpy array from a memoryview ignoring the memoryview's metadata (itemsize, format), and let the dtype dictate interpretation of the memory?

@fperez
Owner

This might be worth pinging the numpy list for, I'm not sure anyone there has even noticed this problem...

@fperez
Owner

I made a new py3k label

@minrk minrk referenced this issue from a commit in minrk/ipython
@minrk minrk don't special case for py3k+numpy
py3k+numpy non-copying recv works fine now, with released pyzmq.  There was no need to make any changes in pyzmq.

closes gh-478
22b4e57
@minrk minrk closed this issue from a commit
@minrk minrk don't special case for py3k+numpy
py3k+numpy non-copying recv works fine now, with released pyzmq.  There was no need to make any changes in pyzmq.

closes gh-478, closes gh-587 (rebased)
487466d
@minrk minrk closed this in 487466d
@fperez fperez referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@damianavila damianavila referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@mattvonrocketstein mattvonrocketstein referenced this issue from a commit in mattvonrocketstein/ipython
@minrk minrk don't special case for py3k+numpy
py3k+numpy non-copying recv works fine now, with released pyzmq.  There was no need to make any changes in pyzmq.

closes gh-478, closes gh-587 (rebased)
398ed43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.