PyZMQ's use of memoryviews breaks reconstruction of numpy arrays #478

Closed
minrk opened this Issue May 28, 2011 · 3 comments

Comments

Projects
None yet
2 participants
@minrk
Member

minrk commented May 28, 2011

Until 9e418fa, IPython would use the Message.buffer attribute for noncopying sends, which would be a memoryview on Python >= 2.7. Since pyzmq does not preserve the format and itemsize attributes across the network, arrays were not being reconstructed properly, resulting in improperly shaped arrays, and the error message:
ValueError: total size of new array must be unchanged

The above commit uses buffer(msg) in place of msg.buffer, to ensure that it is a buffer, rather than a memoryview, so we are safe on Python 2.7, but figuring out the memoryview issue is necessary for Python 3, where buffer objects are no longer available.

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk May 28, 2011

Member

See here for related Python Issue (still open in 3.3) for changing itemsize in memoryviews. I don't see a way to create a new memoryview of the same memory with a different itemsize.

Since this is specific to numpy arrays, the real question is this:
Is it possible to create a numpy array from a memoryview ignoring the memoryview's metadata (itemsize, format), and let the dtype dictate interpretation of the memory?

Member

minrk commented May 28, 2011

See here for related Python Issue (still open in 3.3) for changing itemsize in memoryviews. I don't see a way to create a new memoryview of the same memory with a different itemsize.

Since this is specific to numpy arrays, the real question is this:
Is it possible to create a numpy array from a memoryview ignoring the memoryview's metadata (itemsize, format), and let the dtype dictate interpretation of the memory?

@fperez

This comment has been minimized.

Show comment
Hide comment
@fperez

fperez May 28, 2011

Member

This might be worth pinging the numpy list for, I'm not sure anyone there has even noticed this problem...

Member

fperez commented May 28, 2011

This might be worth pinging the numpy list for, I'm not sure anyone there has even noticed this problem...

@fperez

This comment has been minimized.

Show comment
Hide comment
@fperez

fperez Jun 1, 2011

Member

I made a new py3k label

Member

fperez commented Jun 1, 2011

I made a new py3k label

minrk added a commit to minrk/ipython that referenced this issue Jul 16, 2011

don't special case for py3k+numpy
py3k+numpy non-copying recv works fine now, with released pyzmq.  There was no need to make any changes in pyzmq.

closes gh-478

@minrk minrk closed this in 487466d Aug 16, 2011

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this issue Nov 3, 2014

don't special case for py3k+numpy
py3k+numpy non-copying recv works fine now, with released pyzmq.  There was no need to make any changes in pyzmq.

closes gh-478, closes gh-587 (rebased)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment