PEP 574: update

[pitrou]

Make it a requirement that buffers are contiguous. Efficient tooling for non-contiguous buffers is not available from CPython currently, and consumers shouldn't be required to implement their own handling.

Also add a bit of trivia about an undocumented ZODB-specific cPickle hook.

[pitrou]

cc @ncoghlan

Also cc @skrah about the rudimentary support for non-contiguous buffers (including the slow copy to contiguous).

[pitrou]

Actually, it seems I may have to make the contiguity requirement a bit stronger and mandate C-contiguous buffers. The reason is that, in pure Python, there doesn't seem to be any way to read the byte contents of a Fortran-contiguous buffer in physical memory order. Consider this:

>>> array = _testbuffer.ndarray(list(range(6)), format="B", shape=(3, 2), strides=(1, 3))
>>> array.tolist()
[[0, 3], [1, 4], [2, 5]]
>>> array.c_contiguous
False
>>> array.f_contiguous
True
>>> m = memoryview(array)
>>> m.tobytes()
b'\x00\x03\x01\x04\x02\x05'
>>> bytes(m)
b'\x00\x03\x01\x04\x02\x05'

Or with Numpy:

>>> a = np.arange(12, dtype='int8').reshape((3,4))                                                                                                              
>>> bytes(a)                                                                                                                                                    
b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b'
>>> bytes(a.T)                                                                                                                                                  
b'\x00\x04\x08\x01\x05\t\x02\x06\n\x03\x07\x0b'
>>> memoryview(a).tobytes()                                                                                                                                     
b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b'
>>> memoryview(a.T).tobytes()                                                                                                                                   
b'\x00\x04\x08\x01\x05\t\x02\x06\n\x03\x07\x0b'

... the logical transposition affects the bytes copy, even though the underlying memory contents are identical.

This is not a problem in C, which has naturally access to the memory pointed to by a Py_buffer, but the pure Python pickle implementation does not look like it will be able to deal with Fortran-contiguous buffers. Perhaps we'll be later able to relax the restriction, if we add some appropriate API to memoryview.

It's a bit unfortunate. @skrah do you see a way of achieving this?

[ncoghlan]

Hmm, I thought memoryview(original).cast('B') could handle that, but reading https://docs.python.org/3/library/stdtypes.html#memoryview.cast makes me suspect I never tried it with a multi-dimensional input (since the docs are quite explicit about only handling 1D views).

[skrah]

It can handle ND input:

>> x = np.array(list(range(6)), dtype="int64").reshape(2,3) >> m = memoryview(x) >> m.cast('B').tolist()

[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0]

[pitrou]

But not Fortran-contiguous data, which is the issue here:

>>> m = memoryview(x.T)                                                                                                                               
>>> m.cast('B').tobytes()                                                                                                                             
Traceback (most recent call last):
  File "<ipython-input-6-80a67b12e997>", line 1, in <module>
    m.cast('B').tobytes()
TypeError: memoryview: casts are restricted to C-contiguous views

[ncoghlan]

D'oh :(

I don't believe there was any philosophical objection behind that omission, though - just a lack of a concrete use case to justify the extra complexity in the code.

It does raise a question for the pickle 5 spec though: even if the memoryview limitation is resolved in Python 3.8, would the pickle5 backport library be able to unpickle Fortran-contiguous data on older versions?

[pitrou]

even if the memoryview limitation is resolved in Python 3.8, would the pickle5 backport library be able to unpickle Fortran-contiguous data on older versions?

Unpickling will depend on the __reduce_ex__ implementation rather than on the Unpickler implementation, AFAICT. However, I would have to test it to make sure...

[pitrou]

Hmm... Thinking about this again, I could add the required API to PickleBuffer so that it doesn't have to depend on some Python 3.8 memoryview improvements. I think the implementation should be reasonably simple.

[ncoghlan]

Having it work in the pickle5 API backport would definitely be desirable. That said, adding support for F-contiguous data in memoryview for 3.8+ would likely still be desirable for testing purposes at that point.