Allow objects implemented in pure Python to export PEP 3118 buffers

[ncoghlan]

BPO	13797
Nosy	@rhettinger, @mdickinson, @ncoghlan, @pitrou, @scoder, @alex, @ericsnowcurrently, @vadmium
Dependencies	bpo-10181: Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Files	llindstrom-newbuffer-0dd8ba1c2c2c.tar.gz: Extension module containing BufferMixin and Py_buffer. Source.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-01-16.10:29:10.732>
labels = ['interpreter-core']
title = 'Allow objects implemented in pure Python to export PEP 3118 buffers'
updated_at = <Date 2016-03-23.04:03:54.804>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2016-03-23.04:03:54.804>
actor = 'siemer'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2012-01-16.10:29:10.732>
creator = 'ncoghlan'
dependencies = ['10181']
files = ['31329']
hgrepos = []
issue_num = 13797
keywords = []
message_count = 11.0
messages = ['151347', '151351', '154683', '154684', '154692', '154855', '156440', '156479', '156485', '163058', '195436']
nosy_count = 13.0
nosy_names = ['rhettinger', 'siemer', 'mark.dickinson', 'ncoghlan', 'kermode', 'pitrou', 'scoder', 'Arfrever', 'alex', 'sbt', 'eric.snow', 'martin.panter', 'Jacob.Holm']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue13797'
versions = ['Python 3.6']

[ncoghlan]

Currently, there's no straightforward way to create new classes in Python that delegate the PEP-3118 buffer APIs to another object that supports them (e.g. bytes, bytearray, memoryview, array.array, a NumPy array).

I see a few possible ways to go about this:

1. Add a single new special method, __buffer__. This creates an entry in the underlying getbuffer slot, but leaves releasebuffer empty.

The getbuffer wrapper then calls __buffer__ and delegates the getbuffer call to the returned object.

Any created Py_buffer instances refer directly to the underlying PEP-3118 supporting object, not the original object that defines __buffer__

The downside of this approach is that you can't implement a completely new PEP-3118 supporting object in pure Python (since you don't get access to the flags)

2. As above, but any Py_buffer structs returned are updated to refer to the original object, not the underlying one. A releasebuffer wrapper is also created, that calls back into __buffer__ to retrieve the object again. This approach has problems if __buffer__ can change to refer to different objects over time.

3. Define separate __getbuffer__ and __releasebuffer__ special methods. All C types implementing the PEP-3118 slots get these special methods automatically (as with any other slot). The flag variables are passed into __getbuffer__, and it is expected to return a memoryview object defining the view. The underlying getbuffer slot wrapper stores the current object in the Py_buffer "obj" slot, and the returned memoryview in "internal". The rest of the Py_buffer fields are populated based on the memoryview contents.

The releasebuffer wrapper then reverses this process, by passing the memoryview instance from the internal slot into the __releasebuffer__ call.

[ncoghlan]

Reviewing Stefan's work on bpo-10181 suggests to me that option 3 is the only feasible approach.

(Allowing memoryview subclasses will permit types to pass their own state between __getbuffer__ and __releasebuffer__)

[skrah]

I'm trying to understand what you want to be able to write. Do you
perhaps have a short example? Also, to get the bigger picture: Is
this related to your strview proposal?

http://mail.python.org/pipermail/python-ideas/2011-December/012993.html

[ncoghlan]

Consider a Python wrapper around a bytes object, or mmap or similar that
wants to pass PEP-3118 buffer requests through to the underlying object.
Currently, there's no way to write such a delegation - the delegating class
has to be written in C.

[ncoghlan]

To answer your other question, no, strview isn't related - that's strictly a PEP-3118 *consumer*, which is well supported from the Python side now that memoryview is fixed.

The trick will be to allow a Python implemented object to be a PEP-3118 exporter *without* having to inherit from a C implemented type that does the slot mapping. Since PEP-3118 didn't describe a Python level API for the protocol, it may actually require a new PEP.

One example for what you could do with it: use the new memoryview.cast() to provide multidimensional views on an exporter that only supports 1D exports natively.

[scoder]

FWIW, Cython lets user code implement the buffer interface for extension types using the special methods "__getbuffer__()" and "__releasebuffer__()", so providing the same methods (although with a different signature) also for normal Python types would IMHO fit nicely.

[alex]

FWIW pypy has an __buffer__ method (used exclusively internally, AFAIK), which has semantics similar to your first proposal.

[scoder]

Ok, just for the record: a single __buffer__() special method with delegation-only semantics would also work for Cython. Taking this path would provide a cleaner separation of the (then delegation-only) Python level protocol and the (all purpose) C level protocol. I think I'd prefer that.

I assume __buffer__() takes no arguments, right?

[ncoghlan]

The reason I don't particularly like the "delegation only" API is that the combination of the new memoryview implementation and bytes/mmap/etc to get a flat region of memory to play with means you could do some quite interesting things entirely at the Python level.

If you couldn't even use memoryview.cast() to change the shape of the exported memory, it makes the Python level API clearly inferior in power compared to what you can do in C.

[ncoghlan]

I suggest a PEP for 3.4 as the best way forward for this.

[kermode]

A fourth way to add __getbuffer__ and __releasebuffer__ special methods to a Python class is through a new base class/mixin. The Py_buffer struct pointer passed to __getbuffer__ and __releasebuffer__ is wrapped with another special object type, which exposes the C struct fields as attributes. The base class is a direct subclass of object. Both the base and wrapper classes are implemented in an extension module.

The use case is unit testing. A memoryview object may be adequate for simple C-contiguous arrays, but fails with F-contiguous or any non-contiguous array. It cannot export any arbitrary view, especially a deliberately invalid view, useful for testing error handlers.

My implementation is at https://bitbucket.org/llindstrom/newbuffer . It is used in Pygame 1.9.2 unit tests: https://bitbucket.org/pygame/pygame . The newbuffer module exports two classes, BufferMixin and Py_buffer. The BufferMixin class adds bf_getbuffer and bf_releasebuffer slot functions to base class PyObject, nothing more. The Py_buffer class is low level, exposing pointer fields as integer addresses. It is designed for use with ctypes and is modelled on ctypes.Structure.

Some limitations. In a class declaration, BufferMixin must be first in the inheritance list for the subclass to inherit the PEP-3118 interface. A buffer interface cannot be added to a builtin.

I suggest this is a practical alternative to the three previously proposed solutions to this issue. This option takes its precedence from the ctypes module, which adds dangerous memory level access to Python, but optionally. It does not require modification of the base code, only an addition to the standard library. Finally, this approach has been demonstrated in a real-world application.