Add typing.py class describing a PEP 3118 buffer object

[dmoisset]

BPO	27501
Nosy	@gvanrossum, @rhettinger, @jstasiak, @ilevkivskyi, @dmoisset
Files	buffer.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-07-12.18:07:23.800>
labels = ['type-feature', 'library']
title = 'Add typing.py class describing a PEP 3118 buffer object'
updated_at = <Date 2020-05-19.18:11:43.834>
user = 'https://github.com/dmoisset'

bugs.python.org fields:

activity = <Date 2020-05-19.18:11:43.834>
actor = 'gvanrossum'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-07-12.18:07:23.800>
creator = 'Daniel Moisset'
dependencies = []
files = ['43719']
hgrepos = []
issue_num = 27501
keywords = ['patch']
message_count = 8.0
messages = ['270261', '270315', '270321', '270323', '270334', '270412', '270437', '357522']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'rhettinger', 'jstasiak', 'levkivskyi', 'Daniel Moisset']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue27501'
versions = ['Python 3.6']

[dmoisset]

The buffer protocol is a low level C protocol, but even if it doesn't expose a specific python API, it's currently not possible to say "this object implements the buffer protocol" in Python

This might be useful for some applications, including PEP-484 typing hints. It would be useful to have a collections.abc.Buffer which already registers the types that already support the protocol (bytes, strings, array.array, struct, mmap, ctype arrays/pointers, etc)

[rhettinger]

I'll defer to typing gurus as to whether something like this might make sense in typing.py, but it doesn't make much sense as a collections.abc without an API exposed in pure python and without useful mixin methods. By way of comparison, consider that the named tuple protocol is in typing.NamedTuple but not in collections.abc; instead, we just use _fields to recognize it when the need arises (which is almost never).

I would like collections.abc to remain a place for ABCs that can be usefully subclassed, that provide guaranteed behaviors, and that are in fact related to collections. The module's utility will be impaired if it becomes a dumping ground for miscellaneous type hinting concepts and registrations.

Early on there was some experimentation with the "one-trick ponies" in collections.abc such as Callable and Hashable that turned-out to almost never be used in practice, so we don't want to continue to that line of development when there are more promising avenues such as typing.py.

[dmoisset]

Thanks for the feedback. Just for the context, I opened this issue also based on this email: http://permalink.gmane.org/gmane.comp.python.devel/156411

Probably there are two different discussions to have here:

1. Is the "Buffer" a useful ABC to have even if there's no python API involved? My thoughts (that come from working with typing and PEP-484 and numpy) are "yes"

2. Where is the right place to put it? I suggested collections.abc after seeing the thread I linked, although I wouldn't mind if this was in typing.py or somewhere else.

[rhettinger]

Since collections.abc is the wrong place, I'll change the title of this to typing and see if any of the typing aficionados think it is sufficiently useful (they are deliberately focusing on common use cases first since the whole static typing endeavor is exploratory and this limits of what should be done and what isn't really needed or useful isn't yet well understood).

[dmoisset]

OK. Just to give an obvious example of a place where this would be useful for annotations is in the argument for the memoryview() builtin (currently declared with "# TODO arg can be any obj supporting the buffer protocol" in the standard library typesheds)

[dmoisset]

As a description of the idea I'm attaching a rough patch with an implementation. Some tests fail because of the following:

1. I didn't find a good place to register the Buffer ABC for mmap.mmap, and array.mmap; the modeules themselves are C extensions (and it's possible but weird to register the ABC from the C API); I could import those in typing.py but then I'd make typing depend from mmap and array which also feels wrong; so I didn't do that and the related tests that I added fail.

2. I had a similar situation with io.BytesIO. In that case there's a python module and I'm doing the abc.register there, but now io imports typing, and the test_site fails because many new modules (typing and its dependencies) are imported on startup. The alternative is to make typing depend on io (in this case it could be a reasonable idea)

After these, I'm feeling that typing.py may not be the best place for this (but given that collections.abc isn't either, I'm not sure where's the right place)

[rhettinger]

Looking at the patch, I'm thinking that this endeavor isn't at all worthwhile. We don't need an ABC for everything especially if there in no pure python API involved. I recommend closing this one until there is a proven need and a clear path forward.

[ilevkivskyi]

Cross-ref to the typing issue python/typing#593. It looks like there is some interest in this feature.

[JelleZijlstra]

I just posted PEP-688 proposing a version of this idea.

[AlexWaygood]

@JelleZijlstra, do we still need to keep this issue open now that we have #102500?

[JelleZijlstra]

Agree, closing as a duplicate and let's use #102500 to track the implementation of PEP 688.