memoryview.__len__ should raise an exception for 0d buffers

[EricWieser]

BPO	39610
Nosy	@skrah, @serhiy-storchaka
PRs	gh-83791: Raise TypeError for len(memoryview_0d) #18463

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/skrah'
closed_at = None
created_at = <Date 2020-02-11.15:42:52.359>
labels = ['interpreter-core', '3.8', 'type-bug', '3.7', '3.9']
title = 'memoryview.__len__ should raise an exception for 0d buffers'
updated_at = <Date 2020-09-07.17:25:22.315>
user = 'https://bugs.python.org/EricWieser'

bugs.python.org fields:

activity = <Date 2020-09-07.17:25:22.315>
actor = 'skrah'
assignee = 'skrah'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2020-02-11.15:42:52.359>
creator = 'Eric Wieser'
dependencies = []
files = []
hgrepos = []
issue_num = 39610
keywords = ['patch']
message_count = 7.0
messages = ['361821', '361840', '363159', '376512', '376518', '376519', '376520']
nosy_count = 3.0
nosy_names = ['skrah', 'serhiy.storchaka', 'Eric Wieser']
pr_nums = ['18463']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39610'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[EricWieser]

Right now, the behavior is:

>>> import numpy as np
>>> arr_0d = np.array(42)
>>> mem_0d = memoryview(arr_0d)
>>> len(mem_0d)
1
>>> mem_0d[0]
TypeError: invalid indexing of 0-dim memory

It seems bizarre to have this object pretend to be a sequence when you ask for its length, yet not behave like one when you actually try to use this length. I'd suggest cpython should behave like numpy here, and fail:

>>> len(arr_0d)
TypeError: len() of unsized object

Perhaps TypeError: cannot get length of 0-dim memory would be more appropriate as a message.

---

Wasn't sure how to classify this, feel free to reclassify

[skrah]

The change looks reasonable, but unfortunately this is a long-standing
behavior that originates from before the Python-3.3 memoryview rewrite.

It is also present in 2.7 (the previous implementation) and documented
in 3.3:

https://docs.python.org/3/library/stdtypes.html#memoryview

I think I documented it because it looked strange back then, too.
So I've to think a bit what will be affected and how this behavior
came into place.

[EricWieser]

Thanks for pointing out the docs reference, I updated the patch to reword that section.

There's a sentence right before the one you draw attention to which to me reads as another argument to change this:

``len(view)`` is equal to the length of :class:`~memoryview.tolist`

On Python 2.7, this gives

>>> len(view_0d)
1
>>> len(view_0d.tolist())
NotImplementedError: tolist() only supports one-dimensional objects

On Python 3.8 before my patch, this gives:

>>> len(view_0d)
1
>>> len(view_0d.tolist())
TypeError: object of type 'int' has no len()

On Python 3.8, with my patch, this gives:

>>> len(view_0d)
TypeError: 0-dim memory has no length
>>> len(view_0d.tolist())
TypeError: object of type 'int' has no len()

As I read it, only with my patch is this sentence satisfied by the implementation.

[serhiy-storchaka]

But iterating it raises different type of exception, wish obscure message.

>>> import ctypes
>>> mem_0d = memoryview(ctypes.c_uint8(42))
>>> list(mem_0d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format <B

Should not '<B' and be equivalent to '>B' and just 'B'? Would not adding support of formats with specified endianess make 0-dimensional memoryview objects iterable?

[skrah]

memoryview can store anything, but only access and display
the native format. The reason is that conversions in the manner
of the struct module are very expensive.

You can construct all sorts of formats that memoryview cannot
handle:

>>> x = array(1, "c")
>>> x
array(b'1', dtype='|S1')
>>> y = memoryview(x)
>>> 
>>> list(y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format 1s

I don't think that the message is obscure, since all these
facts are documented.

[serhiy-storchaka]

It is surprising the endianess of byte matters.

[skrah]

list() has to *interpret* the endianness to display it. The endianness
of the *native* format does not matter. The endianness of an explicit
format matters, since list has to be able to convert non-native formats.

I suggest to look at XND, which is a memoryview on steroids and
has a lot of those features.

[mdickinson]

Closed by #18463