Possible OverflowError in __len__ method undocumented (when called via len() function)

[RostyslavDzinko]

BPO	15718
Nosy	@birkenfeld, @terryjreedy, @vstinner, @benjaminp, @vadmium, @serhiy-storchaka
PRs	bpo-15718: Document the upper bound constrain on __len__ return value. #1256 [3.6] bpo-15718: Document the upper bound constrain on the __len__ re… #1259 [3.5] bpo-15718: Document the upper bound constrain on the __len__ re… #1260 [2.7] bpo-15718: Document the upper bound constrain on the __len__ re… #1261 bpo-12159: document sys.maxsize limit in len() function reference #17934

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-04-23.07:12:39.378>
created_at = <Date 2012-08-17.11:17:58.311>
labels = ['type-bug', '3.7', 'docs']
title = 'Possible OverflowError in __len__ method undocumented (when called via len() function)'
updated_at = <Date 2020-01-10.12:13:55.063>
user = 'https://bugs.python.org/RostyslavDzinko'

bugs.python.org fields:

activity = <Date 2020-01-10.12:13:55.063>
actor = 'Zac Hatfield-Dodds'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-04-23.07:12:39.378>
closer = 'serhiy.storchaka'
components = ['Documentation']
creation = <Date 2012-08-17.11:17:58.311>
creator = 'Rostyslav.Dzinko'
dependencies = []
files = []
hgrepos = []
issue_num = 15718
keywords = []
message_count = 10.0
messages = ['168439', '168473', '257312', '257318', '292123', '292125', '292151', '292152', '292153', '292154']
nosy_count = 10.0
nosy_names = ['georg.brandl', 'terry.reedy', 'vstinner', 'benjamin.peterson', 'Arfrever', 'docs@python', 'martin.panter', 'serhiy.storchaka', 'Rostyslav.Dzinko', 'Winterflower']
pr_nums = ['1256', '1259', '1260', '1261', '17934']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue15718'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[RostyslavDzinko]

I've encountered that OverflowError which can happen in __len__ method is still undocumented, though one issue on this problem: http://bugs.python.org/issue12159 ended up with "need to be documented" comment.

Link to documentation: http://docs.python.org/reference/datamodel.html#object.\_\_len__

I think it must be clarified that __len__ return value is constrained to upper boundary (Py_ssize_t c type) when you plan to call it via len() built-in function.

[terryjreedy]

In bpo-12159, Victor correctly labelled this an implementation (IE, CPython) detail (limitation). I don't believe any implementation has to limit the range of len(). So the question is whether we should add a CPython implementation limit note, including the possibility of OverflowError, and if so, to both len() and __len__() entries. I am not sure of the current doc policy.

The second sentence of the len entry is out of date. " The argument may be a sequence (string, tuple or list) or a mapping (dictionary)." Sets and any collections with a size (len method) can also be arguments. I am not sure how to revise that either.

[Winterflower]

The deficiency noticed by Terry has been at least partially corrected in the len() docs
https://docs.python.org/2/library/functions.html#len
Return the length (the number of items) of an object. The argument may be a sequence (such as a string, bytes, tuple, list, or range) or a collection (such as a dictionary, set, or frozen set).

It doesn't mention the fact that the collection has to implement __len__, but I suppose that might be obvious to experienced Python users (isn't obvious to a beginner like me, though). So perhaps this might be a welcome clarification.

The issue regarding OverflowError raised by Rostyslav still remains unresolved.

[vadmium]

bpo-10289 proposes to link from len() to __len__() in the documentation. I think the limitation only needs to be documented for __len__(); there are other ways to invoke it as well.

[terryjreedy]

Has or will builtin classes be modified to follow advice to include a separate __bool__? In 3.6.1,

>>> r = range(19**100)
>>> bool(r)
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    bool(r)
OverflowError: Python int too large to convert to C ssize_t

[serhiy-storchaka]

Yes, and the range object already is fixed by bpo-28876.

[serhiy-storchaka]

New changeset 85157cd by Serhiy Storchaka in branch 'master':
bpo-15718: Document the upper bound constrain on the __len__ return value. (bpo-1256)
85157cd

[serhiy-storchaka]

New changeset f2ed285 by Serhiy Storchaka in branch '3.6':
[3.6] bpo-15718: Document the upper bound constrain on the __len__ return value. (GH-1256) (bpo-1259)
f2ed285

[serhiy-storchaka]

New changeset c358536 by Serhiy Storchaka in branch '3.5':
[3.5] bpo-15718: Document the upper bound constrain on the __len__ return value. (GH-1256) (bpo-1260)
c358536

[serhiy-storchaka]

New changeset bfc7dff by Serhiy Storchaka in branch '2.7':
[2.7] bpo-15718: Document the upper bound constrain on the __len__ return value. (GH-1256). (bpo-1261)
bfc7dff