__len__ called twice in the list() constructor

[kimadeline]

BPO	39829
Nosy	@brettcannon, @rhettinger, @terryjreedy, @methane, @ericsnowcurrently, @pablogsal, @sweeneyde, @godlygeek, @kimadeline, @thatbirdguythatuknownot
PRs	bpo-39829: Optimize __len__() being called twice in the list() constructor #31816

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pablogsal'
closed_at = <Date 2022-03-14.01:24:29.714>
created_at = <Date 2020-03-02.15:36:16.527>
labels = ['interpreter-core', '3.11', '3.9', '3.10', 'performance']
title = '__len__ called twice in the list() constructor'
updated_at = <Date 2022-03-14.01:24:38.330>
user = 'https://github.com/kimadeline'

bugs.python.org fields:

activity = <Date 2022-03-14.01:24:38.330>
actor = 'methane'
assignee = 'pablogsal'
closed = True
closed_date = <Date 2022-03-14.01:24:29.714>
closer = 'methane'
components = ['Interpreter Core']
creation = <Date 2020-03-02.15:36:16.527>
creator = 'kimiguel'
dependencies = []
files = []
hgrepos = []
issue_num = 39829
keywords = ['patch']
message_count = 13.0
messages = ['363186', '363188', '363588', '363745', '363746', '363747', '414724', '414727', '414840', '414889', '415026', '415111', '415112']
nosy_count = 10.0
nosy_names = ['brett.cannon', 'rhettinger', 'terry.reedy', 'methane', 'eric.snow', 'pablogsal', 'Dennis Sweeney', 'godlygeek', 'kimiguel', 'Crowthebird']
pr_nums = ['31816']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue39829'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[kimadeline]

(See bpo-33234)

Recently we added Python 3.8 to our CI test matrix, and we noticed a possible backward incompatibility with the list() constructor.

We found that __len__ is getting called twice, while before 3.8 it was only called once.

Here's an example:

class Foo:
 def __iter__(self):
  print("iter")
  return iter([3, 5, 42, 69])

 def __len__(self):
  print("len")
  return 4

Calling list(Foo()) using Python 3.7 prints:

iter
len

But calling list(Foo()) using Python 3.8 prints:

len
iter
len

It looks like this behaviour was introduced for bpo-33234 with PR #54055.

We realize that this was merged a while back, but at least we wanted to make the team aware of this change in behaviour.

[pablogsal]

Why should that be backwards incompatible? The number of times we can __len__ on the constructor is an implementation detail. The reason is called now twice is because there is an extra check for the preallocation logic, which is detached from the logic of the subsequent list_extend(self, iterable).

On the other hand, there may be a chance for optimization here, but on a very rough first plan, that may require coupling some logic (passing down the calculated length to list_extend() or some helper, which I am not very fond of.

[terryjreedy]

The only specification is that len(ob) calls ob.__len__ and that ob.__len__ should return an 'integer >= 0'. (Adding side effects goes beyond that spec.) I agree that a detectable internal in list is not a bug. Unless there is a realistic performance enhancement in caching the result of the first call, this issue should be closed.