Unexpected Changes in list Iterator

[loechelt]

BPO	730296
Nosy	@gvanrossum, @doerwalter, @rhettinger
Files	list_iterator.tar: Tar file of test python script including output for Python 2.2 and Python 2.3

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2003-05-07.01:30:58.000>
created_at = <Date 2003-04-30.16:45:12.000>
labels = ['interpreter-core']
title = 'Unexpected Changes in list Iterator'
updated_at = <Date 2003-05-07.01:30:58.000>
user = 'https://bugs.python.org/loechelt'

bugs.python.org fields:

activity = <Date 2003-05-07.01:30:58.000>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2003-04-30.16:45:12.000>
creator = 'loechelt'
dependencies = []
files = ['867']
hgrepos = []
issue_num = 730296
keywords = []
message_count = 9.0
messages = ['15783', '15784', '15785', '15786', '15787', '15788', '15789', '15790', '15791']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'doerwalter', 'rhettinger', 'loechelt']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue730296'
versions = ['Python 2.3']

[loechelt]

I encountered an unexpected change in the behavior of
the iterator for the built-in list class. I created a
subclass of the list class in which a start attribute
was added. The intended behavior was to make the
list look shorter by "hiding" an arbitrary number of
starting elements. The start attribute was used to
control the number of hidden elements. In order to
make my shifted list class work in my application,
I had to override a number of special methods,
including the __iter__ method. In Python 2.2.1 and
Python 2.3a2, the default __iter__ method indexed over
the entire, unshifted list even with other special
methods changed to reflect the shifting. Therefore, I
had to override the __iter__ method as well. When
I tested my application on Python 2.3b1, I encountered
an error which I tracked down to my __iter__
method. I found that it had doubled the shift of the
starting value. Eventually, I traced this to a change
in the behavior of the default __iter__ method of the
built-in list class. I was able to recreate the
problem
with a simple test case.

I created a simple shifted list class with a public
start attribute, a __len__ method that subtracts start
from
the length of the list, and a __getitem__ method that
adds start to the index during element access.
Iteration over the list (using the default __iter__
inherited from list) returns all the elements in Python
2.2.1,
but only the elements after the start value in Python
2.3b1. This change in behavior was an unexpected
surprise for me!

[doerwalter]

Logged In: YES
user_id=89016

This is a consequence of the changes done for bug
http://www.python.org/sf/665835.

[loechelt]

Logged In: YES
user_id=142817

Thanks for pointing this prior bug report to me. I found
the discussions very insightful. Regarding the
changes made to __iter__, I have no problems from an
application point of view. I just need to know
the designed behavior and adjust my code accordingly. From
a philosophical point of view, I have mixed
feelings about making __iter__ depend upon an overridden
__getitem__. I can see the advantages of
simplicity in writing extension classes, but you also lose
access to the base __iter__ functionality,
although I cannot think of an example of where that would be
important off the top of my head.

[rhettinger]

Logged In: YES
user_id=80475

This was an intentional change. The idea was to make
sure that calls to iter() would respect an overriden
__getitem__() method.

We're taking another look at the decision and thinking
about whether it made sense with respect to least
surprise, utility, consistency, and performance.

[gvanrossum]

Logged In: YES
user_id=6380

The more I think about it, the more I think that
list.__iter__ over a list subclass instance should give you
the items of the underlying list, and not respect any
overrides of __getitem__. That means that you only have to
override __iter__ if your __getitem__ reorders items or
hides some, not if it simply renumbers them.

[rhettinger]

Logged In: YES
user_id=80475

I concur. Would you like me to revert the changes (2 lines
each to listobject.c and tupleobject.c)?

[gvanrossum]

Logged In: YES
user_id=6380

Yes, please. Thanks for hanging on to this until I saw the
error of my way. :-)

[loechelt]

Logged In: YES
user_id=142817

Thanks. I personally prefer the previous behavior of 2.2 to
the beta. Even though it is slightly more work to override
__iter__ (very minimal at that), there are fewer surprises.
I think of __getitem__ mapping to [] and not (for x in ...),
especially when there is an explicit __iter__ method.

[rhettinger]

Logged In: YES
user_id=80475

Reverted change.
Revised unittest accordingly.

See:
Objects/listobject.c 2.152
Objects/tupleobject.c 2.81
Lib/test/test_types 1.51

Gary, thanks for the bug report.