add to "looping techniques" tutorial a note about modifying sequence

[Ian]

BPO	16225
Nosy	@birkenfeld, @rhettinger, @jcea, @cjerdonek, @serhiy-storchaka
Files	issue-16225-1-default.patch issue-16225-2-default.patch issue-16225-3-default.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-10-16.03:04:55.964>
created_at = <Date 2012-10-14.07:27:28.216>
labels = ['type-feature', 'docs']
title = 'add to "looping techniques" tutorial a note about modifying sequence'
updated_at = <Date 2012-10-16.03:04:55.964>
user = 'https://bugs.python.org/Ian'

bugs.python.org fields:

activity = <Date 2012-10-16.03:04:55.964>
actor = 'chris.jerdonek'
assignee = 'docs@python'
closed = True
closed_date = <Date 2012-10-16.03:04:55.964>
closer = 'chris.jerdonek'
components = ['Documentation']
creation = <Date 2012-10-14.07:27:28.216>
creator = 'Ian'
dependencies = []
files = ['27562', '27573', '27575']
hgrepos = []
issue_num = 16225
keywords = ['patch']
message_count = 18.0
messages = ['172855', '172856', '172861', '172875', '172897', '172899', '172906', '172926', '172930', '172932', '172933', '172934', '172935', '173007', '173008', '173011', '173012', '173013']
nosy_count = 8.0
nosy_names = ['georg.brandl', 'rhettinger', 'jcea', 'chris.jerdonek', 'docs@python', 'python-dev', 'serhiy.storchaka', 'Ian']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue16225'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']

[Ian]

I'm new to Python and I've hit what appears to me to be a bug, but may be a "feature", so a tutorial bug.
I tried to loop through the items in a list, test each and remove those which fail the test.

Simplifying to illustrate:
>>> print test
[1, 2, 3, 4, 5]
>>> for item in test:
...     print item
...     test.remove(item)
... 
1
3
5
>>> print test
[2, 4]

Whereas I would expect to see all items looped through and non left in the list.
I have worked with languages where you are explicitly warned that you must not mess with the loop variable, or where the behaviour you will get is explained in detail, so you can use it. Not having anything flagged up in eg 5.6. Looping Techniques on
http://docs.python.org/tutorial/datastructures.html
I assumed that the behaviour would be safe and intuative.
Yours
Ian

[cjerdonek]

I have worked with languages where you are explicitly warned that you must not mess with the loop variable

There is a warning in this part of the tutorial:

"It is not safe to modify the sequence being iterated over in the loop..."

(from http://docs.python.org/dev/tutorial/controlflow.html#for-statements )

But it may be good to add a note to the section you reference as well.

[cjerdonek]

Attached is a simple way of addressing this (essentially copying the verbiage and example from the other page). If we want, we could make the sample code different so that the reader doesn't see the same thing twice.

[serhiy-storchaka]

It is safe to modify a sequence during iteration if it's size not increased.

>>> words = ['cat', 'window', 'defenestrate']
>>> for i, w in enumerate(words):
...     if len(w) > 6:
...         words[i] = w[:5] + '…'
... 
>>> words
['cat', 'window', 'defen…']

[cjerdonek]

It is safe to modify a sequence during iteration if it's size not increased.

What do you mean by "safe"? The example given by the original commenter does not increase the size either. I believe "safe" is meant in the sense of avoiding possibly unexpected behavior. I can clarify the language if you feel it is ambiguous.

[birkenfeld]

Well, I guess Serhiy meant "neither increase nor decrease".

In the end, the exact behavior will never be clear if you don't state explicitly how it's implemented: a counter that starts at 0 and is increased on __next__ until it's equal to len(list) (as checked at every iteration step). But in general the advice should be: if you want to insert or remove elements during iteration, iterate over a copy.

[cjerdonek]

But in general the advice should be: if you want to insert or remove elements during iteration, iterate over a copy.

I would expand this to cover changing the list in any way. I think the point being made is that iteration doesn't implicitly make a copy. There are cases where modifying the list in place can also yield unexpected results. For example (naive list reversal):

>>> words = ['cat', 'window', 'defenestrate']
>>> for i, word in enumerate(words):
...     words[-i-1] = word
... 
>>> words
['cat', 'window', 'cat']

[cjerdonek]

Attaching revised patch.

[cjerdonek]

Reattaching. I duplicated a variable definition that was defined previously.

[serhiy-storchaka]

I mean it does not lead to crash, hang, etc. Even growing list during iteration can be safe you move forward faster than list grows or if you known where to stop.

Unexpected behavior for one people can be expected for others.

[cjerdonek]

I mean it does not lead to crash, hang, etc.

I agree. I removed the word "safe" in the patch I attached to reduce ambiguity. Regarding unexpected behavior, remember that the tutorial is for beginners/newcomers.

[serhiy-storchaka]

I agree. I removed the word "safe" in the patch I attached to reduce
ambiguity.

Yes, so much the better.

It will be nice somewhere in deep clarify for experts what happens with list
iterator if the list changed.

And iterating over modifyed (if you insert/remove keys, but not when you only
update values) dict or set really yield unexpected results (i.e. they expected
in them undefinability). This is really not recommended.

[cjerdonek]

It will be nice somewhere in deep clarify for experts what happens with list iterator if the list changed.

There is a note somewhat to this effect here:

http://docs.python.org/dev/reference/compound_stmts.html#the-for-statement

"Note There is a subtlety when the sequence is being modified by the loop...."

[rhettinger]

> It will be nice somewhere in deep clarify
> for experts what happens with list iterator if the list changed.

Resist the urge to over-specify. Unless the behavior is tested and known to be consistent across all implementations, I'm content with the current docs:

"Note There is a subtlety when the sequence is being modified by the loop...."

Chris's currently patch seems reasonable to me and I don't think anything further is a good idea.

With nearly any data structure in any language, most programmers learn to be cautious about and generally avoid looping over a structure while mutating it. Adding yet more documentation details won't make the issue go away.

I recommend posting the current patch and closing this issue to be done with it.

[cjerdonek]

Thanks, Raymond. I will be doing that later today.

[python-dev]

New changeset 1f1bf6a3abbc by Chris Jerdonek in branch '3.2':
Issue bpo-16225: Add additional note to tutorial about changing sequence while looping.
http://hg.python.org/cpython/rev/1f1bf6a3abbc

New changeset 8cb14494d33c by Chris Jerdonek in branch '3.3':
Issue bpo-16225: Merge from 3.2: Add additional note to tutorial about looping.
http://hg.python.org/cpython/rev/8cb14494d33c

New changeset e0a407d41af5 by Chris Jerdonek in branch 'default':
Issue bpo-16225: Merge from 3.3: Add additional note to tutorial about looping.
http://hg.python.org/cpython/rev/e0a407d41af5

[python-dev]

New changeset dc006b6212e7 by Chris Jerdonek in branch '2.7':
Issue bpo-16225: Backport from 3.2: Add additional note to tutorial about looping.
http://hg.python.org/cpython/rev/dc006b6212e7

[cjerdonek]

Committed. Thanks for reporting the suggestion, Ian.