IDLE "AttributeError: 'NoneType' object has no attribute 'index'" from Delegator.py

[rhettinger]

BPO	17613
Nosy	@birkenfeld, @rhettinger, @terryjreedy, @orsenthil, @larryhastings, @benjaminp, @ned-deily, @ezio-melotti, @serwy, @rovitotv
Files	stop_colorizer.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serwy'
closed_at = <Date 2013-04-07.17:43:34.257>
created_at = <Date 2013-04-02.00:55:15.294>
labels = ['expert-IDLE', 'type-bug']
title = 'IDLE "AttributeError: \'NoneType\' object has no attribute \'index\'" from Delegator.py'
updated_at = <Date 2013-04-07.17:43:34.255>
user = 'https://github.com/rhettinger'

bugs.python.org fields:

activity = <Date 2013-04-07.17:43:34.255>
actor = 'roger.serwy'
assignee = 'roger.serwy'
closed = True
closed_date = <Date 2013-04-07.17:43:34.257>
closer = 'roger.serwy'
components = ['IDLE']
creation = <Date 2013-04-02.00:55:15.294>
creator = 'rhettinger'
dependencies = []
files = ['29650']
hgrepos = []
issue_num = 17613
keywords = ['patch']
message_count = 18.0
messages = ['185784', '185787', '185793', '185796', '185797', '185800', '185801', '185808', '185809', '185810', '185812', '185814', '185816', '185818', '185819', '185829', '186232', '186233']
nosy_count = 11.0
nosy_names = ['georg.brandl', 'rhettinger', 'terry.reedy', 'orsenthil', 'larry', 'benjamin.peterson', 'ned.deily', 'ezio.melotti', 'roger.serwy', 'Todd.Rovito', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17613'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

[rhettinger]

Running ILDE with 2.7.4 RC 1 on Mac OS X 10.8 and with ActiveTCL 8.5.13 using "python -m idlelib.idle", I object the following traceback messages. I'm unsure what events are triggering it:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk/Tkinter.py", line 1470, in __call__
    return self.func(*args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk/Tkinter.py", line 531, in callit
    func(*args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/idlelib/ColorDelegator.py", line 153, in recolorize
    self.recolorize_main()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/idlelib/ColorDelegator.py", line 186, in recolorize_main
    next = self.index(mark + "+%d lines linestart" %
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/idlelib/Delegator.py", line 10, in __getattr__
    attr = getattr(self.delegate, name) # May raise AttributeError
AttributeError: 'NoneType' object has no attribute 'index'

[rovitotv]

Raymond,
Is then when you simply startup IDLE or are you opening a Python file to edit? The only reason why I ask about editing a file is because ColorDelegator is used to by the EditorWindow to perform the syntax hi-lighting (I think). I have a few Macs so I can try and duplicate. In addition I added a few people to the nosy list that might know better than I do what the problem is. Thanks for the report.

[ned-deily]

Setting as 2.7.4 release blocker pending further analysis. Roger, any ideas? Would having the contents of Raymond's .idlerc files be of help? Raymond, does IDLE terminate when the exception occurs or does it keep running?

[terryjreedy]

Do you get the same traceback with "python -m idlelib"?

[ned-deily]

"python -m idlelib" isn't implemented for 2.7.x, is it?

[serwy]

It's a "race condition" due to ColorDelegator calling .update() in its recolorize method. The call to .update() flushes pending events, including callbacks into Python code that can change out the ColorDelegator in the Percolator chain.

Here's a way to trigger this bug reliably. First we need an unreasonably large file that will make the syntax highlighting routine take a while. This code will generate such a file:

fid = open('/tmp/bigfile.py', 'w')
L = '\n'.join(map(lambda x: "#%i\n'%i'\n" % (x,x), range(10000)))
fid.writelines(L)
fid.close()

These next two steps need to be performed quickly to trigger the "race condition."

$ python -m idlelib.idle -e /tmp/bigfile.py
As soon as it loads, press Ctrl+Shift+S to bring up the Save-As dialog. 

You can take your time now. Overwrite the file. This triggers the filename_change_hook to call ResetColorizer, which removes the current ColorDelegator from the Percolator chain and inserts a new one. Wait a moment for the traceback to arrive.

The attached patch fixes the issue. When the ColorDelegator is removed by Percolator's removefilter, the delegate is set to None. The patch detects that condition and then sets flags to stop the colorizer from continuing.

In testing for this bug, I encountered another one. Pressing Alt-F4 quickly after the window loading bigfile appears triggers another traceback. I'll open a separate issue for that.

[terryjreedy]

Yes, idlelib.__main__.py was only added to 3.3.

[serwy]

Removing Geoge and Larry from nosy. I'm not sure how they got added. I didn't click on anything in the Nost List box.

Ned, did you mean to add Benjamin to the nosy list?

[serwy]

Is there an automatic nosy-list adder routine in the bug-tracker that I should know about? Georg and Larry were re-added to nosy even after I removed them. (I'm sorry for the spam.)

[rhettinger]

Raymond, does IDLE terminate when the exception occurs
or does it keep running?

It keeps running.

[larryhastings]

When an issue is marked as "release blocker", the release managers of all affected versions are automatically added to the issue. I'm the release manager of 3.4, Georg is for 3.3.

[ezio-melotti]

Is there an automatic nosy-list adder routine in the bug-tracker that
I should know about?

Release managers are added automatically when an issue is set as release blocker and one of the releases they manage is selected in the "Versions" field.

[orsenthil]

I dont have ActiveState TCL, but I could use IDLE with the default libs with my Macbook Pro (OS X10.7), followed the steps given to reproduce the error and unfortunately could not.

[ned-deily]

I'm able to reliably reproduce the exception using your test procedure, Roger, using either current head of 3.3 or 2.7, and am not able to reproduce it with stop_colorizer.patch applied. LGTM. So it seems to me the next questions are:

1. Raymond, does this seem like a plausible scenario for the failures you've seen? And if you apply the patch, do you see any further failures?

2. Roger, is this issue new to 2.7.4, 3.2.4, or 3.3.1? How likely do you think it will be seen by users? And what is your recommendation to the release managers about including the patch in any or all of the pending maintenance releases, understanding that the options are to cherry-pick into the final and hope for the best, cherry-pick it and do an RC2, or defer the fix to the next maintenance releases?

[serwy]

On 04/02/2013 12:57 AM, Ned Deily wrote:

2. Roger, is this issue new to 2.7.4, 3.2.4, or 3.3.1?
   No, this bug has been baked into IDLE since the beginning. See line 215
   http://hg.python.org/cpython/file/30713d5288df/Lib/idlelib/ColorDelegator.py
   How likely do you think it will be seen by users?
   For Windows users using pythonw.exe, it would cause IDLE to immediately
   terminate due to bpo-13582. On Linux and Mac, the errors would write to
   stderr, though not likely be visible to the end user. On Linux and Mac
   the error is not "harmful" since IDLE still remains perfectly usable.
   The error does not leave IDLE in an inconsistent state.
   And what is your recommendation to the release managers about including the patch in any or all of the pending maintenance releases, understanding that the options are to cherry-pick into the final and hope for the best, cherry-pick it and do an RC2, or defer the fix to the next maintenance releases?
   I'm ok with cherry-picking it into the final, but in the interest of
   simplicity, I'd recommend deferring it to the next maintenance release.
   There are other bugs, like bpo-8900 that are easier to trigger than this one.

[benjaminp]

Thank you for the analysis.

[python-dev]

New changeset ae05d841bea1 by Roger Serwy in branch '2.7':
bpo-17613: Prevent traceback when removing syntax colorizer in IDLE.
http://hg.python.org/cpython/rev/ae05d841bea1

New changeset 8b793a946acb by Roger Serwy in branch '3.3':
bpo-17613: Prevent traceback when removing syntax colorizer in IDLE.
http://hg.python.org/cpython/rev/8b793a946acb

New changeset b45675223885 by Roger Serwy in branch 'default':
bpo-17613: merge with 3.3.
http://hg.python.org/cpython/rev/b45675223885

[serwy]

Thank you everyone for your patience. I am closing this issue as fixed.