IDLE crashes on *Edit / Find in files ...* command

[fgracia]

BPO	14929
Nosy	@terryjreedy, @serwy
Superseder	bpo-36058: Improve file decoding before re.search
Files	prelim_14929.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2019-02-21.00:18:44.145>
created_at = <Date 2012-05-27.20:47:21.016>
labels = ['expert-IDLE', 'type-bug']
title = 'IDLE crashes on *Edit / Find in files ...* command'
updated_at = <Date 2019-02-21.00:18:44.144>
user = 'https://bugs.python.org/fgracia'

bugs.python.org fields:

activity = <Date 2019-02-21.00:18:44.144>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2019-02-21.00:18:44.145>
closer = 'terry.reedy'
components = ['IDLE']
creation = <Date 2012-05-27.20:47:21.016>
creator = 'fgracia'
dependencies = []
files = ['25740']
hgrepos = []
issue_num = 14929
keywords = ['patch']
message_count = 10.0
messages = ['161727', '161738', '161741', '161753', '161754', '161755', '162208', '222783', '228247', '336159']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'roger.serwy', 'python-dev', 'fgracia']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '36058'
type = 'behavior'
url = 'https://bugs.python.org/issue14929'
versions = ['Python 3.4']

[fgracia]

There is little more that I can add to the title statement.

1. Start IDLE
2. Go to *Edit* menu option
3. Select *Find in files...* option
4. Put some word in the *Find* input box
5. Press *Search files* button

When the command is issued the disappearance of IDLE's window is so quick that almost nothing can be seen. There is however a hint that a new window pops up with some message, that I have been unable to read. With some input strings or if the search is happening in a directory with many *.py files a quickly moving list of messages shows in this window, but the final result is the same: IDLE disappears immediately.

[serwy]

When running IDLE from the terminal on Ubuntu, I get the following error:

Exception in Tkinter callback
Traceback (most recent call last):
  File "/home/serwy/python/cpython/Lib/tkinter/__init__.py", line 1442, in __call__
    return self.func(*args)
  File "./idlelib/GrepDialog.py", line 70, in default_command
    self.grep_it(prog, path)
  File "./idlelib/GrepDialog.py", line 90, in grep_it
    block = f.readlines(100000)
  File "/home/serwy/python/cpython/Lib/codecs.py", line 300, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 120: invalid continuation byte

The reason why IDLE closes suddenly on Windows is described in bpo-13582.

[serwy]

The GrepDialog opens a file using plain "open", without specifying the encoding or how to handle errors. The docs for "open" says that "the default encoding is platform dependent (whatever locale.getpreferredencoding() returns)..." This can be problematic, as files can have different encodings and GrepDialog at present has no way to detect file encodings.

Attached is a preliminary patch to replace code points if the default decoder encounters errors.

[python-dev]

New changeset 12454f78967b by Terry Jan Reedy in branch '3.2':
bpo-14929: Stop Idle 3.x from closing on Unicode decode errors when grepping.
http://hg.python.org/cpython/rev/12454f78967b

New changeset 058c3099e82d by Terry Jan Reedy in branch 'default':
Merge 3.2 bpo-14929
http://hg.python.org/cpython/rev/058c3099e82d

[terryjreedy]

Interesting -- and nasty. I have used this successfully with both 3.2 and now 3.3 on Win7 to search idlelib/*.py files and on a directory of my own files, all written by Idle and been quite pleased with the speed.

I just tried searching /Lib/*.py for 'itertools' and quickly got 10 results (up to decimal.py) before Idle disappeared. OK, import idlelib/idle in console, repeat, and I get the same traceback up to

File "C:\Programs\Python33\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1877: character maps to <undefined>

This is not a problem on 2.7.

Patch fixes problem, so I committed it so it will be in 3.3.0a4 (in a few days).

The default extension is .py. The default encoding for .py files is utf-8. I think that is the default for what Idle writes. So I think this should be the default encoding (explicitly given) at least for .py files.

[terryjreedy]

Hit send too soon;-). With that much done, we can think about a more complete fix. See last paragraph above. Also, perhaps dialog box could have encodings field. People should be able to grep python code for any legal identifier, and this means proper decoding according to the encoding they actually use.

Francisco, thanks for reporting this, and Roger, thanks for quick fix.

[fgracia]

While your are at it, here is another suggestion: what the *Find in files ...* dialog needs most urgently in my opinion is a field for specifying clearly the directory from which the user wants to launch the search.

Also in my modest opinion, having an input field for encondings would be good, although detecting the encodings of the handled files is something that any self respecting software should reliably do by itself.

[terryjreedy]

The 'In files:' field specifies the search directory. I opened bpo-21960 about it being more informative when using Find in Files from the Shell window.

Perfect encoding detection is a fantasy; decent encoding detection by heuristics is slow and not something to do when searchings 100s, 1000s, or more files. What would be reasonable is to check for an encoding cookie. Idle already does this when opening a .py file in the editor. That code should be encapsulated if it is not and reused.

An encoding field for non-python files would be possible, but is a lesser priority to me.

[terryjreedy]

As the 'crash' seems to have been solved, this should probably be closed and a new issue about encodings opened. Also, since the re module used to search lines within files can work with both bytes and strings, there might be an option to search undecoded bytes.

[terryjreedy]

Opened bpo-36058 for followup.