IDLE's deficiency in the completion of file names (Python 32, Windows XP)

[fgracia]

BPO	14937
Nosy	@loewis, @terryjreedy, @serwy
Files	issue14937.patch cpython-14937.patch: Alternative patch issue14937_logic_fix.patch: Fix while loop logic bug

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-09-09.22:18:34.264>
created_at = <Date 2012-05-28.17:17:24.670>
labels = ['expert-IDLE', 'type-bug']
title = "IDLE's deficiency in the completion of file names (Python 32, Windows XP)"
updated_at = <Date 2013-09-09.22:18:34.263>
user = 'https://bugs.python.org/fgracia'

bugs.python.org fields:

activity = <Date 2013-09-09.22:18:34.263>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2013-09-09.22:18:34.264>
closer = 'terry.reedy'
components = ['IDLE']
creation = <Date 2012-05-28.17:17:24.670>
creator = 'fgracia'
dependencies = []
files = ['25751', '25802', '25944']
hgrepos = []
issue_num = 14937
keywords = ['patch']
message_count = 21.0
messages = ['161795', '161796', '161808', '161812', '162167', '162168', '162169', '162171', '162172', '162175', '162196', '162203', '162204', '162692', '162789', '162804', '172699', '172721', '196517', '196599', '197409']
nosy_count = 6.0
nosy_names = ['loewis', 'terry.reedy', 'roger.serwy', 'westley.martinez', 'python-dev', 'fgracia']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14937'
versions = ['Python 3.3', 'Python 3.4']

[fgracia]

I find specially nice the completion feature for filenames of IDLE when they include long paths, something that is more mand more frequent with big disks and infinite amounts of files.

But there is a small problem with it. If the name of any of the directories that intervene in the path to the file has any special character in it (meaning by that I suppose any non ASCII character), the completion mechanism ceases to work.

For instance, if the path were:

d:/Biblioteca/Técnica/informática/Python ...

the cesation of the help would occur after the incorporation of *Técnica*, because its *e* is accented.

There is plenty of cheap software which shows this minor deficiency, but I consider that IDLE for Python 3 should not be in that group.

[loewis]

Would you like to contribute a patch?

[fgracia]

I would be delighted, but unfortunately I am a very poor programmer and do not have the slightest idea of how all this works.

[serwy]

The "open_completions" method in AutoComplete.py is where the bug exists. When mode == COMPLETE_FILES, the code searches for characters within the ASCII set, plus a few others contained in FILENAME_CHARS.

Attached is a patch to also include characters beyond the ASCII set during the backward search. If anyone knows of a better technique, please comment.

Fransisco, thanks for reporting this issue. If you find more, please let us know.

[westleymartinez]

I think a better technique would be to expand FILENAME_CHARS to include more characters.

[westleymartinez]

Also, shouldn't the space character ' ' be included?

[westleymartinez]

Ahh okay, sorry for the triple post, I have an idea. On UNIX, the function should accept any character except: \0 /, and on Windows should accept any character except: \0 \ / : * ? " < > | On classic Macintosh, : is invalid. However, I do not know what characters are valid on other systems. (I believe Mac OS X is the same as UNIX, but am not sure.)

Included is a patch with FILENAME_CHARS left in in case anything still needs it.

[serwy]

I agree that chr(32) should be included in FILENAME_CHARS.

The algorithm for backward searching checks that each character is contained in FILENAME_CHARS. I'm concerned about running time, as expanding FILENAME_CHARS to include all valid Unicode would be a large set to search. Do you know a better algorithm for determining when to perform a file-system completion when in a string?

Also, are there any Unicode code points above 127 that are not valid in a filename? I appreciate that different file systems can have different constraints but I don't know all these subtle points.

[serwy]

Westley, I was responding to msg162168 and didn't see msg162169 yet.

PEP-11 mentions MacOS 9 support was removed in 2.4. Is ":" still invalid in OSX?

I'll need to think about the approach of using an "INVALID_CHARS" list. It looks like it might be a better way to do this. However, here are some issues with the patch.

I received this error:

AttributeError: 'str' object has no attribute 'relpace'

I fixed this minor mistype, but noticed that file auto-completion no longer works for the following:

>> "/

Pressing tab after the forward slash should show a drop-down with the root filesystem. Changing INVALID_CHARS to be:

   if os.name == 'posix':
       INVALID_CHARS = '\0'

allows the root filesystem to show.

[fgracia]

Is there any necessity at all for the IDLE to test the validity of the filenames?

I mean: the file specification is provided by the underlying operating system, so by definition it has to be taken as valid. Testing for its validity is superfluous and, in my opinion, a fault in the general conception of the procedure. IDLE's only role should be to present the data it receives in the best possible form and silently integrate the election made by the user.

Excuse me if I speak nonsense, but I got this idea from studying *Autocomplete.py* and its dependencies. It seems to me that the difficulty resides in submitting natural language strings to the judgement of *PyParse* and *Hyperparser*, which are not competent for it because its purpose is to check the validity of Python code and Python *identifiers*. This confussion is what in my opinion causes this bug (and probably others related to the treatment of natural language text files by the IDLE, as it could also happen with bpo-14929).

[westleymartinez]

You're right. The code shouldn't *have* to check if the name is valid. It should just accept that the name is already valid. This would simplify things.

Here's the problem: the code needs to find the index of where the string with the filename starts. The way the code does it now by checking for a quote in a rather obfuscated way (while i and curline[i-1] in FILENAME_CHARS + SEPS:). So, changing that line to say curline[i-1] != "'" or curline[i-1] != '"' would work (in theory) but I'm really hoping there's a better way.

[python-dev]

New changeset 41e85ac2ccef by Martin v. Löwis in branch '3.2':
Issue bpo-14937: Perform auto-completion of filenames in strings even for non-ASCII filenames.
http://hg.python.org/cpython/rev/41e85ac2ccef

New changeset 9aa8af0761ef by Martin v. Löwis in branch 'default':
Merge 3.2: issue bpo-14937.
http://hg.python.org/cpython/rev/9aa8af0761ef

[loewis]

I have now fixed it by looking for the beginning of the string, and not checking for file name characters at all. There was a related issue that the auto-complete window would disappear if you type a non-ascii character; I have fixed that as well.