clarify that linecache only works on files that can be decoded successfully

[vstinner]

BPO	11726
Nosy	@terryjreedy, @vstinner, @merwok, @bitdancer, @takluyver, @ztane
Files	linecache-encoding-doc.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-03-20.15:20:13.616>
created_at = <Date 2011-03-31.10:07:51.232>
labels = ['type-bug', 'library', 'docs']
title = 'clarify that linecache only works on files that can be decoded successfully'
updated_at = <Date 2017-02-09.12:06:56.168>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-02-09.12:06:56.168>
actor = 'ztane'
assignee = 'docs@python'
closed = True
closed_date = <Date 2015-03-20.15:20:13.616>
closer = 'r.david.murray'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2011-03-31.10:07:51.232>
creator = 'vstinner'
dependencies = []
files = ['38424']
hgrepos = []
issue_num = 11726
keywords = ['patch']
message_count = 14.0
messages = ['132641', '132782', '178884', '237685', '237686', '237786', '238242', '238427', '238428', '238429', '238689', '238690', '238692', '287408']
nosy_count = 8.0
nosy_names = ['terry.reedy', 'vstinner', 'eric.araujo', 'r.david.murray', 'docs@python', 'python-dev', 'takluyver', 'ztane']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11726'
versions = ['Python 3.4', 'Python 3.5']

[vstinner]

linecache document doesn't tell that the module reads the #coding:xxx cookie to get the encoding of the Python file. linecache reads this cookie since 41665 (May 09 2007).

"The linecache module allows one to get any line from any file, ..."

=> "any file"!

And the example uses /etc/passwd which is not a Python file.

Not only it reads the #coding:xxx cookie, but updatecache() tries also a PEP-302 loader to read the file.

linecache should be marked as very specific to Python scripts, or it should be patched to become more generic (don't read the cookie / use loader by default).

Note: the locale encoding may change between to calls to the linecache module.

[terryjreedy]

The help(linecache) Description is more specific as to the intention (based on traceback usage):

"This is intended to read lines from modules imported -- hence if a filename is not found, it will look down the module search path for a file by that name."

My experiments show that this is too specific. It *can* read any file that it can find and decode as utf-8 (default, or you say, locale encoding or coding in cookie line).

Find = absolute path
>>> linecache.getline('c:/programs/pydev/py32/LICENSE', 1)
'A. HISTORY OF THE SOFTWARE\n'

or relative path on sys.path
>>> linecache.getline('idlelib/ChangeLog', 1)
'Please refer to the IDLEfork and IDLE CVS repositories for\n'
>>> linecache.getline('idlelib/extend.txt', 1)
'Writing an IDLE extension\n'

Decode fails on byte illegal for utf-8:
>>> linecache.getline('idlelib/CREDITS.txt', 1)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf6 in position 1566: invalid start byte

(It reads and decodes entire file even though only line 1 was requested. It choked on Löwis. I believe Py3 distributed text files should be utf-8 instead of latin-1.)

If I got rules right, doc should say "Filename must be an absolute path or relative path that can be found on sys.path." and "File must be utf-8 encoded or locale encoded or a Python file with a coding cookie."

(If you tried /etc/passwd, how did it fail?)

[vstinner]

Well, my initial message doesn't convince me anymore today (especially after reading Terry's message), so I prefer to close the issue as rejected. I don't think that it's really a problem :-)

[takluyver]

Someone on reddit ran into this, expecting that linecache can be used for an arbitrary text file:
http://www.reddit.com/r/Python/comments/2yetxc/utf8_encoding_problems/

I was quite surprised that the docs say "allows one to get any line from any file." I've always understood that linecache is specifically for Python files, and the use of tokenize.open() means that it will only work for files that are UTF-8 or have the #coding: magic comment in the first two lines.

I think the docs should at least mention this; I'm happy to work on a patch for it at some point if people agree.

[bitdancer]

Sure, clarifying the docs seems sensible. "Any file" is slightly different from the reality.

[takluyver]

First attempt at describing this attached.

[takluyver]

Anything else I should be doing here?

[python-dev]

New changeset 51341af466e3 by Victor Stinner in branch '3.4':
Issue bpo-11726: clarify linecache doc: linecache is written to cache Python
https://hg.python.org/cpython/rev/51341af466e3