SyntaxError: encoding problem: iso-8859-1 on Windows #65043
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
assignee = None closed_at = <Date 2019-04-01.12:03:46.170> created_at = <Date 2014-03-03.13:54:42.008> labels = ['interpreter-core', '3.8', 'type-bug', '3.7', 'OS-windows'] title = 'SyntaxError: encoding problem: iso-8859-1 on Windows' updated_at = <Date 2019-04-01.12:03:46.165> user = 'https://bugs.python.org/miwa'
activity = <Date 2019-04-01.12:03:46.165> actor = 'methane' assignee = 'none' closed = True closed_date = <Date 2019-04-01.12:03:46.170> closer = 'methane' components = ['Interpreter Core', 'Windows'] creation = <Date 2014-03-03.13:54:42.008> creator = 'miwa' dependencies =  files = ['34276', '47065'] hgrepos =  issue_num = 20844 keywords = ['patch'] message_count = 15.0 messages = ['212637', '212638', '213012', '213014', '213189', '213196', '214330', '221089', '221134', '224351', '233129', '233130', '299935', '339288', '339290'] nosy_count = 10.0 nosy_names = ['vstinner', 'nedbat', 'tim.golden', 'benjamin.peterson', 'miwa', 'methane', 'schlamar', 'zach.ware', 'steven.winfield', 'eryksun'] pr_nums = ['12616', '12647'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue20844' versions = ['Python 3.7', 'Python 3.8']
The text was updated successfully, but these errors were encountered:
Microsoft Windows [Version 6.1.7601]
C:\bug>python Python 3.3.5rc2 (v3.3.5rc2:ca5635efe090, Mar 2 2014, 18:18:29) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> exit()
I can reproduce this one. There are a few conditions which needs to be met:
This fix for bpo-20731 doesn't address this bug completely because it's possible for ftell to return -1 without an actual error, as test2.py demonstrates.
In text mode, CRLF is translated to LF by the CRT's _read function (Win32 ReadFile). So the buffer that's used by FILE streams is already translated. To get the stream position, ftell first calls _lseek (Win32 SetFilePointer) to get the file pointer. Then it adjusts the file pointer for the unwritten/unread bytes in the buffer. The problem for reading is how to tell whether or not LF in the buffer was translated from CRLF? The chosen 'solution' is to just assume CRLF.
The example file test2.py is 33 bytes. At the time fp_setreadl calls ftell(tok->fp), the file pointer is 33, and Py_UniversalNewlineFgets has read the stream up to '#coding:latin-1\n'. That leaves 17 newline characters buffered. As stated above, ftell assumes CRLF, so it calculates the stream position as 33 - (17 * 2) == -1. That happens to be the value returned for an error, but who's checking? In this case, errno is 0 instead of the documented errno constants EBADF or EINVAL.
Here's an example in 2.7.7, since it uses FILE streams:
>>> f = open('test2.py') >>> f.read(16) '#coding:latin-1\n' >>> f.tell() Traceback (most recent call last): File "<stdin>", line 1, in <module> IOError: [Errno 0] Error
Can the file be opened in binary mode in Modules/main.c? Currently it's using
I've just been bitten by this on 3.6.2, Windows Server 2008 R2, when running the setup.py script for QuantLib-SWIG:
It seems there is different behaviour depending on whether:
Some of that has been mentioned previously, but I think the 4096-byte limit might be new, which is why I'm posting.
I've attached a script I used to come up with the results below. It contains:
The file's length is exactly 4096 bytes.
Running this, or slightly modified versions of this, with a 3.6.2 interpreter gave the following results:
I had no issues with python 2.7.13.