detect_encoding should fail with SyntaxError on invalid encoding

[florentx]

BPO	14990
Nosy	@vstinner, @ezio-melotti, @florentx
PRs	bpo-33338: [lib2to3] Synchronize token.py and tokenize.py with stdlib #6572
Files	issue14990_detect_encoding.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-07-07.10:29:50.995>
created_at = <Date 2012-06-03.10:29:02.107>
labels = ['type-bug', 'library', 'expert-unicode']
title = 'detect_encoding should fail with SyntaxError on invalid encoding'
updated_at = <Date 2018-04-23.01:09:05.686>
user = 'https://github.com/florentx'

bugs.python.org fields:

activity = <Date 2018-04-23.01:09:05.686>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2012-07-07.10:29:50.995>
closer = 'flox'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2012-06-03.10:29:02.107>
creator = 'flox'
dependencies = []
files = ['25805']
hgrepos = []
issue_num = 14990
keywords = ['patch']
message_count = 7.0
messages = ['162205', '162206', '162303', '162428', '162429', '164811', '164812']
nosy_count = 4.0
nosy_names = ['vstinner', 'ezio.melotti', 'flox', 'python-dev']
pr_nums = ['6572']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14990'
versions = ['Python 3.2', 'Python 3.3']

[florentx]

I've hit this issue while playing with tokenize for the PEP-8.py module.

The tokenize detect_encoding() should report SyntaxError when the encoding is improperly declared.

However it raises a LookupError in some cases.

$ ./python -m tokenize Lib/test/bad_coding2.py 
unexpected error: unknown encoding: utf8-sig
Traceback (most recent call last):
  File "./Lib/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "./Lib/runpy.py", line 75, in _run_code
    exec(code, run_globals)
  File "./Lib/tokenize.py", line 686, in <module>
    main()
  File "./Lib/tokenize.py", line 656, in main
    tokens = list(tokenize(f.readline))
  File "./Lib/tokenize.py", line 489, in _tokenize
    line = line.decode(encoding)
LookupError: unknown encoding: utf8-sig

[florentx]

This patch seems to fix the issue.

[vstinner]

The patch is correct according to the PEP-263:

If a source file uses both the UTF-8 BOM mark signature and a
magic encoding comment, the only allowed encoding for the comment
is 'utf-8'.  Any other encoding will cause an error.

The fix should also be applied to 3.2.

(Note: Python 3.1 doesn't accept bugfixes anymore.)

[florentx]

It should raise a SyntaxError, if coding is 'utf8'.
I don't agree with the last patch proposed.

If the import report a SyntaxError, 'tokenize' should do the same.

$ ./python Lib/test/bad_coding2.py
  File "Lib/test/bad_coding2.py", line 1
SyntaxError: encoding problem: utf-8

and it complies strictly with PEP-263.

[vstinner]

Oops, I didn't want to attach my patch to the issue. Mine is wrong, whereas yours is the right fix :-)

[python-dev]

New changeset 5020afc0b7c9 by Florent Xicluna in branch '3.2':
Issue bpo-14990: tokenize: correctly fail with SyntaxError on invalid encoding declaration.
http://hg.python.org/cpython/rev/5020afc0b7c9

[florentx]

Thanks. Fixed in trunk too, changeset b4322ad1fec4