tokenization assuming ASCII whitespace; missing multiline case

[jimjjewett]

BPO	13832
Nosy	@benjaminp, @JimJJewett

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-01-19.22:49:12.783>
created_at = <Date 2012-01-19.20:52:36.819>
labels = ['interpreter-core']
title = 'tokenization assuming ASCII whitespace; missing multiline case'
updated_at = <Date 2012-01-20.16:00:40.763>
user = 'https://github.com/JimJJewett'

bugs.python.org fields:

activity = <Date 2012-01-20.16:00:40.763>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2012-01-19.22:49:12.783>
closer = 'benjamin.peterson'
components = ['Interpreter Core']
creation = <Date 2012-01-19.20:52:36.819>
creator = 'Jim.Jewett'
dependencies = []
files = []
hgrepos = []
issue_num = 13832
keywords = []
message_count = 5.0
messages = ['151652', '151658', '151659', '151693', '151694']
nosy_count = 3.0
nosy_names = ['benjamin.peterson', 'python-dev', 'Jim.Jewett']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue13832'
versions = ['Python 3.3']

[jimjjewett]

Parser/parsetok.c was recently changed (e.g. http://hg.python.org/cpython/rev/2bd7f40108b4 ) to raise an error if multiple statements were found in a single-statement compile call. It sensibly ignores trailing whitespace and comments. Unfortunately,

(1) It looks only at (c == ' ' || c == '\t' || c == '\n' || c == '\014') as opposed to using Py_UNICODE_ISSPACE(ch)
(2) It assumes that a "#" means the rest of the line is OK, instead of looking for additional linebreaks.

Not sure whether to mark this a bug or an enhancement, since it is already strictly better than the 3.2 behavior of never warning about extra text.

[python-dev]

New changeset 00c4efbf57c3 by Benjamin Peterson in branch 'default':
check after comments, too (bpo-13832)
http://hg.python.org/cpython/rev/00c4efbf57c3

[benjaminp]

The tokenizer doesn't consider unicode spaces, either.

[jimjjewett]

Ignoring non-ascii whitespace is defensible, and I agree that it should match the rest of the parser. Ignoring 2nd lines is still a problem, and supposedly part of what got fixed. Test case:

s="""x=5  # comment
x=6
"""
compile(s, "<testbadsingle>", 'single')

[benjaminp]

$ ./python 
Python 3.3.0a0 (default:50a4af2ca654+, Jan 20 2012, 10:59:48) 
[GCC 4.5.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> s="""x=5  # comment
... x=6
... """
>>> compile(s, "<blah>", "single")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<blah>", line 1
    x=5  # comment
                 ^
SyntaxError: multiple statements found while compiling a single statement