compile_command exec not raising syntax error with new PGEN Parser

[Carreau]

BPO	40585
Nosy	@gvanrossum, @Carreau, @pablogsal
PRs	bpo-40585: Normalize errors messages in codeop when comparing them #20030

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-05-11.00:41:39.953>
created_at = <Date 2020-05-10.23:03:34.968>
labels = ['interpreter-core', '3.9']
title = 'compile_command exec not raising syntax error with new PGEN Parser'
updated_at = <Date 2020-05-11.03:40:31.514>
user = 'https://github.com/Carreau'

bugs.python.org fields:

activity = <Date 2020-05-11.03:40:31.514>
actor = 'gvanrossum'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-11.00:41:39.953>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2020-05-10.23:03:34.968>
creator = 'mbussonn'
dependencies = []
files = []
hgrepos = []
issue_num = 40585
keywords = ['patch']
message_count = 3.0
messages = ['368594', '368597', '368600']
nosy_count = 3.0
nosy_names = ['gvanrossum', 'mbussonn', 'pablogsal']
pr_nums = ['20030']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue40585'
versions = ['Python 3.9']

[Carreau]

compile_command used to produce syntax error in exec mode:

$ python -c "import codeop; codeop.compile_command('raise = 2', symbol='exec')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "...python3.8/codeop.py", line 125, in compile_command
    return _maybe_compile(_compile, source, filename, symbol)
  File "...python3.8/codeop.py", line 100, in _maybe_compile
    raise err1
  File "...python3.8/codeop.py", line 87, in _maybe_compile
    code1 = compiler(source + "\n", filename, symbol)
  File "...python3.8/codeop.py", line 105, in _compile
    return compile(source, filename, symbol, PyCF_DONT_IMPLY_DEDENT)
  File "<input>", line 1
    raise = 2
          ^
SyntaxError: invalid syntax

This happens to not be the case anymore in master, where it simply return None for above invalid input.

$ python -c "import codeop; codeop.CommandCompiler()('raise = 2', symbol='exec')"

or many other:

 $ python
Python 3.9.0a6+ (heads/master:2cc9b8486d, May 10 2020, 15:52:00)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import codeop;
>>> codeop.compile_command('def a-b', symbol='exec')
>>> codeop.compile_command('await?', symbol='exec')
>>> codeop.compile_command('=!=', symbol='exec')
>>> codeop.compile_command('a await raise b', symbol='exec')
>>> codeop.compile_command('a await raise b?+1', symbol='exec')

It is problematic as this is used in many places to decide whether code is valid, and for example in IPython to know wether we should insert a new line, or try to execute the code.

It seem to be due to the new PGEN parser as setting PYTHONOLDPARSER solve the issue:

$ PYTHONOLDPARSER=1 python -c "import codeop; codeop.CommandCompiler()('raise = 2', symbol='exec')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 171, in __call__
    return _maybe_compile(self.compiler, source, filename, symbol)
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 100, in _maybe_compile
    raise err1
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 87, in _maybe_compile
    code1 = compiler(source + "\n", filename, symbol)
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 136, in __call__
    codeob = compile(source, filename, symbol, self.flags, True)
  File "<input>", line 1
    raise = 2
          ^
SyntaxError: invalid syntax

single and eval appear to behave fine.

[gvanrossum]

Let's analyze this here rather than in bpo-40334 (which is getting top-heavy :-).

[pablogsal]

New changeset 5b956ca by Pablo Galindo in branch 'master':
bpo-40585: Normalize errors messages in codeop when comparing them (GH-20030)
5b956ca