compile() can compile a bare starred expression with PyCF_ONLY_AST flag with the old parser, but not the new one

[adamwill]

BPO	40848
Nosy	@brettcannon, @tiran, @benjaminp, @serhiy-storchaka, @1st1, @AdamWill, @pablogsal, @isidentical

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-06-03.06:22:40.645>
created_at = <Date 2020-06-02.19:08:16.722>
labels = ['interpreter-core', 'type-bug', 'invalid', '3.9', '3.10']
title = 'compile() can compile a bare starred expression with `PyCF_ONLY_AST` flag with the old parser, but not the new one'
updated_at = <Date 2020-06-03.06:22:40.644>
user = 'https://github.com/adamwill'

bugs.python.org fields:

activity = <Date 2020-06-03.06:22:40.644>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-03.06:22:40.645>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2020-06-02.19:08:16.722>
creator = 'adamwill'
dependencies = []
files = []
hgrepos = []
issue_num = 40848
keywords = []
message_count = 4.0
messages = ['370620', '370625', '370626', '370643']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'christian.heimes', 'benjamin.peterson', 'serhiy.storchaka', 'yselivanov', 'adamwill', 'pablogsal', 'BTaskaya']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40848'
versions = ['Python 3.9', 'Python 3.10']

[adamwill]

Not 100% sure this would be considered a bug, but it seems at least worth filing to check. This is a behaviour difference between the new parser and the old one. It's very easy to reproduce:

<mock-chroot> sh-5.0# PYTHONOLDPARSER=1 python3
Python 3.9.0b1 (default, May 29 2020, 00:00:00) 
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from _ast import *
>>> compile("(*starred)", "<unknown>", "exec", flags=PyCF_ONLY_AST)
<ast.Module object at 0x7fe1504532e0>
>>> 
<mock-chroot> sh-5.0# python3
Python 3.9.0b1 (default, May 29 2020, 00:00:00) 
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from _ast import *
>>> compile("(*starred)", "<unknown>", "exec", flags=PyCF_ONLY_AST)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<unknown>", line 1
    (*starred)
              ^
SyntaxError: invalid syntax

That is, you can compile() the expression "(*starred)" with PyCF_ONLY_AST flag set with the old parser, but not with the new one. Without PyCF_ONLY_AST you get a SyntaxError with both parsers, though a with the old parser, the error message is "can't use starred expression here", not "invalid syntax".

[serhiy-storchaka]

There is no promise that the parser can parse invalid Python code. I think there is no a bug here.

[adamwill]

Realized I forgot to give it, so in case it's important, the context here is the black test suite:

psf/black#1441

that test suite has a file full of expressions that it expects to be able to parse this way (it uses ast.parse(), which in turn calls compile() with this flag). A bare (*starred) line is part of that file:

https://github.com/psf/black/blob/master/tests/data/expression.py#L149

and has been for as long as black has existed. Presumably if this isn't going to be fixed we'll need to adapt this black test file to test a starred expression in a 'valid' way, somehow.

[pablogsal]

Yeah, this is not a bug as Serhiy points out. Regarding the error message, we are currently working on trying to unify the error messages in the the parser as much as we can but there is also no promise that they will match 100%.

Many grammar checks currently are done outside the parser (we mention it on PEP-617) so I am not surprised that the old parser is ok with some invalid expressions and PyCF_ONLY_AST. This is one of the things we are trying to change with the new parser precisely.

I will close for now as "not a bug". Feel free to reopen if you think we missed something or if something was left to discuss.