PEG Parser: Cannot used starred expression in parenthesised expr

[lysnikolaou]

BPO	40631
Nosy	@gvanrossum, @serhiy-storchaka, @lysnikolaou, @pablogsal, @miss-islington, @gousaiyang
PRs	bpo-40631: Single-element starred elements in parenthesised expressions cannot be parsed #24014 bpo-40631: Improve error message when deleting single starred element #24015 bpo-40631: Disallow star targets with parentheses around #24019 bpo-40631: Disallow single parenthesized star target #24027 [3.9] bpo-40631: Disallow single parenthesized star target (GH-24027) #24068

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/lysnikolaou'
closed_at = <Date 2021-01-03.02:01:34.480>
created_at = <Date 2020-05-15.12:04:27.756>
labels = ['interpreter-core', '3.9']
title = 'PEG Parser: Cannot used starred expression in parenthesised expr'
updated_at = <Date 2021-01-03.02:01:34.480>
user = 'https://github.com/lysnikolaou'

bugs.python.org fields:

activity = <Date 2021-01-03.02:01:34.480>
actor = 'pablogsal'
assignee = 'lys.nikolaou'
closed = True
closed_date = <Date 2021-01-03.02:01:34.480>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2020-05-15.12:04:27.756>
creator = 'lys.nikolaou'
dependencies = []
files = []
hgrepos = []
issue_num = 40631
keywords = ['patch']
message_count = 12.0
messages = ['368936', '368938', '384086', '384089', '384091', '384092', '384093', '384094', '384096', '384098', '384244', '384247']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'serhiy.storchaka', 'lys.nikolaou', 'pablogsal', 'miss-islington', 'gousaiyang']
pr_nums = ['24014', '24015', '24019', '24027', '24068']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue40631'
versions = ['Python 3.9']

[lysnikolaou]

The new PEG parser fails when a parenthesised expression with a single child (a group) contains a starred expression. Example:

╰─ ./python.exe
Python 3.9.0a6+ (heads/master-dirty:4a12d12186, May 15 2020, 14:53:45)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.parse('(*a)')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lysnikolaou/Repositories/cpython/Lib/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    (*a)
        ^
SyntaxError: invalid syntax

This was valid syntax up until now:

╰─ ./python.exe -X oldparser
Python 3.9.0a6+ (heads/master-dirty:4a12d12186, May 15 2020, 14:53:45)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.dump(ast.parse('(*a)'))
"Module(body=[Expr(value=Starred(value=Name(id='a', ctx=Load()), ctx=Load()))], type_ignores=[])"

[lysnikolaou]

Whoops, false alarm.

It's just that we moved the check for invalid starred expressions to the parser, while it previously was in the compiler.

╰─ ./python.exe -X oldparser
Python 3.9.0a6+ (heads/master-dirty:003708bcf8, May 15 2020, 15:08:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> (*a)
  File "<stdin>", line 1
SyntaxError: can't use starred expression here

Sorry for the noise!

[gousaiyang]

Well, there is actually a bug:

root@f1b4a742d8fc:/# python3.9
Python 3.9.1 (default, Dec  8 2020, 03:24:52)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x = [1, 2]
>>> y = 3
>>> *x, y
(1, 2, 3)
>>> (*x), y
  File "<stdin>", line 1
    (*x), y
     ^
SyntaxError: can't use starred expression here

root@f1b4a742d8fc:/# python3.8
Python 3.8.6 (default, Oct  6 2020, 03:22:36)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x = [1, 2]
>>> y = 3
>>> *x, y
(1, 2, 3)
>>> (*x), y
(1, 2, 3)

This is different from the previous message where the starred expression is "alone" (and thus invalid). Since this bug happens in 3.9 but not in 3.8, it might be due to the PEG parser.

Also,
root@f1b4a742d8fc:/# python3.9
Python 3.9.1 (default, Dec  8 2020, 03:24:52)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> del *x
  File "<stdin>", line 1
    del *x
        ^
SyntaxError: cannot delete starred
>>> del (*x)
  File "<stdin>", line 1
    del (*x)
         ^
SyntaxError: can't use starred expression here

The latter case should also report "SyntaxError: cannot delete starred".

[gvanrossum]

Honestly this seems like a bug in 3.8 to me (if it indeed behaves like this):

>>> (*x), y
(1, 2, 3)

Every time I mistakenly tried (*x) I really meant (*x,), so it's surprising that (*x), y would be interpreted as (*x, y) rather than flagging (*x) as an error.

Please don't "fix" this even if it is a regression.

[lysnikolaou]

It makes sense to me to be able to do (*a), b if I can do *a, b, but I don't really have a strong opinion on it.

[serhiy-storchaka]

Parenthesis can be added around expression. But *a is not an expression (as well as +, or, 1:5, you cannot surround them with parenthesis).

[gousaiyang]

Also the current behavior allows (*x), y = 1 assignment. If (*x) is to be totally disallowed, (*x), y = 1 should also be rejected.

[gvanrossum]

It makes sense to me to be able to do (*a), b if I can do *a, b, but I don't really have a strong opinion on it.

I disagree. *a is not an expression, so the normal rules for parenthesizing those don't apply. I've always thought of *a as a feature of the "comma" syntax. Note too that (**a) is not valid and never was. Also note that 2.7 doesn't support f((*a)). In fact 3.4 doesn't either -- but 3.5 does.

I don't know how this slipped into earlier Python 3 versions -- apparently there aren't tests for this, and it's not used in popular 3rd code either, or we would have found out when we first implemented PEP-617.

Most likely it's due to the general problem where the parser would just accept parenthesized stuff in various places where it shouldn't (e.g. also f(a=1) could be spelled as f((a)=1) -- this was fixed in 3.8).

Also the current behavior allows (*x), y = 1 assignment. If (*x) is to be totally disallowed, (*x), y = 1 should also be rejected.

I agree.

[lysnikolaou]

Yup, this all sounds much more reasonable. Thanks for the explanation, Guido.

Also the current behavior allows (*x), y = 1 assignment. If (*x) is to be totally disallowed, (*x), y = 1 should also be rejected.

This is allowed in 3.9.1 and 3.10.0a2, but not allowed in 3.9.0 and 3.10.0a1. I'll work on finding out when this got messed up and fix it.

[pablogsal]

I'll work on finding out when this got messed up and fix it.

Apparently is

commit bca7014 (HEAD)
Author: Lysandros Nikolaou <lisandrosnik@gmail.com>
Date: Tue Oct 27 00:42:04 2020 +0200

bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111)

~/github/python/master v3.10.0a2~61 9s
❯ ./python -c "(*x), y = 1,2"

~/github/python/master v3.10.0a2~61
❯ git checkout HEAD^
Previous HEAD position was bca7014 bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111)
HEAD is now at c8c4200 bpo-42157: Convert unicodedata.UCD to heap type (GH-22991)

~/github/python/master remotes/welikeparsers/master
❯ make -j -s
CC='gcc -pthread' LDSHARED='gcc -pthread -shared ' OPT='-g -Og -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py -q build

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time

~/github/python/master remotes/welikeparsers/master
❯ ./python -c "(*x), y = 1,2"
File "<string>", line 1
(*x), y = 1,2
^
SyntaxError: can't use starred expression here

[lysnikolaou]

New changeset 2ea320d by Lysandros Nikolaou in branch 'master':
bpo-40631: Disallow single parenthesized star target (GH-24027)
2ea320d

[miss-islington]

New changeset 9a608ac by Lysandros Nikolaou in branch '3.9':
[3.9] bpo-40631: Disallow single parenthesized star target (GH-24027) (GH-24068)
9a608ac