Specialise syntax error of **dict in f-string field

[JNCressey]

BPO	41064
Nosy	@terryjreedy, @ericvsmith, @pablogsal, @E-Paine, @JNCressey
PRs	bpo-41064: Improve syntax error for invalid usage of '**' in f-strings #25006

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-03-24.19:34:39.459>
created_at = <Date 2020-06-21.12:32:38.555>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10']
title = 'Specialise syntax error of **dict in f-string field'
updated_at = <Date 2021-03-24.19:34:42.616>
user = 'https://github.com/JNCressey'

bugs.python.org fields:

activity = <Date 2021-03-24.19:34:42.616>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-03-24.19:34:39.459>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2020-06-21.12:32:38.555>
creator = 'JNCressey'
dependencies = []
files = []
hgrepos = []
issue_num = 41064
keywords = ['patch']
message_count = 7.0
messages = ['371995', '371999', '372007', '389366', '389375', '389421', '389486']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'eric.smith', 'pablogsal', 'epaine', 'JNCressey']
pr_nums = ['25006']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41064'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[JNCressey]

Compare f"{*my_tuple}" with f"{**my_dict}".
Both are syntax errors since you can't use unpacking here, but the bug is as follows:

• For the tuple, IDLE highlights the asterisk and has the helpful message 'SyntaxError: can't use starred expression here',

• But for the dictionary, the first few characters of your code are highlighted, regardless of where the syntax error is located, and the message only says 'SyntaxError: invalid syntax'.

Bug occurs in both 3.8.3 and 3.7.7, I haven't tested it in 3.6 nor in-development versions.

[E-Paine]

This isn't an IDLE issue and is instead due to the core interpreter. This isn't a bug, there just isn't a more specialised error message for the dictionary unpacking failure.

Changing the error message to something more helpful would be an enhancement and therefore would not be back-ported to Python 3.9 and before.

[terryjreedy]

IDLE highlights the location of a SyntaxError according to its .lineno and .offset attributes. For expressions within braces within f strings, the reported offset is respect to the expression augmented with surrounding ()s added, not the actual physical line. From my perspective as IDLE maintainer, I consider this undocumented 'feature' a design bug.

Pablo, I know you did not do this, but you helped design and understand the new compiler. If we cannot change .offset to report the real offset, can we add a new attribute, .line_offset for 3.9 and 3.10? Does the new compiler know where within the line the expression starts? Are there any other cases where .offset is not the real offset with respect to the actual code? Or rather, are there other cases where the actual line is replaced with a constructed line? Is so, are confusing fake ()s always added, that make the replacement *not* a substring of the original? Are there any plans to add new cases like this?

In the absence of compile providing the line offset, every IDE that marks errors in the actual line has to separately code a workaround. However, a change in 3.9 makes this harder. In 3.8, errors within fstring expressions were tagged with .filename '<fstring>'. In 3.9, this is attribute is ''.

The REPL in 3.7 (very soon security fixes only) and 3.8:
>>> f'{*a}'
  File "<stdin>", line 1
SyntaxError: can't use starred expression here
>>> f'{**a}'
  File "<fstring>", line 1
    (**a)
     ^
SyntaxError: invalid syntax
>>> f'{a a}'
  File "<fstring>", line 1
    (a a)
       ^
SyntaxError: invalid syntax


The REPL in 3.9/3.10 with the new parser is more consistent.
>>> f'{*a}'
  File "<stdin>", line 1
    (*a)                  # Does not ^ belong under *?  See below.
       ^
SyntaxError: invalid syntax
>>> f"{'abc' + *a}"
  File "<stdin>", line 1
    ('abc' + *a)          # This was the same in 3.8.
             ^
SyntaxError: invalid syntax

[terryjreedy]

Cressey noted both a compile and IDLE Shell issue. They must be handled separately by different people in separate bpo issues.

The first is about the helpful versus less helpful SyntaxError messages in the following (master compiled today).

>>> f'{*x}'
  File "<stdin>", line 1
    (*x)
     ^
SyntaxError: f-string: can't use starred expression here
>>> f'{**x}'
  File "<stdin>", line 1
    (**x)
     ^
SyntaxError: f-string: invalid syntax

The request is make the 2nd message the same or like the 1st. Seems like it should be possible. I am limiting this bpo issue to this request and opened bpo-43600 for fixing IDLE's highlight location.

Pablo, I leave it to you to handle the message enhancement request. Do you know if there is any other situation like this in which the compiled text is not the input code?

[ericvsmith]

The parens are added in fstring_compile_expr at

cpython/Parser/string_parser.c

Line 391 in 9feae41

// The parentheses are needed in order to allow for leading whitespace within

I don't recall if this is really only a "skip leading whitespace" problem, or if there's some other reason they're required. If it's just a whitespace issue, maybe skipping leading whitespace in fstring_compile_expr is a better idea? One of the reasons I originally didn't want to skip leading whitespace is because I didn't want to add yet another place where we'd have to maintain a "what qualifies as whitespace to the compiler?" decision.

[pablogsal]

I don't recall if this is really only a "skip leading whitespace" problem, or if there's some other reason they're required.

IIRC, this forces the expression inside to be parsed as an expression. This helps the quite a lot the parser. For instance, detecting a bare **x without grouping is more difficult, even with the old parser. With parentheses it must be an expression so is easier to disambiguate.

[pablogsal]

New changeset 8efad61 by Pablo Galindo in branch 'master':
bpo-41064: Improve syntax error for invalid usage of '**' in f-strings (GH-25006)
8efad61