Error message differs when an expression is in an fstring

[lysnikolaou]

BPO	40267
Nosy	@gvanrossum, @ericvsmith, @lysnikolaou, @pablogsal, @isidentical
PRs	bpo-40267: Fix message when last input character produces a SyntaxError #19521

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-04-15.18:24:17.588>
created_at = <Date 2020-04-12.21:59:24.577>
labels = ['interpreter-core', 'type-bug', '3.9']
title = 'Error message differs when an expression is in an fstring'
updated_at = <Date 2020-04-15.18:24:17.587>
user = 'https://github.com/lysnikolaou'

bugs.python.org fields:

activity = <Date 2020-04-15.18:24:17.587>
actor = 'gvanrossum'
assignee = 'none'
closed = True
closed_date = <Date 2020-04-15.18:24:17.588>
closer = 'gvanrossum'
components = ['Interpreter Core']
creation = <Date 2020-04-12.21:59:24.577>
creator = 'lys.nikolaou'
dependencies = []
files = []
hgrepos = []
issue_num = 40267
keywords = ['patch']
message_count = 3.0
messages = ['366272', '366273', '366535']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'eric.smith', 'lys.nikolaou', 'pablogsal', 'BTaskaya']
pr_nums = ['19521']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40267'
versions = ['Python 3.9']

[lysnikolaou]

There are cases, where the error message differs, when an expression is being parsed inside an fstring. For example:

>>> f'{x+}'
  File "<fstring>", line 1
    (x+)
       ^
SyntaxError: unexpected EOF while parsing
>>> (x+)
  File "<stdin>", line 1
    (x+)
       ^
SyntaxError: invalid syntax

Or with lambda definitions:

>>> f'{lambda x:x}'
  File "<fstring>", line 1
    (lambda x)
             ^
SyntaxError: unexpected EOF while parsing
>>> (lambda x)
  File "<stdin>", line 1
    (lambda x)
             ^
SyntaxError: invalid syntax

[lysnikolaou]

It seems that this is actually a bit bigger than this and it is not specific to f-strings.

The error message always changes to unexpected EOF while parsing if there is an error with the last character of the input and no newline follows. For example, as made clear to me by Guido, there are even differences in error messages between exec'ing and eval'ing something:

>>> exec('x+')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    x+
     ^
SyntaxError: invalid syntax
>>> eval('x+')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    x+
     ^
SyntaxError: unexpected EOF while parsing

That's because the tokenizer adds an implicit newline to the input string, before tokenizing it, when the input comes from an exec call. (see

cpython/Parser/tokenizer.c

Line 648 in 14d5331

/* If this is exec input, add a newline to the end of the string if

)

And that's not limited to a character missing, as suggested by the error message. Even when the last character itself generates a SyntaxError, the error message remains "unexpcted EOF while parsing":

>>> x+@
  File "<stdin>", line 1
    x+@
      ^
SyntaxError: invalid syntax
>>> eval('x+@')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    x+@
      ^
SyntaxError: unexpected EOF while parsing

Thus, a very simple fix to the specific fstring problem of this issue would be to add a newline to the string that gets parsed to the parser in fstring_compile_expr in ast.c, but I guess it'd be better to fix the tokenizer itself, if this is considered a bug.

[gvanrossum]

New changeset 9a4b38f by Lysandros Nikolaou in branch 'master':
bpo-40267: Fix message when last input character produces a SyntaxError (GH-19521)
9a4b38f