unterminated string literal tokenization error messages could be better

[benjaminp]

BPO	40176
Nosy	@benjaminp, @alex, @serhiy-storchaka, @ammaraskar, @pablogsal, @miss-islington, @isidentical
PRs	bpo-40176: Improve error messages for unclosed string literals #19346

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-20.21:56:51.092>
created_at = <Date 2020-04-03.17:58:44.806>
labels = ['interpreter-core', 'type-feature', '3.10']
title = 'unterminated string literal tokenization error messages could be better'
updated_at = <Date 2021-01-20.21:56:51.091>
user = 'https://github.com/benjaminp'

bugs.python.org fields:

activity = <Date 2021-01-20.21:56:51.091>
actor = 'BTaskaya'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-20.21:56:51.092>
closer = 'BTaskaya'
components = ['Interpreter Core']
creation = <Date 2020-04-03.17:58:44.806>
creator = 'benjamin.peterson'
dependencies = []
files = []
hgrepos = []
issue_num = 40176
keywords = ['patch']
message_count = 10.0
messages = ['365713', '365730', '365743', '365749', '365765', '366036', '366303', '369541', '369542', '385374']
nosy_count = 7.0
nosy_names = ['benjamin.peterson', 'alex', 'serhiy.storchaka', 'ammar2', 'pablogsal', 'miss-islington', 'BTaskaya']
pr_nums = ['19346']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40176'
versions = ['Python 3.10']

[benjaminp]

It has been pointed out to me that the errors the tokenizer produces for unterminated strings, "EOL while scanning string literal" and "EOF while scanning triple-quoted string literal", contain parsing jargon that make it difficult for new users to understand the problem, likely a missing quote.

[serhiy-storchaka]

It could be even better. Inside the tokenizer we know where the string literal starts and what quotes it uses. The line and the offset of the *start* of the literal can be set in a SyntaxError.

[alex]

Here's my suggestion:

End of line reached without finding the end of string literal. Are you missing a closing quote?

[ammaraskar]

Just re-posting this here from the open PR. Rust's handling of this seems nice and beginner friendly:

error: unterminated double quote string
--> src/main.rs:2:19
|
2 | let message = "Hello world
| __________________^
3 | | println!(message);
4 | | }
| |^

Like Serhiy suggested, it points to the /start/ of the string, rather than the EOL and the message seems nice too.

[isidentical]

>>> message = "sadsa
  File "<stdin>", line 1
    message = "sadsa
              ^
SyntaxError: unterminated double quote

[serhiy-storchaka]

I afraid there may be confusion between triple, double and single quoted string literals. So I suggest to change error messages to just "unterminated triple-quoted string literal" and "unterminated string literal" (or "unterminated single-quoted string literal"). Terms "triple-quoted" and "single-quoted" are used several times in the documentation. Term "double-quoted" is used only once, and I suppose in different meaning.

[isidentical]

Fair point. I changed error messages to what you suggested

>>> (300) * 6 + ca(e, 2 +    "dsadsa)
  File "<stdin>", line 1
    (300) * 6 + ca(e, 2 +    "dsadsa)
                             ^
SyntaxError: unterminated string literal

>>> (300) * 6 + ca(e, 2 +    'dsadsa)
  File "<stdin>", line 1
    (300) * 6 + ca(e, 2 +    'dsadsa)
                             ^
SyntaxError: unterminated string literal


>>> (300) * 6 + ca(e, 2 +    """dsadsa
... 
  File "<stdin>", line 1
    (300) * 6 + ca(e, 2 +    """dsadsa
                             ^
SyntaxError: unterminated triple-quoted string literal

[pablogsal]

New changeset 72e0aa2 by Batuhan Taskaya in branch 'master':
bpo-40176: Improve error messages for trailing comma on from import (GH-20294)
72e0aa2

[pablogsal]

New changeset 275d7e1 by Pablo Galindo in branch '3.9':
[3.9] bpo-40176: Improve error messages for trailing comma on from import (GH-20294) (GH-20302)
275d7e1

[miss-islington]

New changeset a698d52 by Batuhan Taskaya in branch 'master':
bpo-40176: Improve error messages for unclosed string literals (GH-19346)
a698d52