Suggestion for better syntax errors in tokenizer errors

[wyz23x2]

BPO	44317
Nosy	@serhiy-storchaka, @lysnikolaou, @pablogsal, @miss-islington, @wyz23x2
PRs	bpo-44317: Improve tokenizer errors with more informative locations #26555 [3.10] bpo-44317: Improve tokenizer errors with more informative locations (GH-26555) #27079

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-07-10.00:47:54.504>
created_at = <Date 2021-06-05.11:01:10.180>
labels = ['interpreter-core', 'type-feature', '3.10', '3.11']
title = 'Suggestion for better syntax errors in tokenizer errors'
updated_at = <Date 2021-07-10.00:47:54.504>
user = 'https://github.com/wyz23x2'

bugs.python.org fields:

activity = <Date 2021-07-10.00:47:54.504>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-07-10.00:47:54.504>
closer = 'pablogsal'
components = ['Parser']
creation = <Date 2021-06-05.11:01:10.180>
creator = 'wyz23x2'
dependencies = []
files = []
hgrepos = []
issue_num = 44317
keywords = ['patch']
message_count = 7.0
messages = ['395161', '395179', '395182', '395188', '395250', '397233', '397234']
nosy_count = 5.0
nosy_names = ['serhiy.storchaka', 'lys.nikolaou', 'pablogsal', 'miss-islington', 'wyz23x2']
pr_nums = ['26555', '27079']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44317'
versions = ['Python 3.10', 'Python 3.11']

[wyz23x2]

Python 3.10.0b2 (tags/v3.10.0b2:3173141, Jun  1 2021, 09:05:29) [MSC v.1928 64 bit (AMD64)] on win32 
Type "help", "copyright", "credits" or "license" for more information.
>>> 0777
  File "<stdin>", line 1
    0777
       ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers
>>> 000123
  File "<stdin>", line 1
    000123
         ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers

The ^ is placed below the last digit.
However, this is misleading. The error is "leading zeros" and "prefix". So I would expect this:

>>> 0777
  File "<stdin>", line 1
    0777
    ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers
>>> 000123
  File "<stdin>", line 1
    000123
    ^^^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers

Opinions?

[wyz23x2]

Another 2 problems:
1.
>>> 0b1112
  File "<stdin>", line 1
    0b1112
         ^
SyntaxError: invalid digit '2' in binary literal
>>> 0o5780
  File "<stdin>", line 1
    0o5780
        ^
SyntaxError: invalid digit '8' in octal literal
But:
>>> 0x2fag
  File "<stdin>", line 1
    0x2fag
    ^^^^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?
>>> 
Is this expected?

2.
>>> 0o91
  File "<stdin>", line 1
    0o91
     ^
SyntaxError: invalid digit '9' in octal literal
>>> 0b21
  File "<stdin>", line 1
    0b21
     ^
SyntaxError: invalid digit '2' in binary literal

The ^ is misplaced again, even though, say the 0b1112 example above works.

[pablogsal]

Is this expected?

Yes, is an edge case of python identifiying two tokens together except that there is no space:

>>> 3 4
  File "<stdin>", line 1
    3 4
    ^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?

I honestly don't share your concerns that these things are "misleading". The caret is pointing to the token that is incorrect 0777. The tokenizer errors always point at the end of the token (we still have not implemented ranged errors for the tokenizer).

This is true in all the cases you present.

[pablogsal]

PR 26555 does some improvements to your examples:

>>> 0777
  File "<stdin>", line 1
    0777
    ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers
>>> 000007777
  File "<stdin>", line 1
    000007777
    ^^^^^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers
>>> 0b1112
  File "<stdin>", line 1
    0b1112
         ^
SyntaxError: invalid digit '2' in binary literal
>>> 0o91
  File "<stdin>", line 1
    0o91
      ^
SyntaxError: invalid digit '9' in octal literal
>>> 0b21
  File "<stdin>", line 1
    0b21
      ^
SyntaxError: invalid digit '2' in binary literal
>>>

[serhiy-storchaka]

See also bpo-43833.

[pablogsal]

New changeset f24777c by Pablo Galindo Salgado in branch 'main':
bpo-44317: Improve tokenizer errors with more informative locations (GH-26555)
f24777c

[pablogsal]

New changeset 2a722d4 by Miss Islington (bot) in branch '3.10':
bpo-44317: Improve tokenizer errors with more informative locations (GH-26555) (GH-27079)
2a722d4