[PEP 617 new parser] Regression in multiline SyntaxError offsets

[asottile]

BPO	40335
Nosy	@vstinner, @serhiy-storchaka, @asottile, @lysnikolaou, @pablogsal, @gaborbernat, @isidentical
PRs	bpo-40335: Correctly handle multi-line strings in tokenize error scenarios #19619

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-19.17:20:21.611>
created_at = <Date 2020-04-19.21:40:55.175>
labels = ['interpreter-core', 'type-bug', '3.9']
title = '[PEP 617 new parser] Regression in multiline SyntaxError offsets'
updated_at = <Date 2020-10-19.17:20:21.611>
user = 'https://github.com/asottile'

bugs.python.org fields:

activity = <Date 2020-10-19.17:20:21.611>
actor = 'lys.nikolaou'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-19.17:20:21.611>
closer = 'lys.nikolaou'
components = ['Interpreter Core']
creation = <Date 2020-04-19.21:40:55.175>
creator = 'Anthony Sottile'
dependencies = []
files = []
hgrepos = []
issue_num = 40335
keywords = ['patch']
message_count = 13.0
messages = ['366805', '366807', '366905', '367220', '367224', '367226', '367227', '367228', '367233', '367236', '367238', '367414', '378965']
nosy_count = 7.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'Anthony Sottile', 'lys.nikolaou', 'pablogsal', 'gaborjbernat', 'BTaskaya']
pr_nums = ['19619']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40335'
versions = ['Python 3.9']

[asottile]

this was noticed in pyflakes's testsuite: PyCQA/pyflakes#532 (review)

I've created a small script to reproduce the problem

import sys

SRC = b"""\
def foo():
    '''

def bar():
    pass

def baz():
    '''quux'''
"""

try:
    exec(SRC)
except SyntaxError as e:
    print(
        f'{sys.version}\n\n'
        f'{type(e).__name__}:\n'
        f'- line: {e.lineno}\n'
        f'- offset: {e.offset}\n'
        f'- text: {e.text!r}\n'
    )

This was "fixed" in python3.8, but has regressed:

$ python3.6 t2.py
3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0]

SyntaxError:
- line: 8
- offset: 52
- text: "    '''\n\ndef bar():\n    pass\n\ndef baz():\n    '''quux'''\n"

$ python3.8 t2.py
3.8.2 (default, Feb 26 2020, 02:56:10) 
[GCC 7.4.0]

SyntaxError:
- line: 8
- offset: 8
- text: "    '''\n\ndef bar():\n    pass\n\ndef baz():\n    '''quux'''\n"

$ ./python t2.py
3.9.0a5+ (heads/master:3955da8568, Apr 19 2020, 14:29:48) 
[GCC 7.5.0]

SyntaxError:
- line: 8
- offset: 52
- text: "    '''\n\ndef bar():\n    pass\n\ndef baz():\n    '''quux'''\n"

[pablogsal]

The regression happened in:

commit 41d5b94
Author: Lysandros Nikolaou <lisandrosnik@gmail.com>
Date: Sun Apr 12 21:21:00 2020 +0300

bpo-40246: Report a better error message for invalid string prefixes (GH-19476)

Will try to get a look at this shortly

[pablogsal]

New changeset 11a7f15 by Pablo Galindo in branch 'master':
bpo-40335: Correctly handle multi-line strings in tokenize error scenarios (GH-19619)
11a7f15

[asottile]

This seems to have regressed again

$ ./python --version --version
Python 3.9.0a5+ (heads/master:503de7149d, Apr 24 2020, 13:34:49) 
[GCC 7.5.0]
$ ./python t.py
  File "/home/asottile/workspace/cpython/t.py", line 8
    
                                        ^
SyntaxError: invalid string prefix

[pablogsal]

This is likely due to the new parser (see PEP-617). Do you get the same problem running with -X oldparser ?