Make traceback module's formatting of SyntaxError more similar to system formatting

[gvanrossum]

BPO	40612
Nosy	@gvanrossum, @lysnikolaou, @pablogsal, @miss-islington
PRs	bpo-40612: Fix SyntaxError edge cases in traceback formatting #20072

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-05-17.23:52:37.235>
created_at = <Date 2020-05-13.04:18:55.037>
labels = ['3.9']
title = "Make traceback module's formatting of SyntaxError more similar to system formatting"
updated_at = <Date 2020-05-17.23:52:37.235>
user = 'https://github.com/gvanrossum'

bugs.python.org fields:

activity = <Date 2020-05-17.23:52:37.235>
actor = 'gvanrossum'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-17.23:52:37.235>
closer = 'gvanrossum'
components = []
creation = <Date 2020-05-13.04:18:55.037>
creator = 'gvanrossum'
dependencies = []
files = []
hgrepos = []
issue_num = 40612
keywords = ['patch']
message_count = 6.0
messages = ['368759', '368853', '368856', '368857', '368859', '368917']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'lys.nikolaou', 'pablogsal', 'miss-islington']
pr_nums = ['20072']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue40612'
versions = ['Python 3.9']

[gvanrossum]

The traceback module formats several edge cases of SyntaxError different than CPython's default formatting.

• There's an off-by-one error if the column offset (printed as a caret) points past the last character (example: 'a +='). The clipping is wrong and the caret points to the last character.

• If the offset is <= 0, it appears the code silently adds the length of the source text.

• The system formatting suppresses the caret if the offset is -1; the only way to suppress the caret with the traceback module is setting the offset to None (or setting the source text to None).

• The system formatting can position the caret way past the end of the source text; the traceback module clips (also see the first bullet).

I propose to make the traceback module behave the same way as the system module in all cases. I also propose to make both suppress the caret if the offset is <= 0. Finally I propose to make the system formatting limit the offset to just past the end of the source text.

I propose not to bother changing anything in 3.8 or before.

[lysnikolaou]

Agreed on everything.

One thing I don't really understand is if you propose to also strip trailing whitespace. Does "limit the offset to just past the end of the source text" include whitespace or not?

For example, the linked PR does not change this behavior:

➜  cpython git:(pr/20072) ./python
Python 3.9.0a6+ (heads/pr/20072:6df7662ca5, May 14 2020, 20:37:50) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> compile('1 +      ', '<string>', 'exec')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    1 +      
             ^
SyntaxError: invalid syntax

Should we clip just past the end of 1 + here?

[gvanrossum]

My current PR does not strip trailing whitespace. It only strips a single trailing newline (since this is usually but not always present, and we don't want to its presence to cause an extra blank line, nor do we want its absence to cause the text line and the caret line to be run together).

[gvanrossum]

(And, to be clear, I don't *want* to strip trailing spaces.)

[lysnikolaou]

Understood.

[miss-islington]

New changeset 15bc9ab by Guido van Rossum in branch 'master':
bpo-40612: Fix SyntaxError edge cases in traceback formatting (GH-20072)
15bc9ab