Update doctect SyntaxErrors for location range

[akulakov]

BPO	45249
Nosy	@terryjreedy, @blueyed, @pablogsal, @miss-islington, @Fidget-Spinner, @akulakov
PRs	bpo-45249: Add regression test for display of SyntaxError range indicator in doctests #28567 bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges #28575 [3.10] bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges (GH-28575) #28587 bpo-45249: Fix caret location when end_offset is set to 0 #28854 bpo-45249: Fix caret location when end_offset is set to 0 #28855 [3.10] bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855) #28994

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-16.17:37:28.444>
created_at = <Date 2021-09-20.16:30:58.559>
labels = ['type-bug', 'library', '3.10', '3.11']
title = 'Update doctect SyntaxErrors for location range'
updated_at = <Date 2021-10-16.17:51:11.950>
user = 'https://github.com/akulakov'

bugs.python.org fields:

activity = <Date 2021-10-16.17:51:11.950>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-16.17:37:28.444>
closer = 'pablogsal'
components = ['Library (Lib)']
creation = <Date 2021-09-20.16:30:58.559>
creator = 'andrei.avk'
dependencies = []
files = []
hgrepos = []
issue_num = 45249
keywords = ['patch']
message_count = 18.0
messages = ['402257', '402258', '402260', '402263', '402268', '402269', '402271', '402284', '402589', '402652', '402662', '402670', '402739', '402742', '403157', '403588', '404103', '404106']
nosy_count = 6.0
nosy_names = ['terry.reedy', 'blueyed', 'pablogsal', 'miss-islington', 'kj', 'andrei.avk']
pr_nums = ['28567', '28575', '28587', '28854', '28855', '28994']
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45249'
versions = ['Python 3.10', 'Python 3.11']

Linked PRs

• gh-89412: Add missing attributes (added in 3.10) to traceback module docs #105046
• [3.12] gh-89412: Add missing attributes (added in 3.10) to traceback module docs (GH-105046) #105327
• [3.11] gh-89412: Add missing attributes (added in 3.10) to traceback module docs (GH-105046) #105328
• [3.10] gh-89412: Add missing attributes (added in 3.10) to traceback module docs (GH-105046) #105329

[akulakov]

It seems like fine grained error locations do not work in failed doctest traceback output:

version 3.11.0a0

file contents:
------------------

def a(x):
    """
    >>> 1 1
    1
    """
import doctest
doctest.testmod()

OUTPUT
-------

Failed example:
    1 1
Exception raised:
    Traceback (most recent call last):
      File "/Users/ak/opensource/cpython/Lib/doctest.py", line 1348, in __run
        exec(compile(example.source, filename, "single",
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<doctest __main__.a[0]>", line 1
        1 1
        ^
    SyntaxError: invalid syntax. Perhaps you forgot a comma?

The location in doctests that causes this:

cpython/Lib/doctest.py

Line 1348 in 5846c9b

exec(compile(example.source, filename, "single",

[akulakov]

I've ran into this when looking at doctest docs, the issue is that they use the old example where a single column is highlighted, I want to update it to explain why doctest output differs from the one you get from REPL, but I probably need to understand why it happens to provide a good explanation.

Alternatively this may be fixed to be consistent if it's easy enough to do.

[pablogsal]

Hummmm, could you explain a bit more in detail what is the expected output? I can see highlighting in the exec call that you pasted:

    exec(compile(example.source, filename, "single",
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The fact that you see those "^^^^^^" indicate that is working no? What is missing?

[akulakov]

Sorry, I should have noted I’m referring to the line 1 1
Which is underlined by a single caret, but on the command line it has 3
carets as expected.

[pablogsal]

Ah, but that is a different issue. This is not PEP-657, this is a SyntaxError, so is related how those are printed, which I think is separared.

[pablogsal]

Can you try a doctest that fails on something that is not a SyntaxError.

Something like:

>>> def foo(x):
...    return x + 42

>> foo(None)

[akulakov]

Pablo: that works fine, thanks!

I will look into updating the doctest docs, and will close this issue later
today ( or you can close it if you like).

[akulakov]

Thinking a bit more on this, I'm not sure this can be closed, as SyntaxError indicators seems not to be working. I'm not sure if this is limited to doctests or not. I've updated the title to narrow it down to SyntaxErrors.

I can look more into this if needed. If it's just doctests, it doesn't matter because the doctests ignore the indicators anyway, but if it can happen in some other cases, perhaps it's worth fixing.

However I don't have experience with the C parts of python, so I'm not sure where / how to look.

Pablo: let me know if you have any pointers on how to debug this, or if you think this should be closed.

[terryjreedy]

In 3.10+, end_lineno and end_offset fields were added to SyntaxError objects and the args tuple.

>>> try: compile('1 1', '', 'single')
... except SyntaxError as e: print(e.args)
... 
('invalid syntax. Perhaps you forgot a comma?', ('', 1, 1, '1 1', 1, 4))

Here, line 1, offset 4 is the first 1-based column not part of the error.

The builtin default sys.excepthook was modified to read and use this new information and mark (end_offset - offset) columns with '^'s. This default prints what it does to sys.stderr.

The syntax error formatting in the traceback module was not altered. However, a new method, TracebackException._format_syntax_error, was extracted from TracebackException.format_exception_only so that the former could be overridden by software that simulates interaction.

The printed traceback does not come from line 1348. That *executes* the user code, but all Exceptions, including SyntaxError, are caught. If the exception is not expected and the run is not quiet, the exception is output by report_unexpected_exception(), as seen above as 'OUTPUT' and the lines that follows.

cpython/Lib/doctest.py

Line 1264 in 5846c9b

def report_unexpected_exception(self, out, test, example, exc_info):

This calls _exception_traceback(exc_info).

cpython/Lib/doctest.py

Line 244 in 5846c9b

def _exception_traceback(exc_info):

This calls traceback.print_exception, which I believe, for syntax errors, ultimately calls TracebackException._format_syntax_error.

I believe that the options for a fix are either

1. Call default sys.excepthook while capturing its output into a StringIO instance.
2. Assuming I am correct above about _format_syntax_error being called, monkeypatch it. In line 779,
   

   cpython/Lib/traceback.py

   Line 779 in 5846c9b

   yield ' {}^\n'.format(''.join(caretspace))

   
   replace '^' with a field with a calculated number of ^s.

I need to do one of these two for IDLE, and may try both.

[akulakov]

Terry: I got it mostly working using your 2nd suggestion, I will do some testing and should be able to put up a PR in the next couple of days. Thanks for looking at this and explaining!

[pablogsal]

One important thing is that "traceback.print_exception" should correctly print syntax errors .

[pablogsal]

New changeset 20f439b by Pablo Galindo Salgado in branch 'main':
bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges (GH-28575)
20f439b

[miss-islington]

New changeset c7fdd68 by Miss Islington (bot) in branch '3.10':
bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges (GH-28575)
c7fdd68

[pablogsal]

New changeset 1e20582 by Pablo Galindo (Miss Islington (bot)) in branch '3.10':
bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges (GH-28575)
1e20582

[blueyed]

I've noticed a regression/change with the code change for this issue.

When not catching the exception from compile("invalid(", "<stdin>", "single") it has a caret below the opening parenthesis:

Traceback (most recent call last):
  File "…/t-syntaxerror-chained.py", line 2, in <module>
    compile("invalid(", "<stdin>", "single")
  File "<stdin>", line 1
    invalid(
           ^
SyntaxError: '(' was never closed

When using traceback.print_exc however this is missing:

Traceback (most recent call last):
  File "…/t-syntaxerror-chained.py", line 2, in <module>
    compile("invalid(", "<stdin>", "single")
  File "<stdin>", line 1
    invalid(

SyntaxError: '(' was never closed

The file used for testing:

try:
    compile("invalid(", "<stdin>", "single")
except Exception:
    # raise
    __import__("traceback").print_exc()

(this change was noticed between 3.10.0rc2 and the final release with pdbpp's test suite)

I've not investigated further (yet), and also feel free to ask for creating a new issue, but I've figured it would be good to notify you here first (where the code was changed).

[miss-islington]

New changeset fe0d9e2 by Pablo Galindo Salgado in branch 'main':
bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855)
fe0d9e2

[miss-islington]

New changeset 5df35fa by Miss Islington (bot) in branch '3.10':
bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855)
5df35fa