pdb does not drop into debugger upon SyntaxError caused by ast.literal_eval

[KerrickStaley]

BPO	40403
Nosy	@terryjreedy, @xdegaye, @remilapeyre, @isidentical

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-27.03:07:35.981>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.7', 'library']
title = 'pdb does not drop into debugger upon SyntaxError caused by ast.literal_eval'
updated_at = <Date 2020-06-03.09:03:33.251>
user = 'https://bugs.python.org/KerrickStaley'

bugs.python.org fields:

activity = <Date 2020-06-03.09:03:33.251>
actor = 'xdegaye'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-04-27.03:07:35.981>
creator = 'Kerrick Staley'
dependencies = []
files = []
hgrepos = []
issue_num = 40403
keywords = []
message_count = 6.0
messages = ['367363', '370570', '370579', '370592', '370594', '370649']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'xdegaye', 'remi.lapeyre', 'BTaskaya', 'Kerrick Staley']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40403'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-84583: Make pdb enter post-mortem mode even for SyntaxError #110883

[KerrickStaley]

Summary:
When you call ast.literal_eval on a string that does not contain valid Python code, it raises a SyntaxError. This causes pdb to exit instead of stopping execution at the point that the SyntaxError was raised.

To reproduce:

1. Create a Python file foo.py with these contents:

    import ast
    ast.literal_eval('')

2. Run python -m pdb foo.py
3. Type 'c' and hit enter.

Expected behavior:
pdb drops into a shell in ast.py at the point where the SyntaxError occurred.

Actual behavior:
The program exits, and a SyntaxError traceback is displayed.

System configuration:
Python 3.8.2 on Arch Linux.

[remilapeyre]

I've looked into this, in Bdb both the part where the code is compiled and the one where the code is run are in the run() method (https://github.com/python/cpython/blob/master/Lib/bdb.py#L565-L585):

    def run(self, cmd, globals=None, locals=None):
        """Debug a statement executed via the exec() function.
        globals defaults to __main__.dict; locals defaults to globals.
        """
        if globals is None:
            import __main__
            globals = __main__.__dict__
        if locals is None:
            locals = globals
        self.reset()
        if isinstance(cmd, str):
            cmd = compile(cmd, "<string>", "exec")
        sys.settrace(self.trace_dispatch)
        try:
            exec(cmd, globals, locals)
        except BdbQuit:
            pass
        finally:
            self.quitting = True
            sys.settrace(None)

This is an issue as SyntaxError may come from two lines

• the call to compile() which means the code being run is not valid Python, in this case the current behaviour of PDB to exit is correct as there is nothing to debug
• the call to exec() in which case a SyntaxError can happen like in the report, and PDB should go in post mortem debug mode.

One way to fix the issue would be to catch the error in compile() and wrap it in a BdbSyntaxError so that PDB can differentiate between the two, another to keep BDB as it is and change PDB so that it compiles the code first, and call run() in a second step. I think the last one is better and will start writing a PR for this.

[remilapeyre]

This is related to bpo-16180, it may be possible to improve the situation by trying to determine whether the SyntaxError is in the file or came during its execution by looking at the filename but it's probably very brittle:

             # In most cases SystemExit does not warrant a post-mortem session.
             print("The program exited via sys.exit(). Exit status:", end=' ')
             print(sys.exc_info()[1])
-        except SyntaxError:
-            traceback.print_exc()
-            sys.exit(1)
-        except:
+        except Exception as e:
+            if (type(e) is SyntaxError and
+                e.filename == os.path.abspath(mainpyfile)):
+                traceback.print_exc()
+                sys.exit(1)
             traceback.print_exc()
             print("Uncaught exception. Entering post mortem debugging")
             print("Running 'cont' or 'step' will restart the program")

[xdegaye]

In Kerrick's example ast.literal_eval('') could be ast.literal_eval(some_code) instead where some_code is a string containing dynamically generated Python code. pdb post-mortem debugging must allow finding the syntax error in this code. The patch proposed in bpo-16180 by Terry may fix this problem (and bpo-16180).

[remilapeyre]

Yes, the patch by Terry Reedy fixes this issue while still breaking the loop from def f: pass.

It will start the debugger once for def f: pass which may be weird as in this case no user code has been executed and it will be in bdb which may confuse users:

Traceback (most recent call last):
  File "/Users/remi/src/cpython/Lib/pdb.py", line 1703, in main
    pdb._runscript(mainpyfile)
  File "/Users/remi/src/cpython/Lib/pdb.py", line 1572, in _runscript
    self.run(statement)
  File "/Users/remi/src/cpython/Lib/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/Users/remi/src/cpython/tests.py", line 1
    def f: pass
         ^
SyntaxError: invalid syntax
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> <string>(1)<module>()
(Pdb) bt
  /Users/remi/src/cpython/Lib/pdb.py(1703)main()
-> pdb._runscript(mainpyfile)
  /Users/remi/src/cpython/Lib/pdb.py(1572)_runscript()
-> self.run(statement)
  /Users/remi/src/cpython/Lib/bdb.py(580)run()
-> exec(cmd, globals, locals)
> <string>(1)<module>()

Perhaps we should should test whether the exception happened there and not drop in the debugger in that case?

[xdegaye]

Perhaps we should should test whether the exception happened there and not drop in the debugger in that case?

The same kind of problem occurs for any post-mortem debugging raised by an uncaught exception in the user code: the backtrace displayed by the 'bt' command shows frames that are owned by the pdb and bdb modules; and worse, the 'up' command allows to move to these frames. See for example below what happens when debugging foo.py that contains only the line "1/0". IMO this problem should be handled in another issue and in that case, in your example, 'bt' output would be empty.

$ python -m pdb foo.py
> /tmp/foo.py(1)<module>()
-> 1/0
(Pdb) c
Traceback (most recent call last):
  File "/usr/lib/python3.8/pdb.py", line 1703, in main
    pdb._runscript(mainpyfile)
  File "/usr/lib/python3.8/pdb.py", line 1572, in _runscript
    self.run(statement)
  File "/usr/lib/python3.8/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/tmp/foo.py", line 1, in <module>
    1/0
ZeroDivisionError: division by zero
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /tmp/foo.py(1)<module>()
-> 1/0
(Pdb) bt
  /usr/lib/python3.8/pdb.py(1703)main()
-> pdb._runscript(mainpyfile)
  /usr/lib/python3.8/pdb.py(1572)_runscript()
-> self.run(statement)
  /usr/lib/python3.8/bdb.py(580)run()
-> exec(cmd, globals, locals)
  <string>(1)<module>()
> /tmp/foo.py(1)<module>()
-> 1/0
(Pdb)

[gaogaotiantian]

Fixed in #110883