except statement block incorrectly assumes end of scope(?).

[SaimRaza]

BPO	36537
Nosy	@warsaw, @xdegaye

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-04-05.18:43:36.195>
labels = ['3.7', 'type-bug', 'library']
title = 'except statement block incorrectly assumes end of scope(?).'
updated_at = <Date 2019-04-12.22:24:17.161>
user = 'https://bugs.python.org/SaimRaza'

bugs.python.org fields:

activity = <Date 2019-04-12.22:24:17.161>
actor = 'Saim Raza'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-04-05.18:43:36.195>
creator = 'Saim Raza'
dependencies = []
files = []
hgrepos = []
issue_num = 36537
keywords = []
message_count = 12.0
messages = ['339510', '339517', '339542', '339550', '339566', '339577', '340033', '340035', '340036', '340043', '340114', '340117']
nosy_count = 4.0
nosy_names = ['barry', 'SilentGhost', 'xdegaye', 'Saim Raza']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36537'
versions = ['Python 3.7']

[SaimRaza]

If pdb.set_trace() is the last statement in the first code snippet, variable 'err' is (wrongly?) excluded from locals(). Adding any code after the pdb.set_trace() statement makes 'err' available in locals.

In [2]: try:
...: raise ValueError("I am ValueError")
...: except ValueError as err:
...: print("err" in locals())
...: import pdb; pdb.set_trace()
...:
True
--Return--

<ipython-input-2-9d1238ac9b7a>(5)<module>()->None
-> import pdb; pdb.set_trace()
(Pdb) print("err" in locals())
False <-------------- BUG??
(Pdb) c

In [3]: try:
...: raise ValueError("I am ValueError")
...: except ValueError as err:
...: print("err" in locals())
...: import pdb; pdb.set_trace()
...: import os # Dummy code - makes variable 'err' available inside the debugger.
...:
True

<ipython-input-3-3cefa7fe6ddd>(6)<module>()->None
-> import os # Dummy code - makes variable err available inside debugger
(Pdb) print("err" in locals())
True

In [4]: sys.version_info
Out[4]: sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)

FTR, the variable 'err' is available in both cases in case of Python 2.7. Also, this happens with ipdb as well.

Please note that I am aware that I need to assign the variable 'err' to some variable inside the except block to access it outside the except block. However, the pdb statement is still inside the except block in both cases.

[SilentGhost]

Can confirm also on 3.6, but this seems to be something particular to set_trace, when pdb is entered via post_mortem, the variable is available in locals().

[xdegaye]

This is not a bug.
On the first pdb session from the OP:

True
--Return--

<ipython-input-2-9d1238ac9b7a>(5)<module>()->None
-> import pdb; pdb.set_trace()
(Pdb) print("err" in locals())
False <-------------- BUG??

Pdb has stopped on a '--Return--' line event, so we are already outside the scope of the except clause and the 'err' local variable cannot be accessed outside the except block. The following code snippet shows that pdb stops at the line following the set_trace() call:

==================

def foo():
    try:
        1/0
    except Exception as err:
        import pdb; pdb.set_trace()
    x= 1

foo()

==================

And the corresponding pdb session:

$ python foo.py
> /path/to/foo.py(6)foo()
-> x= 1
(Pdb) err
*** NameError: name 'err' is not defined
(Pdb)

[SaimRaza]

https://docs.python.org/3/reference/compound_stmts.html#the-try-statement says

=============

except E as N:
    foo

=============

is converted to the following statement:

=============

except E as N:
    try:
        foo
    finally:
        del N

=============

In the examples in this thread, foo is 'import pdb; pdb.set_trace()'. So, I was expecting 'del N' to be executed only after executing foo. This implies that err should have been available in the scope.

Further, isn't the code execution supposed to be stopped at the set_trace() line and not on the next line. Should it at least respect the scope (indentation) and not execute the '--Return--' line event until the user actually executes the next line (x=1 in your example)?

[xdegaye]

@saim wrote

https://docs.python.org/3/reference/compound_stmts.html#the-try-statement says

This is correct, the disassembly of foo.py below demonstrates that point: BEGIN_FINALLY at bytecode index 36 starts the implementation of the 'finally' clause that deletes 'err'. But pdb is a line debugger, not a debugger at the bytecode level, so when 'foo = 1' is replaced by 'import pdb; pdb.set_trace()', the first line where pdb may stop is at line 6 and since foo() last line is line 5, pdb stops at the 'return' event for this function.

======= foo.py ========

def foo():
    try:
        1/0
    except Exception as err:
        foo = 1

import dis
dis.dis(foo)

===========================
2 0 SETUP_FINALLY 12 (to 14)

3 2 LOAD_CONST 1 (1)
4 LOAD_CONST 2 (0)
6 BINARY_TRUE_DIVIDE
8 POP_TOP
10 POP_BLOCK
12 JUMP_FORWARD 38 (to 52)

4 >> 14 DUP_TOP
16 LOAD_GLOBAL 0 (Exception)
18 COMPARE_OP 10 (exception match)
20 POP_JUMP_IF_FALSE 50
22 POP_TOP
24 STORE_FAST 0 (err)
26 POP_TOP
28 SETUP_FINALLY 8 (to 38)

5 30 LOAD_CONST 1 (1)
32 STORE_FAST 1 (foo)
34 POP_BLOCK
36 BEGIN_FINALLY
>> 38 LOAD_CONST 0 (None)
40 STORE_FAST 0 (err)
42 DELETE_FAST 0 (err)
44 END_FINALLY
46 POP_EXCEPT
48 JUMP_FORWARD 2 (to 52)
>> 50 END_FINALLY
>> 52 LOAD_CONST 0 (None)
54 RETURN_VALUE
===========================

[xdegaye]

The *** Tracing *** section [1] of Objects/lnotab_notes.txt in the cpython repository provides detailed explanations for when the line trace function is called.

[1]

cpython/Objects/lnotab_notes.txt

Line 63 in 9d7b2c0

*** Tracing ***

[SaimRaza]

This is pretty unintuitive from a user's stand point. Now, I need to *inconveniently* put some dummy code after the ser_trace call every time I want to access the exception inside the except block.

Also, this is a change in behavior from Python 2.7. Is this documented somewhere?

[SilentGhost]

Saim, a .post_mortem could be used instead. As I noted, it works just fine.

[SaimRaza]

Thanks, SilentGhost! However, should we try to fix set_trace as well to avoid hassles to other users?

[SilentGhost]

I cannot imagine that the fix would be straightforward or that there is much use of this particular pattern. Perhaps, a note in the docs suggesting post_mortem() for except clauses over set_trace() would be more appropriate.

[xdegaye]

Now, I need to *inconveniently* put some dummy code after the ser_trace call every time I want to access the exception inside the except block

Knowing that pdb stops just before the interpreter executes the line after pdb.set_trace(), what prevents you from writing the pdb.set_trace() statement before the last line of the except clause ?

[SaimRaza]

Knowing that pdb stops just before the interpreter executes the line after pdb.set_trace(), what prevents you from writing the pdb.set_trace() statement before the last line of the except clause ?

Of course, that can be done. The issue here is to get the correct and expected behavior from pdb/ipdb.

On a similar note, users might want to put just the set_trace() call and no other code in the block while writing the code or during debugging the exception caught.