Feature idea for iPDB: follow context/cause chains

[jdtsmith]

One of the debug issues with code bases that adopt the style of raising (from) in try..except blocks is that the traceback chain becomes disjoint (the classic "The above exception was the direct cause of the following exception").

In practice this means iPDB can only go up and down the call stack of the "last" such exception thrown.

But it's pretty easy to walk along that chain, since exception values include a __context__ pointer. I wrote a simple magic %chain to do just this (more info in the doc string): https://gist.github.com/jdtsmith/7e8fb07a63b20681aee74b61897535dd

It simply alters sys.last_traceback, which is where iPDB starts. Works well. But I wonder if this wouldn't be better to implement in iPDB itself, e.g. a chain command there. Perhaps even just let the user connect to the next context chain when you run off the end of a traceback, using up/down. This might require restarting PDB under the hood (not sure), but would be a very nice addition.

[MrMino]

This is the reason for which I added pypa/pip#9428 back in the day, and wanted to make a command for it in ipdb, but I never figured out a good UX for it. Remember, that apart from __context__ there's also __cause__, and they point to different things. It's very easy to get lost in this while debugging, so something like where command for exception chain would be in order too.

Also, remember that you could just do ipdb.post_mortem(exc.__context__.__traceback__).

[jdtsmith]

Thanks, glad to hear there is some interest in the concept.

there's also __cause__

Right. But by investigation, I formed the impression that if __cause__ is included, __context__ always points to it (i.e. sys.last_value.__context__ is sys.last_value.__cause__). So following __context__ is enough. But maybe I'm not creative enough in nesting errors ;). Do you have an example handy where e.__cause__ is not e.__context__? I couldn't get this to happen.

In any case, it's easy to prefer __cause__ if it's there (as PEP 3134 dictates).

I never figured out a good UX for it. ... It's very easy to get lost in this while debugging, so something like where command for exception chain would be in order too.

Give %chain a try and see what you think? %chain -p gives quick orientation that serves like where for the chain, showing your place along the chain. The only issue with it currently is you have to exit and then re-enter iPDB to move along the chain and debug, which isn't ideal for breakpoints, etc. Would be preferable to have this happen inside iPDB.

Do you happen to know if you can substitute the traceback in a running PDB sessions cleanly?

[MrMino]

In any case, it's easy to prefer __cause__ if it's there (as PEP 3134 dictates).

Yes, that's what I was aiming at. While the API itself seems final, the PEP does not explicitly state that __context__ will always be there when __cause__ is. In order to be safe, one should be symmetric to how these attributes are treated by the language:

The chain of exceptions is traversed by following the cause and context attributes, with cause taking priority.

---

Do you happen to know if you can substitute the traceback in a running PDB sessions cleanly?

You can issue (i)pdb.post_mortem while running a post mortem, so that's something you could use. I don't know if you can do it dynamically, and to be honest - not sure if you'd want that.

---

There's also an issue in viewing current exception, which has its root cause in Python trying to prevent circular references:

> <ipython-input-10-0b2d5e58ec4c>(4)<cell line: 1>()
      1 try:
      2     raise Exception()
      3 except Exception as current_exception:
----> 4     import ipdb; ipdb.set_trace()
      5 

ipdb> current_exception
*** NameError: name 'current_exception' is not defined

I think it would be convenient to tackle this one too. You already need to find the exception for this to work anyway, so if you give the user an excval command (the exception sibling to retval), then it's gravy.

[jdtsmith]

Thanks for your thoughts here. I'm thinking mostly about the interface. It's inconvenient to leave an iPDB session to switch to another position along the __context__ chain, e.g. if you have breakpoints setup, etc. So it would be great for up/down to simply prompt you (or allow a flag) to moving to the next link on the chain, once you've traversed the TB. Do you think this is possible/desirable?

[MrMino]

I don't think you can do that in the context of a running trace per-se. Opening a post-mortem doesn't make you loose any variables nor breakpoints either. It just hides some of those things while you inspect the traceback - you can q your way back.

Something tells me this might be a good candidate for implementation inside bdb/pdb. That doesn't block us from implementing it here first though.

All that said, in its good old debugger-is-magic fashion, opening a post-mortem inside running session does weird things to the overal behavior:

In [18]: def n():                               
    ...:     try:                               
    ...:         try:                           
    ...:             0/0                        
    ...:         except Exception as e:         
    ...:             raise Exception("2") from e
    ...:     except Exception as e2:            
    ...:         print(repr(e2))                
    ...:                                        
    ...:                                    
In [19]: %debug n()  # s, then hit n until the second exception handler

...

   > <ipython-input-15-92adf36b14ff>(8)n()
      6             raise Exception("2") from e
      7     except Exception as e2:
----> 8         print(repr(e2))
      9
     10
ipdb> b 1
Breakpoint 1 at <ipython-input-15-92adf36b14ff>:1
ipdb> var = 123
ipdb> var
123
ipdb> import ipdb; ipdb.post_mortem(e2.__cause__.__traceback__)
> <ipython-input-15-92adf36b14ff>(4)n()
      1 def n():
      2     try:
      3         try:
----> 4             0/0
      5         except Exception as e:
      6             raise Exception("2") from e
      7     except Exception as e2:
      8         print(repr(e2))
      9
     10

ipdb> var
123
ipdb> b
ipdb> ll
      1 def n():
      2     try:
      3         try:
      4             0/0
      5         except Exception as e:
      6             raise Exception("2") from e
      7     except Exception as e2:
----> 8         print(repr(e2))
      9

# ↑ Weirdness ↑   
# Why does it remember linepos from the parent session?

ipdb> b
ipdb> q # This goes back to parent session
ipdb> b
Num Type         Disp Enb   Where
1   breakpoint   keep yes   at <ipython-input-15-92adf36b14ff>:1
ipdb> var
123
ipdb> q # This exits ipdb

In [20]:

[jdtsmith]

Very interesting; might be a bit hard to keep track of the recursion. I wonder if there is a way to replace the __traceback__ of the current, live post-mortem/debug?

Here's %chain -p, run from outside (with a slight tweak to throw an exception on function invocation):

In [5]: def n():
   try:
       try:
           0/0
       except Exception as e:
           raise Exception("2") from e
   except Exception as e2:
       raise Exception(f"Foo: {0/0}")

[jdtsmith]

How about this? Kind-of-sort-of works (complaining about f_globals on exit):

In [10]: n()
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Input In [5], in n()
      3 try:                           
----> 4     0/0                        
      5 except Exception as e:         

ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Exception                                 Traceback (most recent call last)
Input In [5], in n()
      5     except Exception as e:         
----> 6         raise Exception("2") from e
      7 except Exception as e2:            

Exception: 2

During handling of the above exception, another exception occurred:

ZeroDivisionError                         Traceback (most recent call last)
Input In [10], in <cell line: 1>()
----> 1 n()

Input In [5], in n()
      6         raise Exception("2") from e
      7 except Exception as e2:            
----> 8     raise Exception(f"Foo: {0/0}")

ZeroDivisionError: division by zero

In [11]: %debug
> <ipython-input-5-0a9a7f152f0a>(8)n()
      5         except Exception as e:
      6             raise Exception("2") from e
      7     except Exception as e2:
----> 8         raise Exception(f"Foo: {0/0}")
      9 

ipdb> import sys; get_ipython().InteractiveTB.pdb.interaction(None, sys.last_value.__context__.__traceback__)
> <ipython-input-5-0a9a7f152f0a>(6)n()
      4             0/0
      5         except Exception as e:
----> 6             raise Exception("2") from e
      7     except Exception as e2:
      8         raise Exception(f"Foo: {0/0}")

ipdb> import sys; get_ipython().InteractiveTB.pdb.interaction(None, sys.last_value.__context__.__context__.__traceb
ack__)
> <ipython-input-5-0a9a7f152f0a>(4)n()
      2     try:
      3         try:
----> 4             0/0
      5         except Exception as e:
      6             raise Exception("2") from e