Fix up through generators in postmortem debugger; closes #6251

[rhelmot]

By passing the lowest traceback element into the debugger, we require it to use frame.f_back to find older frames. However, f_back is always None for generator frames in the postmortem context. By providing a higher-up traceback element, pdb can traverse down the traceback.tb_next links, which do work correctly across generator calls.

...or at least, that's my understanding of the situation after spending half an hour researching the issue. This fix seems to work for my use cases, but I don't really understand the depths of pdb and can't tell if this may cause adverse affects elsewhere.

(I would also love to write a test for this, but I have no idea if we can or want to drive the debugger automatically for tests?)

[rhelmot]

...any updates on this? Is there something I can provide to help move this along?

[Carreau]

Sorry we missed the pull-request. I will have a look. Marking as 7.0 to make sure to have a look at it.

[Carreau]

I believe most of the debugger test are in IPython/core/tests/test_debugger.py are doctests, and ran through a subprocess. Not 10% sure about that though.

[Carreau]

Ok I was wrong for above, the only way I can see is to use subprocess and run python -i and check the output. Something similar to IPython/terminal/tests/test_embed.py

I'll see if I can write a scaffold for that.

[Carreau]

Ok, I'm writing some test, but this seem to not be quite right:

In [1]: def f(x):
   ...:     raise Exception

In [2]: gen = (f(x) for x in [0])

In [3]: for x in gen:
   ...:     pass
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-3-5b2155f9c4b2> in <module>
----> 1 for x in gen:
      2     pass

<ipython-input-2-b7f6c0ea3068> in <genexpr>(.0)
----> 1 gen = (f(x) for x in [0])

<ipython-input-1-a13efb0dffe1> in f(x)
      1 def f(x):
----> 2     raise Exception

Exception:


In [4]: %debug
> <ipython-input-1-a13efb0dffe1>(2)f()
      1 def f(x):
----> 2     raise Exception


ipdb> u
> <ipython-input-2-b7f6c0ea3068>(1)<genexpr>()
----> 1 gen = (f(x) for x in [0])

ipdb> u
> <ipython-input-3-5b2155f9c4b2>(1)<module>()
----> 1 for x in gen:
      2     pass

ipdb> u                                         |
> <ipython-input-1-a13efb0dffe1>(2)f()          |
      1 def f(x):                               |
----> 2     raise Exception                     | These two frames are wrong.
                                                |
ipdb> u                                         |
> <ipython-input-2-b7f6c0ea3068>(1)<genexpr>()  |
----> 1 gen = (f(x) for x in [0])               |
                                                
ipdb> u
*** Oldest frame

It show frames in the wrong place (or re-go through frames it should not)

I'm going to try to polish the tests framework to test that, and we can figure out what to do. I'm going to likely push this thing for 7.1

[rhelmot]

Okay, I read the pdb/bdb source more carefully and force pushed what should be a real fix. Explanation in the commit message.

The takeaway here for future reference is that the only place the frame passed into interaction is used is in Bdb.get_stack, which walks the stack backwards from the frame and forwards from the traceback to construct the full list of stack frames. The frame can be None and this will still work, but if the traceback and frame don't correspond to the same frame, it will duplicate parts of the stack.

[Carreau]

Thanks, I'll have a look, thy to commit a test framework (or at least a draft of it).

[Carreau]

Ok, I just pushed what should be test for this patch.

It's a bit cumbersome to write, as you will see, but let's not optimize until we actually have need to write this often.

[Carreau]

Once test passes i'll merge.
Many thanks for your patience.

@meeseeksdev[bot] untag help wanted
@meeseeksdev[bot] tag hacktoberfest

[Carreau]

Tests are passing, let's get that in, many thanks !