Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore SIGINT when in a debugger REPL #165

Merged
merged 100 commits into from
Aug 3, 2022
Merged

Ignore SIGINT when in a debugger REPL #165

merged 100 commits into from
Aug 3, 2022

Commits on Jul 27, 2022

  1. Configuration menu
    Copy the full SHA
    3cac323 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6b7b583 View commit details
    Browse the repository at this point in the history
  3. Refine the handler for child vs. root cases

    This gets very close to avoiding any possible hangs to do with tty
    locking and SIGINT handling minus a special case that will be detailed
    below.
    
    Summary of implementation changes:
    
    - convert `_mk_pdb()` -> `with _open_pdb() as pdb:` which implicitly
      handles the `bdb.BdbQuit` case such that debugger teardown hooks are
      always called.
    - rename the handler to `shield_sigint()` and handle a variety of new
      cases:
      * the root is in debug but hasn't been cancelled -> call
        `Actor.cancel_soon()`
      * the root is in debug but *has* been called (`Actor.cancel_soon()`
        already called) -> raise KBI
      * a child is in debug *and* has a task locking the debugger -> ignore
        SIGINT in child *and* the root actor.
    - if the debugger instance is provided to the handler at acquire time,
      on SIGINT handling completion re-print the last pdb++ REPL output so
      that the user realizes they are still actively in debug.
    - ignore the unlock case where a race condition of "no task" holding the
      lock causes the `RuntimeError` normally associated with the "wrong
      task" doing so (not sure if this is a `trio` bug?).
    - change debug logs to runtime level.
    
    Unhandled case(s):
    
    - a child is maybe in debug mode but does not itself have any task using
      the debugger.
        * ToDo: we need a way to decide what to do with
          "intermediate" child actors who themselves either are not in
          `debug_mode=True` but have children who *are* such that a SIGINT
          won't cause cancellation of that child-as-parent-of-another-child
          **iff** any of their children are in in debug mode.
    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    4e60c17 View commit details
    Browse the repository at this point in the history
  4. Make mypy happy

    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    345573e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    42f9d10 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    e519526 View commit details
    Browse the repository at this point in the history
  7. Fix example name typo

    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    99c4319 View commit details
    Browse the repository at this point in the history
  8. Try overriding _GeneratorContextManager.__exit__(); didn't work..

    Using either of `@pdb.hideframe` or `__tracebackhide__` on stdlib
    methods doesn't seem to work either.. This all seems to have something
    to do with async generator usage I think ?
    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    7964a9f View commit details
    Browse the repository at this point in the history
  9. Drop all the @cm.__exit__() override attempts..

    None of it worked (you still will see `.__exit__()` frames on debugger
    entry - you'd think this would have been solved by now but, shrug) so
    instead wrap the debugger entry-point in a `try:` and put the SIGINT
    handler restoration inside `MultiActorPdb` teardown hooks.
    
    This seems to restore the UX as it was prior but with also giving the
    desired SIGINT override handler behaviour.
    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    aea8f63 View commit details
    Browse the repository at this point in the history
  10. A .open_context() example that causes a hang!

    Finally! I think this may be the root issue we've been seeing in
    production in a client project.
    
    No idea yet why this is happening but the fault-causing sequence seems
    to be:
    - `.open_context()` in a child actor
    - enter the debugger via `tractor.breakpoint()`
    - continue from that entry via `c` command in REPL
    - raise an error just after inside the context task's body
    
    Looking at logging it appears as though the child thinks it has the tty
    but no input is accepted on the REPL and a further `ctrl-c` results in
    some teardown but also a further hang where both parent and child become
    unresponsive..
    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    21dccb2 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    8f4bbf1 View commit details
    Browse the repository at this point in the history
  12. Add notes around py3.10 stdlib bug from pdb++

    There's a bug that's triggered in the stdlib without latest `pdb++`
    installed; add a note for that.
    
    Further inside `wait_for_parent_stdin_hijack()` don't `.started()` until
    the interactor stream has been opened to avoid races when debugging this
    `._debug.py` module (at the least) since we usually don't want the
    spawning (parent) task to resume until we know for sure the tty lock has
    been acquired. Also, drop the random checkpoint we had inside
    `_breakpoint()`, not sure it was actually adding anything useful since
    we're (mostly) carefully shielded throughout this func.
    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    8892204 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    74b819a View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    bb732ce View commit details
    Browse the repository at this point in the history
  15. Only cancel/get-result from a ctx if transport is up

    There's no point in sending a cancel message to the remote linked task
    and especially no reason to block waiting on a result from that task if
    the transport layer is detected to be disconnected. We expect that the
    transport shouldn't go down at the layer of the message loop
    (reconnection logic should be handled in the transport layer itself) so
    if we detect the channel is not connected we don't bother requesting
    cancels nor waiting on a final result message.
    
    Why?
    
    - if the connection goes down in error the caller side won't have a way
      to know "how long" it should block to wait for a cancel ack or result
      and causes a potential hang that may require an additional ctrl-c from
      the user especially if using the debugger or if the traceback is not
      seen on console.
    - obviously there's no point in waiting for messages when there's no
      transport to deliver them XD
    
    Further, add some more detailed cancel logging detailing the task and
    actor ids.
    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    bf0ac31 View commit details
    Browse the repository at this point in the history
  16. Make Actor._process_messages() report disconnects

    The method now returns a `bool` which flags whether the transport died
    to the caller and allows for reporting a disconnect in the
    channel-transport handler task. This is something a user will normally
    want to know about on the caller side especially after seeing
    a traceback from the peer (if in tree) on console.
    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    206c7c0 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    41924c8 View commit details
    Browse the repository at this point in the history
  18. Type annot updates

    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    f2671ed View commit details
    Browse the repository at this point in the history
  19. Avoid attr error XD

    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    2819b6a View commit details
    Browse the repository at this point in the history
  20. Pre-declare disconnected flag

    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    89b44f8 View commit details
    Browse the repository at this point in the history
  21. Add back in async gen loop

    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    dd23e78 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    fe0fd1a View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    4fd924c View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    7bb5add View commit details
    Browse the repository at this point in the history
  25. Just warn on IPC breaks

    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    4be13b7 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    0062c96 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    d47d0e7 View commit details
    Browse the repository at this point in the history
  28. Always propagate SIGINT when no locking peer found

    A hopefully significant fix here is to always avoid suppressing a SIGINT
    when the root actor can not detect an active IPC connections (via
    a connected channel) to the supposed debug lock holding actor. In that
    case it is most likely that the actor has either terminated or has lost
    its connection for debugger control and there is no way the root can
    verify the lock is in use; thus we choose to allow KBI cancellation.
    
    Drop the (by comment) `try`-`finally` block in
    `_hijoack_stdin_for_child()` around the `_acquire_debug_lock()` call
    since all that logic should now be handled internal to that locking
    manager. Try to catch a weird error around the `.do_longlist()` method
    call that seems to sometimes break on py3.10 and latest `pdbpp`.
    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    deaca7d View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    c7035be View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    418e74e View commit details
    Browse the repository at this point in the history
  31. Readme formatting tweaks

    goodboy committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    2f5a604 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2022

  1. Always undo SIGINT overrides, cancel detached children

    Ensure that even when `pdb` resumption methods are called during a crash
    where `trio`'s runtime has already terminated (eg. `Event.set()` will
    raise) we always revert our sigint handler to the original. Further
    inside the handler if we hit a case where a child is in debug and
    (thinks it) has the global pdb lock, if it has no IPC connection to
    a parent, simply presume tty sync-coordination is now lost and cancel
    the child immediately.
    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    f07e9db View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b29def8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e2453fd View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2a61aa0 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    201c026 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    18c525d View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    439d320 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    abb0053 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    ff3f595 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    56c1909 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    519f4c3 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    4e08605 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    d0dcd55 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    a90ca4b View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    9bc38cb View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    bd7d507 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    8b9f342 View commit details
    Browse the repository at this point in the history
  18. Pin to trio >= 0.20

    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    19fb77f View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    64909e6 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    4dcc212 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    b9eb601 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    925d5c1 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    56b30a9 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    70ad0f6 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    8358361 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    a101971 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    ef8dc02 View commit details
    Browse the repository at this point in the history
  28. Test: drop expect prompt

    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    a723501 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    dadd5e6 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    617d57d View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    ba7b355 View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    a2e9019 View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    adbebd3 View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    6bdcbdb View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    4779bad View commit details
    Browse the repository at this point in the history
  36. Always consider the debugger when exiting contexts

    When in an uncertain teardown state and in debug mode a context can be
    popped from actor runtime before a child finished debugging (the case
    when the parent is tearing down but the child hasn't closed/completed
    its tty lock IPC exit phase) and the child sends the "stop" message to
    unlock the debugger but it's ignored bc the parent has already dropped
    the ctx. Instead we call `._debug.maybe_wait_for_deugger()` before these
    context removals to avoid the root getting stuck thinking the lock was
    never released.
    
    Further, add special `Actor._cancel_task()` handling code inside
    `_invoke()` which continues to execute the method despite the IPC
    channel to the caller being broken and thus avoiding potential hangs due
    to a target (child) actor task remaining alive.
    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    b21f2e1 View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    808d7ae View commit details
    Browse the repository at this point in the history
  38. Configuration menu
    Copy the full SHA
    cb0c47c View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    bd362a0 View commit details
    Browse the repository at this point in the history
  40. Factor lock-state release logic into helper

    The common logic to both remove our custom SIGINT handler as well
    as signal the actor global event that pdb is complete. Call this
    whenever we exit a post mortem call and thus any time some rpc task
    get's debugged inside `._actor._invoke()`.
    
    Further, we have to manually print the REPL prompt on 3.9 for some wack
    reason, so stick a version guard in the sigint handler for that..
    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    b01daa5 View commit details
    Browse the repository at this point in the history
  41. Drop ctlc tests on Py3.9...

    After many tries I just don't think it's worth it to make the tests work
    since the repl UX in `pdbpp` is so unreliable in the latest release and
    honestly we're trying to go 3.10+ ASAP.
    
    Further,
    - entirely drop the pattern matching inside the `do_ctlc()` for now.
    - add a `subactor_error` parametrization that catches a case that
      previously caused a hang (when you use 'next' immediately after the
      first crash/debug lock (the fix was pushed just before this commit).
    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    a4538a3 View commit details
    Browse the repository at this point in the history
  42. Configuration menu
    Copy the full SHA
    c0cd99e View commit details
    Browse the repository at this point in the history
  43. Configuration menu
    Copy the full SHA
    1d4d55f View commit details
    Browse the repository at this point in the history
  44. Configuration menu
    Copy the full SHA
    20c660f View commit details
    Browse the repository at this point in the history
  45. Configuration menu
    Copy the full SHA
    a4bac13 View commit details
    Browse the repository at this point in the history
  46. Avoid infinite wait for EOF

    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    457499b View commit details
    Browse the repository at this point in the history
  47. Configuration menu
    Copy the full SHA
    6f01c78 View commit details
    Browse the repository at this point in the history
  48. Configuration menu
    Copy the full SHA
    5e23b3c View commit details
    Browse the repository at this point in the history
  49. Configuration menu
    Copy the full SHA
    08cf03c View commit details
    Browse the repository at this point in the history
  50. Configuration menu
    Copy the full SHA
    91f034a View commit details
    Browse the repository at this point in the history
  51. Configuration menu
    Copy the full SHA
    937ed99 View commit details
    Browse the repository at this point in the history
  52. Try less times for EOF

    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    87b2ccb View commit details
    Browse the repository at this point in the history
  53. Configuration menu
    Copy the full SHA
    8896ba2 View commit details
    Browse the repository at this point in the history
  54. Configuration menu
    Copy the full SHA
    aca9a6b View commit details
    Browse the repository at this point in the history
  55. Configuration menu
    Copy the full SHA
    acfbae4 View commit details
    Browse the repository at this point in the history
  56. Configuration menu
    Copy the full SHA
    a9aaee9 View commit details
    Browse the repository at this point in the history
  57. Configuration menu
    Copy the full SHA
    e4771ee View commit details
    Browse the repository at this point in the history
  58. Configuration menu
    Copy the full SHA
    c5c7a90 View commit details
    Browse the repository at this point in the history
  59. Configuration menu
    Copy the full SHA
    54de72d View commit details
    Browse the repository at this point in the history
  60. Configuration menu
    Copy the full SHA
    fa43888 View commit details
    Browse the repository at this point in the history
  61. Put pygments back to default

    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    02c3b9a View commit details
    Browse the repository at this point in the history
  62. Configuration menu
    Copy the full SHA
    8115759 View commit details
    Browse the repository at this point in the history
  63. Configuration menu
    Copy the full SHA
    2d387f2 View commit details
    Browse the repository at this point in the history
  64. Configuration menu
    Copy the full SHA
    7f6169a View commit details
    Browse the repository at this point in the history
  65. Configuration menu
    Copy the full SHA
    e4006da View commit details
    Browse the repository at this point in the history
  66. Configuration menu
    Copy the full SHA
    650313d View commit details
    Browse the repository at this point in the history
  67. Add nooz

    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    65540f3 View commit details
    Browse the repository at this point in the history
  68. Configuration menu
    Copy the full SHA
    8f1fe23 View commit details
    Browse the repository at this point in the history
  69. List deps in CI

    goodboy committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    cc5f60b View commit details
    Browse the repository at this point in the history