-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ignore SIGINT when in a debugger REPL #165
Commits on Jul 27, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 3cac323 - Browse repository at this point
Copy the full SHA 3cac323View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6b7b583 - Browse repository at this point
Copy the full SHA 6b7b583View commit details -
Refine the handler for child vs. root cases
This gets very close to avoiding any possible hangs to do with tty locking and SIGINT handling minus a special case that will be detailed below. Summary of implementation changes: - convert `_mk_pdb()` -> `with _open_pdb() as pdb:` which implicitly handles the `bdb.BdbQuit` case such that debugger teardown hooks are always called. - rename the handler to `shield_sigint()` and handle a variety of new cases: * the root is in debug but hasn't been cancelled -> call `Actor.cancel_soon()` * the root is in debug but *has* been called (`Actor.cancel_soon()` already called) -> raise KBI * a child is in debug *and* has a task locking the debugger -> ignore SIGINT in child *and* the root actor. - if the debugger instance is provided to the handler at acquire time, on SIGINT handling completion re-print the last pdb++ REPL output so that the user realizes they are still actively in debug. - ignore the unlock case where a race condition of "no task" holding the lock causes the `RuntimeError` normally associated with the "wrong task" doing so (not sure if this is a `trio` bug?). - change debug logs to runtime level. Unhandled case(s): - a child is maybe in debug mode but does not itself have any task using the debugger. * ToDo: we need a way to decide what to do with "intermediate" child actors who themselves either are not in `debug_mode=True` but have children who *are* such that a SIGINT won't cause cancellation of that child-as-parent-of-another-child **iff** any of their children are in in debug mode.
Configuration menu - View commit details
-
Copy full SHA for 4e60c17 - Browse repository at this point
Copy the full SHA 4e60c17View commit details -
Configuration menu - View commit details
-
Copy full SHA for 345573e - Browse repository at this point
Copy the full SHA 345573eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 42f9d10 - Browse repository at this point
Copy the full SHA 42f9d10View commit details -
Configuration menu - View commit details
-
Copy full SHA for e519526 - Browse repository at this point
Copy the full SHA e519526View commit details -
Configuration menu - View commit details
-
Copy full SHA for 99c4319 - Browse repository at this point
Copy the full SHA 99c4319View commit details -
Try overriding
_GeneratorContextManager.__exit__()
; didn't work..Using either of `@pdb.hideframe` or `__tracebackhide__` on stdlib methods doesn't seem to work either.. This all seems to have something to do with async generator usage I think ?
Configuration menu - View commit details
-
Copy full SHA for 7964a9f - Browse repository at this point
Copy the full SHA 7964a9fView commit details -
Drop all the
@cm.__exit__()
override attempts..None of it worked (you still will see `.__exit__()` frames on debugger entry - you'd think this would have been solved by now but, shrug) so instead wrap the debugger entry-point in a `try:` and put the SIGINT handler restoration inside `MultiActorPdb` teardown hooks. This seems to restore the UX as it was prior but with also giving the desired SIGINT override handler behaviour.
Configuration menu - View commit details
-
Copy full SHA for aea8f63 - Browse repository at this point
Copy the full SHA aea8f63View commit details -
A
.open_context()
example that causes a hang!Finally! I think this may be the root issue we've been seeing in production in a client project. No idea yet why this is happening but the fault-causing sequence seems to be: - `.open_context()` in a child actor - enter the debugger via `tractor.breakpoint()` - continue from that entry via `c` command in REPL - raise an error just after inside the context task's body Looking at logging it appears as though the child thinks it has the tty but no input is accepted on the REPL and a further `ctrl-c` results in some teardown but also a further hang where both parent and child become unresponsive..
Configuration menu - View commit details
-
Copy full SHA for 21dccb2 - Browse repository at this point
Copy the full SHA 21dccb2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8f4bbf1 - Browse repository at this point
Copy the full SHA 8f4bbf1View commit details -
Add notes around py3.10 stdlib bug from
pdb++
There's a bug that's triggered in the stdlib without latest `pdb++` installed; add a note for that. Further inside `wait_for_parent_stdin_hijack()` don't `.started()` until the interactor stream has been opened to avoid races when debugging this `._debug.py` module (at the least) since we usually don't want the spawning (parent) task to resume until we know for sure the tty lock has been acquired. Also, drop the random checkpoint we had inside `_breakpoint()`, not sure it was actually adding anything useful since we're (mostly) carefully shielded throughout this func.
Configuration menu - View commit details
-
Copy full SHA for 8892204 - Browse repository at this point
Copy the full SHA 8892204View commit details -
Configuration menu - View commit details
-
Copy full SHA for 74b819a - Browse repository at this point
Copy the full SHA 74b819aView commit details -
Configuration menu - View commit details
-
Copy full SHA for bb732ce - Browse repository at this point
Copy the full SHA bb732ceView commit details -
Only cancel/get-result from a ctx if transport is up
There's no point in sending a cancel message to the remote linked task and especially no reason to block waiting on a result from that task if the transport layer is detected to be disconnected. We expect that the transport shouldn't go down at the layer of the message loop (reconnection logic should be handled in the transport layer itself) so if we detect the channel is not connected we don't bother requesting cancels nor waiting on a final result message. Why? - if the connection goes down in error the caller side won't have a way to know "how long" it should block to wait for a cancel ack or result and causes a potential hang that may require an additional ctrl-c from the user especially if using the debugger or if the traceback is not seen on console. - obviously there's no point in waiting for messages when there's no transport to deliver them XD Further, add some more detailed cancel logging detailing the task and actor ids.
Configuration menu - View commit details
-
Copy full SHA for bf0ac31 - Browse repository at this point
Copy the full SHA bf0ac31View commit details -
Make
Actor._process_messages()
report disconnectsThe method now returns a `bool` which flags whether the transport died to the caller and allows for reporting a disconnect in the channel-transport handler task. This is something a user will normally want to know about on the caller side especially after seeing a traceback from the peer (if in tree) on console.
Configuration menu - View commit details
-
Copy full SHA for 206c7c0 - Browse repository at this point
Copy the full SHA 206c7c0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 41924c8 - Browse repository at this point
Copy the full SHA 41924c8View commit details -
Configuration menu - View commit details
-
Copy full SHA for f2671ed - Browse repository at this point
Copy the full SHA f2671edView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2819b6a - Browse repository at this point
Copy the full SHA 2819b6aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 89b44f8 - Browse repository at this point
Copy the full SHA 89b44f8View commit details -
Configuration menu - View commit details
-
Copy full SHA for dd23e78 - Browse repository at this point
Copy the full SHA dd23e78View commit details -
Configuration menu - View commit details
-
Copy full SHA for fe0fd1a - Browse repository at this point
Copy the full SHA fe0fd1aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4fd924c - Browse repository at this point
Copy the full SHA 4fd924cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7bb5add - Browse repository at this point
Copy the full SHA 7bb5addView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4be13b7 - Browse repository at this point
Copy the full SHA 4be13b7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0062c96 - Browse repository at this point
Copy the full SHA 0062c96View commit details -
Configuration menu - View commit details
-
Copy full SHA for d47d0e7 - Browse repository at this point
Copy the full SHA d47d0e7View commit details -
Always propagate SIGINT when no locking peer found
A hopefully significant fix here is to always avoid suppressing a SIGINT when the root actor can not detect an active IPC connections (via a connected channel) to the supposed debug lock holding actor. In that case it is most likely that the actor has either terminated or has lost its connection for debugger control and there is no way the root can verify the lock is in use; thus we choose to allow KBI cancellation. Drop the (by comment) `try`-`finally` block in `_hijoack_stdin_for_child()` around the `_acquire_debug_lock()` call since all that logic should now be handled internal to that locking manager. Try to catch a weird error around the `.do_longlist()` method call that seems to sometimes break on py3.10 and latest `pdbpp`.
Configuration menu - View commit details
-
Copy full SHA for deaca7d - Browse repository at this point
Copy the full SHA deaca7dView commit details -
Configuration menu - View commit details
-
Copy full SHA for c7035be - Browse repository at this point
Copy the full SHA c7035beView commit details -
Configuration menu - View commit details
-
Copy full SHA for 418e74e - Browse repository at this point
Copy the full SHA 418e74eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2f5a604 - Browse repository at this point
Copy the full SHA 2f5a604View commit details
Commits on Aug 2, 2022
-
Always undo SIGINT overrides, cancel detached children
Ensure that even when `pdb` resumption methods are called during a crash where `trio`'s runtime has already terminated (eg. `Event.set()` will raise) we always revert our sigint handler to the original. Further inside the handler if we hit a case where a child is in debug and (thinks it) has the global pdb lock, if it has no IPC connection to a parent, simply presume tty sync-coordination is now lost and cancel the child immediately.
Configuration menu - View commit details
-
Copy full SHA for f07e9db - Browse repository at this point
Copy the full SHA f07e9dbView commit details -
Configuration menu - View commit details
-
Copy full SHA for b29def8 - Browse repository at this point
Copy the full SHA b29def8View commit details -
Configuration menu - View commit details
-
Copy full SHA for e2453fd - Browse repository at this point
Copy the full SHA e2453fdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2a61aa0 - Browse repository at this point
Copy the full SHA 2a61aa0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 201c026 - Browse repository at this point
Copy the full SHA 201c026View commit details -
Configuration menu - View commit details
-
Copy full SHA for 18c525d - Browse repository at this point
Copy the full SHA 18c525dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 439d320 - Browse repository at this point
Copy the full SHA 439d320View commit details -
Configuration menu - View commit details
-
Copy full SHA for abb0053 - Browse repository at this point
Copy the full SHA abb0053View commit details -
Configuration menu - View commit details
-
Copy full SHA for ff3f595 - Browse repository at this point
Copy the full SHA ff3f595View commit details -
Configuration menu - View commit details
-
Copy full SHA for 56c1909 - Browse repository at this point
Copy the full SHA 56c1909View commit details -
Configuration menu - View commit details
-
Copy full SHA for 519f4c3 - Browse repository at this point
Copy the full SHA 519f4c3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4e08605 - Browse repository at this point
Copy the full SHA 4e08605View commit details -
Configuration menu - View commit details
-
Copy full SHA for d0dcd55 - Browse repository at this point
Copy the full SHA d0dcd55View commit details -
Configuration menu - View commit details
-
Copy full SHA for a90ca4b - Browse repository at this point
Copy the full SHA a90ca4bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9bc38cb - Browse repository at this point
Copy the full SHA 9bc38cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for bd7d507 - Browse repository at this point
Copy the full SHA bd7d507View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8b9f342 - Browse repository at this point
Copy the full SHA 8b9f342View commit details -
Configuration menu - View commit details
-
Copy full SHA for 19fb77f - Browse repository at this point
Copy the full SHA 19fb77fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 64909e6 - Browse repository at this point
Copy the full SHA 64909e6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4dcc212 - Browse repository at this point
Copy the full SHA 4dcc212View commit details -
Configuration menu - View commit details
-
Copy full SHA for b9eb601 - Browse repository at this point
Copy the full SHA b9eb601View commit details -
Configuration menu - View commit details
-
Copy full SHA for 925d5c1 - Browse repository at this point
Copy the full SHA 925d5c1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 56b30a9 - Browse repository at this point
Copy the full SHA 56b30a9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 70ad0f6 - Browse repository at this point
Copy the full SHA 70ad0f6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8358361 - Browse repository at this point
Copy the full SHA 8358361View commit details -
Configuration menu - View commit details
-
Copy full SHA for a101971 - Browse repository at this point
Copy the full SHA a101971View commit details -
Configuration menu - View commit details
-
Copy full SHA for ef8dc02 - Browse repository at this point
Copy the full SHA ef8dc02View commit details -
Configuration menu - View commit details
-
Copy full SHA for a723501 - Browse repository at this point
Copy the full SHA a723501View commit details -
Configuration menu - View commit details
-
Copy full SHA for dadd5e6 - Browse repository at this point
Copy the full SHA dadd5e6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 617d57d - Browse repository at this point
Copy the full SHA 617d57dView commit details -
Configuration menu - View commit details
-
Copy full SHA for ba7b355 - Browse repository at this point
Copy the full SHA ba7b355View commit details -
Configuration menu - View commit details
-
Copy full SHA for a2e9019 - Browse repository at this point
Copy the full SHA a2e9019View commit details -
Configuration menu - View commit details
-
Copy full SHA for adbebd3 - Browse repository at this point
Copy the full SHA adbebd3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6bdcbdb - Browse repository at this point
Copy the full SHA 6bdcbdbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4779bad - Browse repository at this point
Copy the full SHA 4779badView commit details -
Always consider the debugger when exiting contexts
When in an uncertain teardown state and in debug mode a context can be popped from actor runtime before a child finished debugging (the case when the parent is tearing down but the child hasn't closed/completed its tty lock IPC exit phase) and the child sends the "stop" message to unlock the debugger but it's ignored bc the parent has already dropped the ctx. Instead we call `._debug.maybe_wait_for_deugger()` before these context removals to avoid the root getting stuck thinking the lock was never released. Further, add special `Actor._cancel_task()` handling code inside `_invoke()` which continues to execute the method despite the IPC channel to the caller being broken and thus avoiding potential hangs due to a target (child) actor task remaining alive.
Configuration menu - View commit details
-
Copy full SHA for b21f2e1 - Browse repository at this point
Copy the full SHA b21f2e1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 808d7ae - Browse repository at this point
Copy the full SHA 808d7aeView commit details -
Configuration menu - View commit details
-
Copy full SHA for cb0c47c - Browse repository at this point
Copy the full SHA cb0c47cView commit details -
Configuration menu - View commit details
-
Copy full SHA for bd362a0 - Browse repository at this point
Copy the full SHA bd362a0View commit details -
Factor lock-state release logic into helper
The common logic to both remove our custom SIGINT handler as well as signal the actor global event that pdb is complete. Call this whenever we exit a post mortem call and thus any time some rpc task get's debugged inside `._actor._invoke()`. Further, we have to manually print the REPL prompt on 3.9 for some wack reason, so stick a version guard in the sigint handler for that..
Configuration menu - View commit details
-
Copy full SHA for b01daa5 - Browse repository at this point
Copy the full SHA b01daa5View commit details -
After many tries I just don't think it's worth it to make the tests work since the repl UX in `pdbpp` is so unreliable in the latest release and honestly we're trying to go 3.10+ ASAP. Further, - entirely drop the pattern matching inside the `do_ctlc()` for now. - add a `subactor_error` parametrization that catches a case that previously caused a hang (when you use 'next' immediately after the first crash/debug lock (the fix was pushed just before this commit).
Configuration menu - View commit details
-
Copy full SHA for a4538a3 - Browse repository at this point
Copy the full SHA a4538a3View commit details -
Configuration menu - View commit details
-
Copy full SHA for c0cd99e - Browse repository at this point
Copy the full SHA c0cd99eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1d4d55f - Browse repository at this point
Copy the full SHA 1d4d55fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 20c660f - Browse repository at this point
Copy the full SHA 20c660fView commit details -
Configuration menu - View commit details
-
Copy full SHA for a4bac13 - Browse repository at this point
Copy the full SHA a4bac13View commit details -
Configuration menu - View commit details
-
Copy full SHA for 457499b - Browse repository at this point
Copy the full SHA 457499bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6f01c78 - Browse repository at this point
Copy the full SHA 6f01c78View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5e23b3c - Browse repository at this point
Copy the full SHA 5e23b3cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 08cf03c - Browse repository at this point
Copy the full SHA 08cf03cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 91f034a - Browse repository at this point
Copy the full SHA 91f034aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 937ed99 - Browse repository at this point
Copy the full SHA 937ed99View commit details -
Configuration menu - View commit details
-
Copy full SHA for 87b2ccb - Browse repository at this point
Copy the full SHA 87b2ccbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8896ba2 - Browse repository at this point
Copy the full SHA 8896ba2View commit details -
Configuration menu - View commit details
-
Copy full SHA for aca9a6b - Browse repository at this point
Copy the full SHA aca9a6bView commit details -
Configuration menu - View commit details
-
Copy full SHA for acfbae4 - Browse repository at this point
Copy the full SHA acfbae4View commit details -
Configuration menu - View commit details
-
Copy full SHA for a9aaee9 - Browse repository at this point
Copy the full SHA a9aaee9View commit details -
Configuration menu - View commit details
-
Copy full SHA for e4771ee - Browse repository at this point
Copy the full SHA e4771eeView commit details -
Configuration menu - View commit details
-
Copy full SHA for c5c7a90 - Browse repository at this point
Copy the full SHA c5c7a90View commit details -
Configuration menu - View commit details
-
Copy full SHA for 54de72d - Browse repository at this point
Copy the full SHA 54de72dView commit details -
Configuration menu - View commit details
-
Copy full SHA for fa43888 - Browse repository at this point
Copy the full SHA fa43888View commit details -
Configuration menu - View commit details
-
Copy full SHA for 02c3b9a - Browse repository at this point
Copy the full SHA 02c3b9aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8115759 - Browse repository at this point
Copy the full SHA 8115759View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2d387f2 - Browse repository at this point
Copy the full SHA 2d387f2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7f6169a - Browse repository at this point
Copy the full SHA 7f6169aView commit details -
Configuration menu - View commit details
-
Copy full SHA for e4006da - Browse repository at this point
Copy the full SHA e4006daView commit details -
Configuration menu - View commit details
-
Copy full SHA for 650313d - Browse repository at this point
Copy the full SHA 650313dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 65540f3 - Browse repository at this point
Copy the full SHA 65540f3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8f1fe23 - Browse repository at this point
Copy the full SHA 8f1fe23View commit details -
Configuration menu - View commit details
-
Copy full SHA for cc5f60b - Browse repository at this point
Copy the full SHA cc5f60bView commit details