Wen? Multiprocessing-native debugger now! #129

goodboy · 2020-07-23T18:01:38Z

This is a first attempt at accomplishing #113 😮

Surprisingly it all works without using any threads, greenback, or other dark magic; it's just tractor bby 💋.

Hopefully we can get some feedback from lurkers 🕵️

This first edition definitely works as well as other alternatives but there's a bunch of stuff we can do to make it better.

Caveats:

~~only first level actors can be debugged this way since the parent requires access to the subprocesse's stdin pipe~~
- any actor in a tree is now supported!
~~2nd depth level actors are completely untested for now~~
- they are definitely now tested!
no clue if this will work on windows (and I even tried, see Does the debugger work on Windows? Yes, yes it does, but how do we test it? #158)

Immediate TODOs:

(we'll need to defer this until Fancy remote debugging (aka tab completion and editor controls) #130 is adressed) 1. disable the "cooked" mode when processing stdin (you'll see some commented code where I already started on this) such that each character is relayed immediately to the child stdin pipe.
- This should get us out of the box completions and line editor control from the remote pdbpp 🎉
- I'm working off this example code; if there are terminal nerds about please feel free to yell about anything wrong with this approach :bowing_man
2. lock access to the parent's stdin such that no more then one subactor can hijack it at a time; this of course is required in order to avoid all sorts of console mayhem
- I've got locking working but not switching between two subactors requesting the parents stdio..
3. somehow test all this? I think taking at look at how pytest does it is probably a good start.
4. get feedback from some real users and figure out if debug_mode is cool or not.
- thinking this will come with .dev0 usage so marking complete for now.

For the future:

make an issue summarizing all this: see Fancy remote debugging (aka tab completion and editor controls) #130
- for multi-host systems we need a way to of course relay to stdin in the child over a Channel. There seems to be a variety of projects that do this mostly listed in REPL-driven, multi-process-native debugger features #113. I want to use pdb++ since I think it's the best UX available, but we may need to hook into its sys.stdin in a sneaky way.
  - epdb seems to have code for this (on regular pdb) but I haven't tested it.
- we probably want Add support for talking to our stdin/stdout/stderr as streams python-trio/trio#174 to have a better api experience while hijacking the parent's stdin, also see:
- do we want to support overriding the breakpoint() built-in (I also have the draft code commented in here)
  - users will still have to await breakpoint() unless we implement it a different way

goodboy · 2020-07-23T18:02:29Z

@guilledk If you get a chance try to play with this as well :)

goodboy · 2020-07-24T19:23:45Z

Alright, so after much frustration trying to get tab completion to just work I went back to trip and found that it did just work and I finally got the 🤦 that of course it's not going to work with our code since the child process isn't attached the parent's (nor any) tty 😿

Of course rlcompleter (and thus pdbp and pdbp++) rely on a tty at startup for any readline support which is what gives all the fancy tab completion and control character functionality. I'm pretty sure even somehow hacking a pty around breakpoint() calls isn't going to work either especially since there's a whole package (ptyprocess) built to solve the problem of launching subprocs under ptys, which, btw does work with pdb when you enter from the subproc.

I think instead of doing a big writeup here I'm going to make a new issue and discuss my findings.
2. - 4. should still get complete before this lands.

Oh and one further approach we may want to take to get the fancy stuff for now is just don't hook up subproc stdin to a pipe and instead let it have the parent's tty and then just use a trio.FIFOLock around access (including overriding breakpoint() access by the root process). This might get us 90% of the way I'd like things to work without being super correct and it should work for nested subprocs I think.

goodboy · 2020-07-26T01:23:29Z

@guilledk yeah, I'm not following that change?

The debugging stuff I don't think should have an effect here?
Maybe i'm missing something.

This reverts commit 2b53c74.

This reverts commit 1710b64.

goodboy · 2020-10-13T20:49:12Z

Worth mentioning is that I have not idea if this stuff works no windows since there doesn't seem to be any pexpect support?

Might have to dig into this so that likely means a deferral issue is coming.

goodboy · 2020-10-14T01:48:17Z

tests/test_cancellation.py

+        portal = await tn.run_in_actor('sleeper', spin_for)
+
+
+@no_windows


Probably need an issue for this as well.

goodboy · 2020-10-14T01:56:58Z

tractor/_actor.py

@@ -171,13 +176,16 @@ class Actor:
    _root_n: Optional[trio.Nursery] = None
    _service_n: Optional[trio.Nursery] = None
    _server_n: Optional[trio.Nursery] = None
+    _lifetime_stack: ExitStack = ExitStack()


Oh right, we might want tests for this 🤣..

Maybe it's ok to just slip it in for now and worry later?

goodboy · 2020-10-14T12:35:39Z

tests/test_debugger.py

+
+
+# TODO:
+# - recurrent entry to breakpoint() from single actor *after* and an


Actually, I think most of these are covered now but wouldn't hurt to do an audit in the future.

I don't have much juice left to keep pushing here 😿

goodboy · 2020-10-14T12:37:51Z

tests/test_debugger.py

+    at each level up the tree.
+    """
+
+    # TODO: inside this script there's still a bug where if the parent


This is actually fixed now and is addressed by solving #156.

The test below actually demonstrates this is now solved.

goodboy added enhancement New feature or request help wanted Extra attention is needed discussion experiment Exploratory design and testing labels Jul 23, 2020

goodboy requested review from ryanhiebert and salotz July 23, 2020 18:01

goodboy mentioned this pull request Jul 24, 2020

REPL-driven, multi-process-native debugger features #113

Open

goodboy mentioned this pull request Jul 24, 2020

Fancy remote debugging (aka tab completion and editor controls) #130

Open

goodboy force-pushed the flaky_tests branch from dfe0f15 to d3acb8d Compare July 24, 2020 21:09

goodboy mentioned this pull request Jul 24, 2020

Drop trio-run-in-process, use pure trio process spawner, test out of channel ctrl-c subactor cancellation #128

Merged

5 tasks

goodboy removed the request for review from ryanhiebert July 24, 2020 21:36

goodboy force-pushed the flaky_tests branch 4 times, most recently from cf8a980 to dddbeb0 Compare July 25, 2020 17:42

goodboy requested a review from guilledk July 25, 2020 22:23

goodboy force-pushed the multiproc_debug branch from a3c79e9 to d23b82d Compare July 25, 2020 22:30

This comment has been minimized.

Sign in to view

goodboy force-pushed the flaky_tests branch 9 times, most recently from df5e702 to 5a27065 Compare July 27, 2020 02:53

Support debug mode only on the trio backend

e3c2694

goodboy marked this pull request as ready for review October 13, 2020 18:21

goodboy added 6 commits October 13, 2020 14:42

Revert "Change to relative conftest.py imports"

6669660

This reverts commit 2b53c74.

Revert "Make tests a package (for relative imports)"

a49deb4

This reverts commit 1710b64.

Factor repodir() helper into conftest.py

a934eb0

On windows .spawn dne?

fd59f4a

Skip it on windows too

15edcc6

Skip sync sleep test on windows

1b6ee2e

goodboy force-pushed the multiproc_debug branch from 5a1a245 to 1b6ee2e Compare October 13, 2020 19:27

Report on skipped tests

0177268

goodboy force-pushed the multiproc_debug branch 2 times, most recently from bcb2714 to 8b4e2bb Compare October 14, 2020 01:02

goodboy mentioned this pull request Oct 14, 2020

Does the debugger work on Windows? Yes, yes it does, but how do we test it? #158

Open

goodboy force-pushed the multiproc_debug branch from 8b4e2bb to 0177268 Compare October 14, 2020 01:23

Drop travisCI; it's slower and has worse windows support.

1c25f25

goodboy force-pushed the multiproc_debug branch 3 times, most recently from bf90705 to 0510ab2 Compare October 14, 2020 01:43

Add gh actions badge

bba47e4

goodboy force-pushed the multiproc_debug branch from 0510ab2 to bba47e4 Compare October 14, 2020 01:46

goodboy commented Oct 14, 2020

View reviewed changes

goodboy mentioned this pull request Oct 14, 2020

"Context stack" tests #160

Open

guilledk approved these changes Oct 14, 2020

View reviewed changes

goodboy commented Oct 14, 2020

View reviewed changes

Comments tweak

61a8df3

goodboy merged commit 7115d6c into master Oct 14, 2020

goodboy deleted the multiproc_debug branch October 14, 2020 13:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wen? Multiprocessing-native debugger now! #129

Wen? Multiprocessing-native debugger now! #129

goodboy commented Jul 23, 2020 •

edited

goodboy commented Jul 23, 2020

goodboy commented Jul 24, 2020 •

edited

This comment has been minimized.

goodboy commented Jul 26, 2020 •

edited

goodboy commented Oct 13, 2020

goodboy Oct 14, 2020

goodboy Oct 14, 2020

goodboy Oct 14, 2020

goodboy Oct 14, 2020 •

edited

		portal = await tn.run_in_actor('sleeper', spin_for)


		@no_windows



		# TODO:
		# - recurrent entry to breakpoint() from single actor after and an

Wen? Multiprocessing-native debugger now! #129

Wen? Multiprocessing-native debugger now! #129

Conversation

goodboy commented Jul 23, 2020 • edited

Caveats:

Immediate TODOs:

For the future:

goodboy commented Jul 23, 2020

goodboy commented Jul 24, 2020 • edited

This comment has been minimized.

goodboy commented Jul 26, 2020 • edited

goodboy commented Oct 13, 2020

goodboy Oct 14, 2020

Choose a reason for hiding this comment

goodboy Oct 14, 2020

Choose a reason for hiding this comment

goodboy Oct 14, 2020

Choose a reason for hiding this comment

goodboy Oct 14, 2020 • edited

Choose a reason for hiding this comment

goodboy commented Jul 23, 2020 •

edited

goodboy commented Jul 24, 2020 •

edited

goodboy commented Jul 26, 2020 •

edited

goodboy Oct 14, 2020 •

edited