forward stdout from forked processes #2791
Conversation
|
supercedes #2734 |
|
Come to think of it, I don't think this can work on Windows, due to the lack of a real fork. |
|
last failure was a travis timeout - it actually passes now. |
|
multiprocessing.Pool.map seems to do something funny with stdout import multiprocessing as mp
def f(x):
print x
return x
pool = mp.Pool()
pool.map(f, range(200))running on terminal will print the numbers but not in this patch. May be we miss a flush on join or something along that line? |
|
Must be a missed flush. |
km can be specified, but default behavior remains the same (global KM)
tests new mp.Process behavior for PR ipython#2734
python 3 doesn't seem to like the parametric, plus a few missing print parentheses
and add docstring / copyright
s/sub_channel/iopub_channel/
simple Process(target=f) doesn't work for interactively defined f.
because it isn't using zmq anyway
|
adding an explicit flush results in expected output |
|
I know explicit flush will fix the issue. And this is probably second order issue. |
reduces issues with mp.Pool
|
It should be a bit better behaved now. I made |
|
This may not has to do with this PR but I notice that sometimes python warning(written to stderr) isn't flushed at the end of code cell and it leaks to the next cell execution instead. |
|
That would not be related to this. It would be exceedingly peculiar, since sys.stderr is flushed at the end of all execute requests. Does this warning perhaps come from a thread? |
|
@minrk I would like to help review this but it looks like pretty complex code in some critical parts of the code base. I would like to understand the motivation/design better. Maybe we could Skype later to go over the design? |
|
works for me! |
|
@minrk @ellisonbg did you had a chance to Skype to discuss that, is this ok to merge ? |
|
We haven't had a chance - I think it's ready, but I need someone else's eyes on it since it's my code. |
|
and i didn't really look through the code closely - just tested it empirically on a problem i was having. |
|
OK @minrk I am plugging back into code reviews and will be one HipChat most of the week and available on skype all days but Tuesday 4/9. |
|
Ok, hopefully we can get this in tomorrow. See you on hip chat. |
|
I am guessing that my best times will be 1-3 and 7-9. Cheers, Brian |
| """flush any messages waiting on the queue""" | ||
| for channel in (KM.shell_channel, KM.iopub_channel): | ||
| for channel in (km.shell_channel, km.iopub_channel): |
There was a problem hiding this comment.
I guess this will have to change after the kernel manager PR is merged.
There was a problem hiding this comment.
Yes, there will need to be some rewriting - that PR (and the two that depend on it) change so much, that I am running out of stuff I can reasonably do before they are merged.
|
I left a few minor comments, but the code looks really good. I am +1 for merging this after you look at my mostly trivial comments. |
forward stdout from forked processes uses zmq instead of multiprocessing, because mp has too many issues. - messages are sent via PUSH/PULL from subprocesses - messages are sent at flush time, not at write time - subprocess messages - no threads, no sync events, etc. some basic tests are included closes #2438
forward stdout from forked processes uses zmq instead of multiprocessing, because mp has too many issues. - messages are sent via PUSH/PULL from subprocesses - messages are sent at flush time, not at write time - subprocess messages - no threads, no sync events, etc. some basic tests are included closes ipython#2438
third attempt, in collaboration with @piti118
uses zmq instead of multiprocessing, because mp has too many issues.
some basic tests are included
todo: