Multiprocessing in ipython notebook kernel crash

[crbates]

The simple script:

from multiprocessing import Process
def f(name):
    print('hello', name)

p = Process(target=f, args=('bob',))
p.start()
p.join()

Works fine in an ipython notebook in 0.13 but crashes the kernel in the head branch. The kernel spits out a ZMQ error: Assertion failed: ok (mailbox.cpp:84).

[Carreau]

seem to be
3cbec13 use brief dtype, rather than full object

[minrk]

@Carreau - no, it definitely isn't

@crbates - what version of pyzmq? If < 2.2.0.1, please update. There should be a fork-related fix there.

[Carreau]

hum, strange, that what git bisect told me... I might have type bad instead of good once... i'll retry.

[minrk]

@Carreau - the important thing is this is not an IPython bug at all, so git bisect on IPython shouldn't find anything. The fix is in pyzmq.

[crbates]

I am running pyzmq 2.2.0.1 and python 2.7.3. @minrk If it isn't an ipython bug then why does it break when I switch between master and the 13.1 branch?

[minrk]

My apologies - there is something IPython related going on here, I didn't notice the print statement. The issue is that zmq sockets cannot be passed across a fork, and sys.stdout is linked to a zmq socket. IPython is going to need to detect forks and discard (and possibly recreate) sockets when a fork happens.

I don't know how the crash occurs in the parent process in master, that's super weird. I would expect the child to crash, though.

[crbates]

It seems that the child does crash. So is there a way to fork processes from running notebooks/engines? I would like to fork a process from a notebook/engine so that the notebook/engine itself can act as a stable interface to a data acquisition process.

[minrk]

The commit where the behavior changed is 2aee9e3, which has the effect of changing small messages to not use zero-copy. So I suspect the difference is that the state of zmq buffers is different when fork is called in master. In any case, if IPython detects a fork and avoids re-using the sockets the crash should be avoided. I would consider it an anomaly that 0.13 does not crash in this case.

[crbates]

Yeah after further testing it also crashes the ipython kernel in 0.13 some of the time. I guess I'll have to customize an ipython engine instance to do what I want.

[minrk]

For reference, the issue is specifically with the print statement. You should have no issue if you can avoid writing to stdout, or redirect stdout/err back to sys.__stdout__/err prior to any print statements in the forked process.

[hmeine]

Looks like this bug should be re-opened, and #2438 be closed as a duplicate, no?

[minrk]

It doesn't matter which one is kept open, but #2438 has a more precise description of the actual issue.