We really need to have an easy and clean way to embed a normal IPython kernel, minus the zmq communications part, in-process. The first obvious use case is using the Qt console with a kernel running in its own namespace. While this brings the obvious drawback that a kernel crash/segfault will take the whole app down, there are valid scenarios where this price is acceptable given the ability to directly interact in Qt with the objects in the kernel.
@rkern has a first hackish implementation in Gist that can serve for now as a stop-gap and should be the point of reference for a clean, long-term solution to this problem. See this thread on the ipython-dev list for the conversation on that gist.
Ultimately we want to abstract our kernel management api cleanly enough that it's a simple matter of changing one or two lines of code in a client to switch from in- to out-of-process kernel usage. At that point, we could trivially produce single and dual process versions of all our clients (console, qtconsole and html notebook) where the single-process ones would not have a zmq dependency. We're not quite there yet, but I'm creating this issue so we can track the problem and discuss it, until we have (one or more) actual PRs to work from.
Someone on Stackoverflow has come up with a rough implementation of this, if I'm reading it correctly:
Another commenter seems to have found a bug with that: http://stackoverflow.com/a/11940456/434217
I'm not sure whether my Stackoverflow approach implements what you're looking for or not -- it allows the Qt console to be embedded in an already-running Qt application, but still relies on ZMQ. It also hijacks stdout and stderr, which the gist referenced above seems to address. I've used this approach with IPython v0.12 and v0.13, though I haven't hit the bug @takluyver mentioned.
I think we'd be OK with having a dependency on ZMQ, at least for the time being - it could be cut out later. The main driver is for applications that want an embedded Qt console which can directly act on objects in the same process.
The long and the short of it is we've wanted to do this for some time, but it's never quite made it to the top of anyone's priority list. If it's something you're interested in, we'd love to have you work on it, and we'd be very happy to answer any questions about the architecture. Well, I imagine we would - it's not a bit of the code I know that well, so I guess I'm volunteering someone else's support ;-) . You can of course mix and match the best bits from your approach and @rkern's code in the gist.
I can take a crack at updating and merging what @rkern and I have put together, to see if it is suitable for integration with IPython. My main relationship with IPython is as a user and not a developer, so I am certain to have architecture questions :)
Great, thanks! We're always keen to convert users to developers, so feel free to ask questions here or on the ipython-dev mailing list. The latter is probably better, because more people will see it.
I've created a new gist that merges the ideas discussed so far, and I'm interested in some feedback (I will also post to the mailing list). The biggest issues I've run into are
Dispatching standard and error output to the expected location (one of possibly several widgets, the terminal that started python, etc)
Running both the IPKernelApp (which uses a blocking zmq IOLoop) and PyQt.QApplication (also blocking)
This code seems to work for me on v0.13 and v0.14dev, but there must be better ways to do some of this. Perhaps someone with more knowledge of IPython internals can point me in the right direction?
@rkern's approach seems to sidestep the KernelApp entirely, but requires lots of code duplication to create a functional KernelManager.
@ChrisBeaumont, while I applaud the simplicity of your approach, I think that we ultimately want to avoid using IPython's application machinery for this. @rkern's approach, while preliminary, is in the right spirit. Here is my tentative plan for implementing this feature:
This approach has several advantages:
@fperez, @bgranger, I will be working on this at Enthought for a short time (~ 3 weeks) before I return to school. Although that is not a very long time, I should be able to produce a preliminary implementation of this feature. Do you have any suggestions about this proposal? In particular, is it sane? :)
Whoops, I intended to mention @ellisonbg there.
I confess that I haven't been closely following it, but did pull request #2724 (just merged) address this, or is there more to be done before we close it?
Yes, this is finished.
@ChrisBeaumont i wanted to try your example, but it stalls when calling outstream_factory(self.session, self.iopub_socket, 'stdout') in KernelApp.initio().
outstream_factory(self.session, self.iopub_socket, 'stdout')
any idea why?
or: what is the recommended approach to do this in ipython 0.13?