In process kernel for Qt frontend #1085

Closed
fperez opened this Issue Dec 1, 2011 · 11 comments

Comments

Projects
None yet
5 participants
Owner

fperez commented Dec 1, 2011

We really need to have an easy and clean way to embed a normal IPython kernel, minus the zmq communications part, in-process. The first obvious use case is using the Qt console with a kernel running in its own namespace. While this brings the obvious drawback that a kernel crash/segfault will take the whole app down, there are valid scenarios where this price is acceptable given the ability to directly interact in Qt with the objects in the kernel.

@rkern has a first hackish implementation in Gist that can serve for now as a stop-gap and should be the point of reference for a clean, long-term solution to this problem. See this thread on the ipython-dev list for the conversation on that gist.

Ultimately we want to abstract our kernel management api cleanly enough that it's a simple matter of changing one or two lines of code in a client to switch from in- to out-of-process kernel usage. At that point, we could trivially produce single and dual process versions of all our clients (console, qtconsole and html notebook) where the single-process ones would not have a zmq dependency. We're not quite there yet, but I'm creating this issue so we can track the problem and discuss it, until we have (one or more) actual PRs to work from.

Owner

takluyver commented Aug 16, 2012

Someone on Stackoverflow has come up with a rough implementation of this, if I'm reading it correctly:
http://stackoverflow.com/a/11525205/434217

Another commenter seems to have found a bug with that: http://stackoverflow.com/a/11940456/434217

Contributor

ChrisBeaumont commented Aug 16, 2012

I'm not sure whether my Stackoverflow approach implements what you're looking for or not -- it allows the Qt console to be embedded in an already-running Qt application, but still relies on ZMQ. It also hijacks stdout and stderr, which the gist referenced above seems to address. I've used this approach with IPython v0.12 and v0.13, though I haven't hit the bug @takluyver mentioned.

Owner

takluyver commented Aug 16, 2012

I think we'd be OK with having a dependency on ZMQ, at least for the time being - it could be cut out later. The main driver is for applications that want an embedded Qt console which can directly act on objects in the same process.

The long and the short of it is we've wanted to do this for some time, but it's never quite made it to the top of anyone's priority list. If it's something you're interested in, we'd love to have you work on it, and we'd be very happy to answer any questions about the architecture. Well, I imagine we would - it's not a bit of the code I know that well, so I guess I'm volunteering someone else's support ;-) . You can of course mix and match the best bits from your approach and @rkern's code in the gist.

Contributor

ChrisBeaumont commented Aug 16, 2012

I can take a crack at updating and merging what @rkern and I have put together, to see if it is suitable for integration with IPython. My main relationship with IPython is as a user and not a developer, so I am certain to have architecture questions :)

Owner

takluyver commented Aug 16, 2012

Great, thanks! We're always keen to convert users to developers, so feel free to ask questions here or on the ipython-dev mailing list. The latter is probably better, because more people will see it.

Contributor

ChrisBeaumont commented Aug 30, 2012

I've created a new gist that merges the ideas discussed so far, and I'm interested in some feedback (I will also post to the mailing list). The biggest issues I've run into are

  1. Dispatching standard and error output to the expected location (one of possibly several widgets, the terminal that started python, etc)

  2. Running both the IPKernelApp (which uses a blocking zmq IOLoop) and PyQt.QApplication (also blocking)

This code seems to work for me on v0.13 and v0.14dev, but there must be better ways to do some of this. Perhaps someone with more knowledge of IPython internals can point me in the right direction?

@rkern's approach seems to sidestep the KernelApp entirely, but requires lots of code duplication to create a functional KernelManager.

Contributor

epatters commented Aug 31, 2012

@ChrisBeaumont, while I applaud the simplicity of your approach, I think that we ultimately want to avoid using IPython's application machinery for this. @rkern's approach, while preliminary, is in the right spirit. Here is my tentative plan for implementing this feature:

  1. Create an EmbeddedKernel object that encapsulates an InteractiveShell and provides execute, object_info, complete, and related methods. These methods will take the same arguments as their counterparts defined on the kernel manager channels, but will directly return their results.
  2. Create an EmbeddedKernelManager that synchronously queries the EmbeddedKernel, but presents an interface largely identical to KernelManager (i.e., dispatches using call_handlers). Some of the kernel management methods like restart_kernel will have different semantics, and all ZMQ-specific attributes (session, ports, etc.) will be removed.
  3. Create a QtEmbeddedKernelManager subclass with an interface identical to QtKernelManager.

This approach has several advantages:

  1. The embedded kernel manager will not depend on ZMQ.
  2. Frontends will require essentially no modification. A typical frontend is written without assuming anything about the location of the kernel. To choose between in-process vs out-of-process kernels, simply plug in the appropriate kernel manager. This does not, of course, preclude the possibility of providing additional functionality when an in-process kernel is present.
  3. There remains a clean separation between the KernelManager abstraction and the kernel itself.

@fperez, @bgranger, I will be working on this at Enthought for a short time (~ 3 weeks) before I return to school. Although that is not a very long time, I should be able to produce a preliminary implementation of this feature. Do you have any suggestions about this proposal? In particular, is it sane? :)

Thanks,
Evan

Contributor

epatters commented Sep 1, 2012

Whoops, I intended to mention @ellisonbg there.

epatters was assigned Sep 1, 2012

Owner

takluyver commented Jan 17, 2013

I confess that I haven't been closely following it, but did pull request #2724 (just merged) address this, or is there more to be done before we close it?

Contributor

epatters commented Jan 17, 2013

Yes, this is finished.

epatters closed this Jan 17, 2013

@ChrisBeaumont i wanted to try your example, but it stalls when calling outstream_factory(self.session, self.iopub_socket, 'stdout') in KernelApp.initio().

any idea why?

or: what is the recommended approach to do this in ipython 0.13?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment