Documentation for custom kernel + a non-standard event loop

[kmike]

I'm writing a custom IPython kernel (for Lua, via lupa), and want it to run in a PyQT event loop. The final code looks simple, and it is very cool IPython allows to implement this:

from IPython.kernel.zmq.kernelbase import Kernel
from IPython.kernel.zmq.kernelapp import IPKernelApp
from IPython.kernel.zmq.eventloops import loop_qt4
from IPython.lib.guisupport import get_app_qt4

class MyKernel(Kernel):
    # ... it creates some Qt windows in its __init__ method

def start():
    get_app_qt4()  # ensure QApplication is created
    kernel = IPKernelApp.instance(kernel_class=MyKernel)
    kernel.initialize()
    kernel.kernel.eventloop = loop_qt4
    kernel.start()

But it took me several hours to figure out how to write this code. There are docs about individual pieces:

• Integrating with GUI event loops;
• Making simple Python wrapper kernels;
• Kernel API reference
• IPKernelApp API reference
• eventloops API reference
• examples: internal_ipkernel.py and ipkernel_qtapp.py

"Integrating with GUI event loops" was very helpful to get a general understanding, but it looks specific to %gui magic. It explains how to write a new event loop integration function, but it doesn't explain how to enable it.

To understand it better, I've written a polling version of qt event loop, registered it as "qt-timer" as explained in docs - and now what? I tried to assign 'IPKernelApp.gui' to 'qt-timer', it didn't work. kernel.eventloop attribute is not documented in Kernel docs. enable_gui could have worked, but it was not clear where to get Kernel instance from (more on this below), and also when to call this function.

Making simple Python wrapper kernels shows how to write and start a kernel; in the end I had to use another start method. We never constructed a Kernel instance explicitly (it is created somewhere in the guts of IPKernelApp), kernel_app.kernel attribute is not documented in IPKernelApp docs, and it is not in the examples, but other docs (e.g. API reference for even loops) assume we have this object.

In the example IPKernelApp instance (not a Kernel instance) is called a "kernel" which adds to confusion. Actually, it makes sense to call IPKernelApp a "kernel" because it looks closer to a "kernel" described in How IPython Works than a Kernel class, but that's another question.

In the example the event loop is enabled via kernel.initialize options:

kernel.initialize(['python', '--matplotlib=%s' % gui,
#'--log-level=10'
])

Try figure out what should be "initialize" method arguments from IPKernelApp docs :) Selecting the event loop via --matplotlib option is weird if you don't need matplotlib.

In the end, these docs combined + some source code reading helped, and the individual doc pieces are good. But IMHO they don't play together currently.

No concrete proposals from me, sorry :)

[Carreau]

Thanks for the feedback,

If you find external resources or write soem on your own, please add them to the wiki .

If you are looking at lua have you seen facebook lua kernel ?

The use of IPkernel and wrapper kernel is a bit strange, I personally never though of that.
We'll see what we can do.

[kmike]

Hi @Carreau,

Yes, I've seen Facebook Lua kernel, it looks great. We can't use it because in https://github.com/scrapinghub/splash we're using Lua scripts to control PyQt4.QtWebKit.QWebView - most code is in Python (yeah, that's complicated; there is also Twisted event loop involved). In addition to that, iTorch uses LuaJIT while Splash uses Lua 5.2; they are not compatible. But we'll probably borrow some code (a completer?).

Once I finish the IPython integration I'll add a link to the wiki, thanks for the suggestion.

[dsblank]

@kmike Just FYI, if you are building a kernel off of the wrapper kernel API, you might be interested in using Calysto MetaKernel which is designed to be a drop-in replacement, and gives you magics, shell, completions, and help infrastructure.

[kmike]

@dsblank thanks for the link!

[Carreau]

@kmike Was just pointing out LuaJIT kernel, you don't have to justify your choices :-)
Anyway happy to see a new kernel :-)

@dsblank Ah, I should have though of Calysto MetaKernel also... thanks for pointing that out !

[dsblank]

@kmike You're welcome! I didn't mention some other benefits: you can use this to run Lua code in parallel, and have access to the IPython widgets (both could be more fully developed). If you are using something like subprocess, you can take a look at ProcessMetaKernel. Feedback, ideas welcome!

[takluyver]

I think having a wrapper kernel that also integrates with a python-land
event loop is undocumented because we never thought of such a possibility:
you're breaking new ground. ;-)