add ability to bind to a ruby kernel #2643

Carreau · 2012-12-03T13:35:27Z

This need the iruby kernel of my repo, (https://github.com/Carreau/iruby)
And to be modified as ruby interpreter and zmq-kernel path in IPython/zmq/rubykernel.py

Launch with (for example)
ipython notebook
--MappingKernelManager.kernel_manager_class='IPython.zmq.rubykernel.RubyKernelManager'

need zmq, uuid and hmac gem, and ruby 1.9.3

I would love to change the ruby interpreter path to ruby1.9.3 but Even with latest ruby installed, it is not in my path, so I don't know how what the ruby way is.

Also any idea of how we want to managed external kernel ? Do we put them in main IPython repo ? keep them separate ?

this (for now) need the iruby patch of my branch, and to modify a hardcoded path in IPython/zmq/rubykernel.py Launch with ipython notebook --MappingKernelManager.kernel_manager_class='IPython.zmq.rubykernel.RubyKernelManager' need zmq, uuid and hmac gem, and ruby 1.9.3

ellisonbg · 2012-12-03T17:34:55Z

The other kernels should be separate repos/projects under the ipython org. I think we should try to make these "packages" in whatever language they are written. So for the ruby kernel, we should be able to easily create a gem and install it like any other ruby package.

ellisonbg · 2012-12-03T17:42:05Z

IPython/frontend/html/notebook/kernelmanager.py

@@ -28,6 +28,7 @@

 from IPython.config.configurable import LoggingConfigurable
 from IPython.utils.importstring import import_item
+from IPython.zmq.rubykernel import RubyKernelManager


Does this really need to be imported here?

No, you're right.

Carreau · 2012-12-03T20:36:50Z

So, we need to build a 'PopenKernelManager' I guess with a configurable command line....

This should handle pretty much every basic things that are not python...
Do we make some kind of "template" for the command line ? Like %(cfile) replaced by connexion file ?

Carreau · 2012-12-04T09:11:41Z

does this look better ?

I just have to add this in my config to run the ruby kernel :

[.ipython/profile_ruby/ipython_notebook_config.py]

c = get_config()

c.SubprocessKernelManager.kernel_launch_program = 
         ['/.../ruby','~/.../kernel.rb','{connexion_file_name}']
c.MappingKernelManager.kernel_manager_class=
         'IPython.zmq.subprocesskernel.SubprocessKernelManager'

and start notebook with --profile=ruby

I think it should be able to run pretty much anything...

minrk · 2012-12-13T21:47:35Z

The existing kernel manager already starts a subprocess, and the launch command could already be customized (via passing launch). For instance, the iruby project which you reference didn't need to touch IPython at all to work. It seems this subclass just adds one configurable as an easier way to do what was already possible. Why not just add this configurable to the base KernelManager class?

Carreau · 2012-12-13T22:09:11Z

I didn't looked deep enough in kernelmanager and missed the configurability.

And actually the original iruby added another kernel launcher to launch the ruby kernel.

I'll try to see if I can do what you suggest.

minrk · 2012-12-14T00:19:10Z

And actually the original iruby added another kernel launcher to launch the ruby kernel.

Yes, I wrote it :) - but the important thing is that IPython itself did not need to be modified - the kernel launch args were not configurable, but the launch function was, so it needed a subclass assigned via the KMClass configurable. But that subclass is totally unnecessary if the launch command is configurable in the base class. There is a benefit to adding the configurable to the base class rather than a subclass, because that also means it is inherited by other subclasses (Qt, Blocking, etc.).

The reason it isn't there is probably just that the base KM is not a Configurable, but that's just a historical artifact since it was written before the Config stuff - it absolutely should be a Configurable.

Carreau · 2012-12-14T13:10:59Z

Question to @minrk,

Making kernelmanager.launch_kernel configurable will still have people be obliged to write their own launcher function in python to run a custom kernel, Am I wrong ?

Would it make more sens to make entry_point.base_launch_kernel configurable, and/or to provide a way to have launching process so that you don't have to write any python ?

The reason would be that then, the different kernel wouldn't have to deal with writing/installing any python files/module.

I might be missing something in the way this works though...

minrk · 2012-12-14T21:05:44Z

Ah, I forgot about the remnants still in entry_point. That means a little bit more needs to change than I thought.

But no, don't make launch_kernel configurable, make the Popen list itself configurable,
and make appropriate changes to base_launch_kernel to allow that.

minrk · 2012-12-14T21:18:19Z

a bit more:

base_launch_kernel does two things:

turn some functional arguments into a Popen command
add some wrapper stuff for pipes, etc.

It should be split so those are separate actions, and a custom Popen command skips the whole command assembly step.

ellisonbg · 2013-01-11T05:06:03Z

I just had a look at the following modules in IPython.zmq:

entry_point.py
ipkernel.py
kernelapp.py
kernelmanager.py

At least from a first glance, they are all sort of a rats nest of functions and classes that call each other. It is tempting to start refactoring this in a clean way, but I worry that it is actually quite subtle because of the different ways that kernels are used. But I also remember that this code grew in a very organic way and hasn't been cleaned up since the beginning. @minrk do you have a sense of why things are such a mess - is that needed? Is part of this the complexity of IPython.parallel also using the same kernel?

minrk · 2013-01-11T05:26:04Z

I think kernelmanager.py should be fairly well contained. What's wrong with it? Maybe you want to split the channels into their own file? I don't see any particular reason to do that.
entry_point shouldn't exist, as it's a vestige of the pre-Application way of launching things. Almost all of it is gone now, with just two utility functions left. They can go somewhere else, if there is a better name for it. I never did move them, unlike the rest of its original contents which I did move various other more appropriate places, because I couldn't come up with a better one.
kernelapp and ipkernel are the way they are because there used to be two kinds of kernels - they should definitely be cleaned up, ipkernel especially. Probably move IPKernelApp to its own ipkernelapp or add it to kernelapp. And the extra launch functions at the bottom probably belong wherever the contents of entry_point end up.

It is totally appropriate to clean this up, but do note that from IPython.zmq.ipkernel import IPKernelApp is currently in use as a public API. Be aware of PR #2724, because it should probably result in a zmq-free bases of Kernel / KernelManager, from with the zmq ones will inherit. Any attempt at cleanup will conflict severely with that work.

minrk · 2013-01-11T05:30:59Z

Forgot to mention: No, there shouldn't be anything related to IPython.parallel in the complexity here. The last major step of merging the two kernels is to make IPEngineApp a minimal subclass of IPKernelApp.

ellisonbg · 2013-01-11T05:48:17Z

Thanks Min that really helps. I guess with #2724 I will just wait. I can accomplish everything I need to with the current design, it is just a bit messy.

I think the only problem with kernelmanager.py is how the launch_kernel function is customized in a wonky way by passing a launcher kwarg to start_kernel. It should probably be a configurable attribute. I have thought about separating the channels into separate code, but that is a minor point.
I would separate the files into three clean ones: put the actual kernel implementation in one file, put the applications in another, and put the launching code in another.
Is there any need to have separate KernelApp and IPKernelApp classes. It is not at all clear why the two are needed and it means that IPKernelApp has multiple inheritance.

In what way is from IPython.zmq.ipkernel import IPKernelApp a public API? If it makes sense I would love to combine the two applications and put the result into its own file, to leave ipkernel with just the kernel.

minrk · 2013-01-11T05:58:42Z

In what way is from IPython.zmq.ipkernel import IPKernelApp a public API?

If you want to start a kernel and customize it in any way whatsoever, you must make this import. It is equivalent to the embedded classes - IPython.embed is fine if you don't want to change anything, but you have to import the class itself to do any customization. People are doing this (maybe not many).

I would separate the files into three clean ones: put the actual kernel implementation in one file, put the applications in another, and put the launching code in another.

Sensible

Is there any need to have separate KernelApp and IPKernelApp classes. It is not at all clear why the two are needed and it means that IPKernelApp has multiple inheritance.

The main reason was to separate IPython-shell-specific startup from just 'generic interpreter startup'. Again, the reason being that we used to have a pure Python Kernel. I think the main argument against merging them is that we may again, but that's not a particularly strong one. I would maybe just put IPKernelApp as-is after KernelApp in zmq.kernelapp, but merging the two is fine as well.

ellisonbg · 2013-01-11T06:04:11Z

OK great, I will try to help review #2724 and keep my eye on these things
for cleanup.

On Thu, Jan 10, 2013 at 9:58 PM, Min RK notifications@github.com wrote:

In what way is from IPython.zmq.ipkernel import IPKernelApp a public API?

If you want to start a kernel and customize it in any way whatsoever, you
must make this import. It is equivalent to the embedded classes -
IPython.embed is fine if you don't want to change anything, but you have
to import the class itself to do any customization. People are doing this
(maybe not many).

I would separate the files into three clean ones: put the actual kernel
implementation in one file, put the applications in another, and put the
launching code in another.

Sensible

Is there any need to have separate KernelApp and IPKernelApp classes. It
is not at all clear why the two are needed and it means that IPKernelApp
has multiple inheritance.

The main reason was to separate IPython-shell-specific startup from just
'generic interpreter startup'. Again, the reason being that we used to have
a pure Python Kernel. I think the main argument against merging them is
that we may again, but that's not a particularly strong one. I would maybe
just put IPKernelApp as-is after KernelApp in zmq.kernelapp, but merging
the two is fine as well.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2643#issuecomment-12132833.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

ellisonbg · 2013-01-17T06:05:46Z

I cleaned up the kernel manager code a lot in #2775 but didn't really change how the custom launcher is specified. Bu tin either case, I don't think the approach in this branch is the way to do. In the long run, I think we will want to build an additional argument into MultiKernelManager.start_kernel, namely, the name of the kernel to start: ipython, ruby, etc. This list should be where the configuration happens, Each kernel name should be associated with the required launcher. For now can we just get by with what we have until we really get to thinking about multi-kernel support an what that will look like?

minrk · 2013-01-17T06:18:43Z

@ellisonbg that approach doesn't get users anywhere without updates to IPython itself. I think the point here is that we should have a configurable for launching custom kernels, either with

A) a custom launch_kernel function, as a DottedObjectName
or B) a custom Popen command-line list.

Now, maybe this should be a mapping by kernel-name, but in any case, we should expose the fact that the KernelManager as it already is can launch custom kernels. It's just not exposed to config.

It should be straightforward to add one or both of these, particularly now that KernelManager itself is Configurable, as it always should have been.

ellisonbg · 2013-01-17T22:40:16Z

I wasn't very clear in my description. I agree that these things should
absolutely be a configurable attribute, not something that is merely passed
into the function in the wonky way that launcher currently is. And I think
that somethink like the API you are proposing makes sense. I just think we
should close this PR and open an issue to have that discussion about what
this should look like. I agree that the work shouldn't be too difficult,
but we also probably want to cleanup the entry_point/base_kernel_launcher
mess.

On Wed, Jan 16, 2013 at 10:18 PM, Min RK notifications@github.com wrote:

@ellisonbg https://github.com/ellisonbg that approach doesn't get users
anywhere without updates to IPython itself. I think the point here is that
we should have a configurable for launching custom kernels, either with

A) a custom launch_kernel function, as a DottedObjectName
or B) a custom Popen command-line list.

Now, maybe this should be a mapping by kernel-name, but in any case, we
should expose the fact that the KernelManager as it already is can launch
custom kernels. It's just not exposed to config.

It should be straightforward to add one or both of these, particularly now
that KernelManager itself is Configurable, as it always should have been.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2643#issuecomment-12355928.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

Carreau · 2013-01-22T13:12:26Z

FYI, perl kernel :
https://github.com/timo/iperl6kernel

ellisonbg · 2013-01-23T20:14:13Z

@minrk and @Carreau I never heard back from you if we can close this PR and open an issue for the broader kernel launcher work.

minrk · 2013-01-23T21:06:22Z

Actually you did, just not here. I was going to submit a new PR with the changes discussed here (split base_launch_kernel into its two separate actions, and expose Popen command as KM config, avoiding the need for subclass), and suggest closing this when I submit that, rather than closing without having a replacement to point to.

ellisonbg · 2013-01-23T21:08:49Z

Hehe, if I actually were old I could make some joke about my memory going
bad. Just forgot...sorry for wasting your time...

On Wed, Jan 23, 2013 at 1:06 PM, Min RK notifications@github.com wrote:

Actually you did, just not here. I was going to submit a new PR with the
changes discussed here (split base_launch_kernel into its two separate
actions, and expose Popen command as KM config, avoiding the need for
subclass), and suggest closing this when I submit that, rather than closing
without having a replacement to point to.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2643#issuecomment-12622293.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

minrk · 2013-01-26T04:37:49Z

replaced by #2854

ellisonbg reviewed Dec 3, 2012
View reviewed changes

Carreau added 2 commits December 4, 2012 08:23

subprocess kernel manager

3ddce3a

working generic subprocess

bb0b397

Carreau added 2 commits December 4, 2012 11:27

remove useless "as expanduser"

6c544db

fix english

5e10e8d

Carreau mentioned this pull request Dec 4, 2012

zeromq/IPython notebook integration. JuliaLang/julia#1675

Closed

Carreau mentioned this pull request Dec 18, 2012

Table of contents #1692

Open

Carreau mentioned this pull request Jan 7, 2013

IClojure as IPython kernel cosmin/IClojure#27

Open

jdmarch mentioned this pull request Jan 15, 2013

In-process kernel support (take 3) #2724

Merged

Carreau mentioned this pull request Jan 22, 2013

qtconsole should use kernel_info_request to choose pygments lexer #2827

Closed

This was referenced Jan 26, 2013

Allow custom kernel Popen command #2853

Closed

Move kernel code into IPython.kernel #2854

Merged

minrk closed this Jan 26, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add ability to bind to a ruby kernel #2643

add ability to bind to a ruby kernel #2643

Carreau commented Dec 3, 2012

ellisonbg commented Dec 3, 2012

ellisonbg Dec 3, 2012

Carreau Dec 3, 2012

Carreau commented Dec 3, 2012

Carreau commented Dec 4, 2012

minrk commented Dec 13, 2012

Carreau commented Dec 13, 2012

minrk commented Dec 14, 2012

Carreau commented Dec 14, 2012

minrk commented Dec 14, 2012

minrk commented Dec 14, 2012

ellisonbg commented Jan 11, 2013

minrk commented Jan 11, 2013

minrk commented Jan 11, 2013

ellisonbg commented Jan 11, 2013

minrk commented Jan 11, 2013

ellisonbg commented Jan 11, 2013

ellisonbg commented Jan 17, 2013

minrk commented Jan 17, 2013

ellisonbg commented Jan 17, 2013

Carreau commented Jan 22, 2013

ellisonbg commented Jan 23, 2013

minrk commented Jan 23, 2013

ellisonbg commented Jan 23, 2013

minrk commented Jan 26, 2013

add ability to bind to a ruby kernel #2643

add ability to bind to a ruby kernel #2643

Conversation

Carreau commented Dec 3, 2012

ellisonbg commented Dec 3, 2012

ellisonbg Dec 3, 2012

Choose a reason for hiding this comment

Carreau Dec 3, 2012

Choose a reason for hiding this comment

Carreau commented Dec 3, 2012

Carreau commented Dec 4, 2012

minrk commented Dec 13, 2012

Carreau commented Dec 13, 2012

minrk commented Dec 14, 2012

Carreau commented Dec 14, 2012

minrk commented Dec 14, 2012

minrk commented Dec 14, 2012

ellisonbg commented Jan 11, 2013

minrk commented Jan 11, 2013

minrk commented Jan 11, 2013

ellisonbg commented Jan 11, 2013

minrk commented Jan 11, 2013

ellisonbg commented Jan 11, 2013

ellisonbg commented Jan 17, 2013

minrk commented Jan 17, 2013

ellisonbg commented Jan 17, 2013

Carreau commented Jan 22, 2013

ellisonbg commented Jan 23, 2013

minrk commented Jan 23, 2013

ellisonbg commented Jan 23, 2013

minrk commented Jan 26, 2013