Move kernel code into IPython.kernel #2854

minrk · 2013-01-26T04:37:06Z

inprocess Kernel in kernel.inprocess
zmq Kernel in kernel.zmq
KernelManager stuff in top-level kernel

Main functional change: allow custom kernel Popen command

reissue of #2853, because it didn't seem to want to track the right commits

If you use my iruby branch (every other one I found had failures in the msg spec / session), you can run ruby kernels with qtconsole/console/notebook with:

ipython console --KernelManager.kernel_cmd='["ruby", "lib/kernel.rb", "{connection_file}"]' --Session.key=''

supersedes #2643

Carreau · 2013-01-26T10:15:40Z

Will try to get a look but need to go now.
Would you like to put the ruby kernel in its own repo somewhere on IPython Organisation ?

Carreau · 2013-01-26T15:26:12Z

FYI travis build is done and passes.
we can still put the function of entry point elsewhere later.

Maybe just to be annoying, make_kernel_cmd does not reflect the fact that it's only for python kernel.

minrk · 2013-01-28T23:05:20Z

@Carreau I've renamed it make_ipkernel_cmd.

I decide to consolidate IPython.lib.kernel and IPython.zmq.entry_point into a single IPython.utils.kernel (IPython.lib.kernel imports from here with a deprecation warning, to avoid breaking apps like vim-ipython).

ellisonbg · 2013-01-28T23:33:56Z

@minrk is this consolidation work going to be in this PR? Should I wait to
review it?

On Mon, Jan 28, 2013 at 3:05 PM, Min RK notifications@github.com wrote:

@Carreau https://github.com/Carreau I've renamed it make_ipkernel_cmd.

I decide to consolidate IPython.lib.kernel and IPython.zmq.entry_point
into a single IPython.utils.kernel (IPython.lib.kernel imports from here
with a deprecation warning, to avoid breaking apps like vim-ipython).

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2854#issuecomment-12810762.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

minrk · 2013-01-29T00:04:10Z

it's already done here, go ahead and have a look. It's basically just copy/paste the two files into the new location and updated imports.

ellisonbg · 2013-01-29T04:02:01Z

We are doing enough reorganization in this PR, some things I am wondering about...

Creating an IPython.kernel subpackage. This would be for the parts of the kernel at are zmq independent. This would be for the base kernel implementation, kernelmanager, kernelmanagerabc, and your kernel utilities (maybe as IPython.kernel.launcher or utils?)
Consolidate the zmq kernel apps into a single module, possibly combining into one class, but minimally putting them in one file.

I think this would provide a clean set of abstractions that better reflect how the code is organized. I know this expands the work for this PR a bit more, but you are much of the way there...

Cheers,

Brian

where it always should have been.

minrk · 2013-01-29T04:16:59Z

Sure, so:

(checklist added to top)

move IPython.zmq.kernelmanagerabc to IPython.kernel.kernelmanagerabc
move IPython.utils.kernel to IPython.kernel.launcher / connect
move IPython.zmq.kernelmanager to IPython.kernel.kernelmanager.
move IPython.ipkernel.Kernel to IPython.kernel.kernel.
move zmq.ipkernelapp.IPKernelApp to zmq.kernelapp (I'll look at merging the classes, and see if it makes sense).

anything more than that?

ellisonbg · 2013-01-29T04:19:48Z

If most of the things in the kernel utils module are related to launching kernels, then I think that launcher is also a good module name.

ellisonbg · 2013-01-29T04:19:59Z

I have also changed a few things in the todo list...

minrk · 2013-01-29T04:33:48Z

If most of the things in the kernel utils module are related to launching kernels, then I think that launcher is also a good module name.

It is not, but I can split it into two or three files. A minority is directly for launching kernels, the rest is for dealing with connection files (locating them, writing them, etc.), and connecting to kernels (ssh tunnels, etc.). In any case, I think the official API for these should be IPython.kernel, so the specific filename or number of files is not so important.

minrk · 2013-01-29T04:35:30Z

I disagree with moving zmq.kernel and zmq.kernelmanager into IPython.kernel. These are 100% zmq-specific. I think only if/when we build zmq-less base classes should they go into IPython.kernel.

ellisonbg · 2013-01-29T04:50:28Z

I think splitting utils into a few files would make things easier to find
when developing.

I forgot about what the Kernel object had in it - you are right that it is
full zmq.

The reason I thought about moving kernelmanager out of zmq is that it is
used for launching kernels of any type - ipython, ruby, etc. I more
picture the stuff in zmq as being our particular zmq based kernel. When
I think about there being a bunch of different kernel implementations, I
ask myself "where does our kernel live" I think the answer is in
IPython.zmq. I am even imagining that someday we might split it out as a
separate project. In that context, kernelmanager is much more generic
foundational code as it start and manages channels for any kernel. I don't
deny that it does include zmq enabled stuff though, so if we want to
localize all of our zmq code there that makes sense. Although there are a
lot of other places where zmq is used (parallel, qtconsole, etc.). Does
this make sense.

On Mon, Jan 28, 2013 at 8:35 PM, Min RK notifications@github.com wrote:

I disagree with moving zmq.kernel and zmq.kernelmanager into
IPython.kernel. These are 100% zmq-specific. I think only if/when we build
zmq-less base classes should they go into IPython.kernel.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2854#issuecomment-12820175.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

minrk · 2013-01-29T05:31:57Z

It sounds like your argument is backwards. All the stuff in IPython.zmq is specific to our kernel, and you are suggesting that we create a new module called "IPython.kernel" that has everything but code directly involved in the kernel.

Although there are a lot of other places where zmq is used (parallel, qtconsole, etc.). Does this make sense.

If we move kernelmanager to IPython.kernel, zmq.session will be the only import from IPython.zmq in either the notebook or qtconsole (aside from a couple of classes used to generate default config files).

minrk · 2013-01-29T05:50:48Z

I should say - your argument does make sense, in that IPython.zmq contains 'our zmq-based IPython kernel', and these other things like KernelManager might belong somewhere else. I just think 'IPython.kernel' isn't the right name for it, since there is something called the "IPython kernel" and IPython.kernel is strictly code that is not that kernel.

There is one point though - if the zmq KernelManager belongs in this new package, but the zmq Kernel lives in IPython.zmq, that would suggest that the inprocess KernelManager and Kernel do not themselves belong together in the same subpackage, right?

To my mind, we have subpackages containing an implementation of a Kernel (currently zmq and inprocess), and their unique peers (associated KernelManagers) are also defined adjacently. Things that belong in IPython.kernel should be strictly reusable, which the zmq KernelManager certainly is not - it can only talk to kernels that talk the wire protocol (distinct from the message protocol in general) as implemented in zmq.Session. In this way, IPython.kernel makes sense as the generic kernel-related code, IPython.zmq is zmq-specific kernel-related code, etc. The KernelManager is 100% zmq-specific. If we do build a generic zmq-free base class, then that would belong in IPython.kernel under this scheme.

- merged IPKernelApp into KernelApp, they are no longer separate classes - embed_kernel moved to its own file - ipkernel now only contains the Kernel class - associated imports updated

minrk · 2013-01-29T06:31:32Z

I'v re-read, and I just realized that you want both Kernel and KernelManager in IPython.kernel. I think that actually makes sense. If we do end up getting the zmq-dependent functionality out of the base Kernel and KernelManager classes, then we will add the zmq-specific subclasses back to IPython.zmq, and IPython.kernel can be zmq-free.

minrk · 2013-01-29T06:33:09Z

quick question: embed_kernel was in zmq.ipkernel, but I gave it its own file. Should that file now be in IPython.kernel or IPython.zmq? I'm leaning towards IPython.kernel.

minrk · 2013-01-29T06:34:11Z

I also presume blockingkm should come along too? I don't see a reason why blockingkm would not live adjacent to km.

ellisonbg · 2013-01-29T06:35:44Z

I am feeling pretty sick tonight so I am going to bed right now. But maybe
we chat IRC about this tomorrow afternoon. In the meantime if you want
keep on coding, just use your best judgement.

On Mon, Jan 28, 2013 at 10:34 PM, Min RK notifications@github.com wrote:

I also presume blockingkm should come along too? I don't see a reason why
blockingkm would not live adjacent to km.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2854#issuecomment-12822389.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

Queue.get(timeout=None) has stupid uninterruptible behavior, so wait for a week instead.

minrk · 2013-01-29T06:59:21Z

Feel better! I'm still 50/50 on moving Kernel and KernelManger from zmq to kernel, so I will wait until we chat.

Here's the gist, as I see it:

it makes sense to have the base Kernel and KernelManager in IPython.kernel, and currently the ZMQ versions are our bases
however
KernelManager, as it is, is tightly bound to the zmq wire protocol as implemented in zmq.Session
the same is true of Kernel
BlockingKernelManager should be adjacent to KernelManager
embed_kernel and IPKernelApp should be adjacent to Kernel
it does make some sense for KernelManager to not be adjacent to Kernel, but IPython.kernel is not an appropriate name for the non-kernel part of that equation.

minrk · 2013-01-30T03:35:52Z

Okay, @ellisonbg, I think this is ready for a look.

ellisonbg · 2013-01-30T22:49:26Z

IPython/kernel/kernelmanager.py

+        """
+    )
+    def _kernel_cmd_changed(self, name, old, new):
+        print 'kernel cmd changed', new


Is this a debug message?

it certainly is. Fixed.

ellisonbg · 2013-01-30T22:55:52Z

OK I have looked through the code, run the test suite and run all of the main ipython applications. Everything works perfectly. I really like how this turned out and think it is such a huge improvement. The code is much more transparent and logically organized. There may be bug that this PR is introducing, but I think the only way we will find them is by using it. Because of this, I think we should merge this unless you want to get some other eyes on this.

ellisonbg · 2013-01-31T17:59:13Z

I think this is ready to go - I would go ahead with the merge.

Move kernel code into IPython.kernel inprocess Kernel in kernel.inprocess zmq Kernel in kernel.zmq KernelManager stuff and ulities in top-level kernel Main functional change: allow custom kernel Popen command - [x] adds `KernelManager.kernel_cmd` configurable for launching a custom kernel - [x] splits entry_point.base_launch_kernel into two steps: making the launch cmd and launching the subprocess - [x] figure out where the entry_point functions belong, if it should be anywhere else - [x] move IPython.zmq.kernelmanagerabc to IPython.kernel.kernelmanagerabc - [x] move IPython.lib.kernel/IPython/zmq.entry_point to IPython.kernel.launcher / connect - [x] move zmq.ipkernelapp.IPKernelApp to zmq.kernelapp (I'll look at merging the classes, and see if it makes - [x] move IPython.zmq to IPython.kernel.zmq - [x] move IPython.inprocess to IPython.kernel.inprocess - [x] move embed_kernel from zmq.ipkernelapp to zmq.embed - [x] move MultiKernelManager to IPython.kernel.multikernelmanager. - [x] move IPython.zmq.blockingkernelmanager to IPython.kernel.blockingkernelmanager. - [x] move IPython.zmq.kernelmanager to IPython.kernel.kernelmanager. - [x] move IPython.ipkernel.Kernel to IPython.kernel.kernel.

A few changes left out from PR ipython#2854 prevented pager or set_next_input (%load) from working in the notebook.

Move kernel code into IPython.kernel inprocess Kernel in kernel.inprocess zmq Kernel in kernel.zmq KernelManager stuff and ulities in top-level kernel Main functional change: allow custom kernel Popen command - [x] adds `KernelManager.kernel_cmd` configurable for launching a custom kernel - [x] splits entry_point.base_launch_kernel into two steps: making the launch cmd and launching the subprocess - [x] figure out where the entry_point functions belong, if it should be anywhere else - [x] move IPython.zmq.kernelmanagerabc to IPython.kernel.kernelmanagerabc - [x] move IPython.lib.kernel/IPython/zmq.entry_point to IPython.kernel.launcher / connect - [x] move zmq.ipkernelapp.IPKernelApp to zmq.kernelapp (I'll look at merging the classes, and see if it makes - [x] move IPython.zmq to IPython.kernel.zmq - [x] move IPython.inprocess to IPython.kernel.inprocess - [x] move embed_kernel from zmq.ipkernelapp to zmq.embed - [x] move MultiKernelManager to IPython.kernel.multikernelmanager. - [x] move IPython.zmq.blockingkernelmanager to IPython.kernel.blockingkernelmanager. - [x] move IPython.zmq.kernelmanager to IPython.kernel.kernelmanager. - [x] move IPython.ipkernel.Kernel to IPython.kernel.kernel.

A few changes left out from PR ipython#2854 prevented pager or set_next_input (%load) from working in the notebook.

minrk mentioned this pull request Jan 26, 2013

add ability to bind to a ruby kernel #2643

Closed

minrk mentioned this pull request Jan 28, 2013

find_connection_file is moving ivanov/vim-ipython#51

Merged

minrk added 7 commits January 28, 2013 20:06

split base_launch_kernel into make_kernel_cmd / launch_kernel

ce41e2c

remove obsolete launch_kernel in ipkernel

10878ea

allow custom kernel_cmd in KernelManager

4949988

allow KM._launch_kernel to be overridden in subclasses

2942ebe

s/make_kernel_cmd/make_ipkernel_cmd/

7d49969

move IPython.lib.kernel to IPython.utils.kernel

85f7734

where it always should have been.

move IPython.zmq.entry_point to IPython.utils.kernel

395ed45

minrk added 2 commits January 28, 2013 20:23

move utils.kernel (formerly entry_point and lib.kernel) to kernel.util

980674d

move kernelmanagerabc into IPython.kernel

704b7e0

split kernel.util into kernel.connect and kernel.launcher

fa603a8

minrk added 2 commits January 28, 2013 22:25

test IPython.kernel

896b92c

move IPKernelApp from zmq.ipkernel to zmq.kernelapp

891a79b

- merged IPKernelApp into KernelApp, they are no longer separate classes - embed_kernel moved to its own file - ipkernel now only contains the Kernel class - associated imports updated

never use Queue.get(timeout=None)

d72b217

Queue.get(timeout=None) has stupid uninterruptible behavior, so wait for a week instead.

minrk added 4 commits January 29, 2013 14:19

move zmq.KernelManagers into IPython.kernel

52c3240

move multikernelmanager to IPython.kernel

8c06dce

mv IPython.zmq to IPython.kernel.zmq

37f3225

move IPython.inprocess to IPython.kernel.inprocess

4d24518

define and test IPython.kernel public API

d18872e

ellisonbg reviewed Jan 30, 2013
View reviewed changes

remove debug print statement

53080ea

minrk merged commit 5db1672 into ipython:master Jan 31, 2013

minrk added a commit to minrk/ipython that referenced this pull request Feb 1, 2013

fix payload keys

d72667c

A few changes left out from PR ipython#2854 prevented pager or set_next_input (%load) from working in the notebook.

minrk mentioned this pull request Feb 1, 2013

fix payload keys #2872

Merged

Carreau mentioned this pull request Feb 17, 2013

From rmagic to an R console #2947

Closed

minrk deleted the 2853b branch March 31, 2014 23:36

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this pull request Nov 3, 2014

fix payload keys

4f0662e

A few changes left out from PR ipython#2854 prevented pager or set_next_input (%load) from working in the notebook.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move kernel code into IPython.kernel #2854

Move kernel code into IPython.kernel #2854

minrk commented Jan 26, 2013

Carreau commented Jan 26, 2013

Carreau commented Jan 26, 2013

minrk commented Jan 28, 2013

ellisonbg commented Jan 28, 2013

minrk commented Jan 29, 2013

ellisonbg commented Jan 29, 2013

minrk commented Jan 29, 2013

ellisonbg commented Jan 29, 2013

ellisonbg commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 29, 2013

ellisonbg commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 29, 2013

ellisonbg commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 30, 2013

ellisonbg Jan 30, 2013

minrk Jan 31, 2013

ellisonbg commented Jan 30, 2013

ellisonbg commented Jan 31, 2013

Move kernel code into IPython.kernel #2854

Move kernel code into IPython.kernel #2854

Conversation

minrk commented Jan 26, 2013

Carreau commented Jan 26, 2013

Carreau commented Jan 26, 2013

minrk commented Jan 28, 2013

ellisonbg commented Jan 28, 2013

minrk commented Jan 29, 2013

ellisonbg commented Jan 29, 2013

minrk commented Jan 29, 2013

ellisonbg commented Jan 29, 2013

ellisonbg commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 29, 2013

ellisonbg commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 29, 2013

ellisonbg commented Jan 29, 2013

minrk commented Jan 29, 2013

minrk commented Jan 30, 2013

ellisonbg Jan 30, 2013

Choose a reason for hiding this comment

minrk Jan 31, 2013

Choose a reason for hiding this comment

ellisonbg commented Jan 30, 2013

ellisonbg commented Jan 31, 2013