Merge divergent Kernel implementations by minrk · Pull Request #1630 · ipython/ipython

minrk · 2012-04-19T04:40:41Z

Work in progress.

The principal change is to merge the Kernel implementation used in IPython.parallel into that in ipkernel.

Changes involved in that:

Use IOLoop/ZMQStreams in the Kernel
add apply_request handler
allow multiple input streams (control, mux, task)

Related changes along the way:

add support for silent in run_cell(). Previously, execute_requests used run_code, which meant it didn't support IPython syntax.
parallel execute() uses execute_request instead of apply

~~Changes still to make:~~

pykernel has been removed, but various related configuration options remain, and should be scrubbed.

I don't know if this code belongs in 0.13 or should go in first thing in 0.14.dev. It's a pretty big change, and test coverage for this code is still relatively poor.

fperez · 2012-04-19T06:20:54Z

Great, thanks for getting this in motion. I'll start digging into it.

takluyver · 2012-04-19T09:59:22Z

Re pykernel removal: Am I right that pykernel was our pure-Python kernel, without all the IPython secret sauce? Is there something similar that replaces it? It strikes me as potentially useful, both as an easy to understand example, and as a Python execution server for non-interactive use.

minrk · 2012-04-19T17:55:48Z

It's removed for a few reasons:

it s ~never used
it is frequently broken
it fails to comply with the message spec quite a bit
it is almost entirely duplicate code of ipkernel

For use as an example, one can always find it in 0.12.

takluyver · 2012-04-19T18:01:52Z

OK, that's fair enough.

minrk · 2012-04-23T19:31:14Z

references removed to pykernel from Qt and KernelApp

minrk · 2012-04-24T22:44:51Z

#1647 revealed an issue in streamkernel, which would necessarily conflict quite severely with this, as that file has been removed. So a fix for that is now included.

minrk · 2012-05-08T01:29:23Z

Test results for commit c1a2747 merged into master
Platform: darwin

python2.6: OK (libraries not available: matplotlib pymongo qt tornado wx wx.aui)
python2.7: OK
python3.2: Failed, log at https://gist.github.com/2631856 (libraries not available: matplotlib pymongo qt tornado wx wx.aui)

Not available for testing: python3.1

takluyver · 2012-05-08T09:55:30Z

The test does seem to be correctly picking up a stray reference to streamkernel - I don't know why it only appears on Python 3.

https://github.com/minrk/ipython/blob/0c2728a924dcb7bdcbddb07f40ca7545186ee1a5/IPython/parallel/apps/ipengineapp.py#L47

minrk · 2012-05-08T20:54:44Z

Indeed, apparently I messed up a bit in my last rebase.

minrk · 2012-05-08T20:57:26Z

Test results for commit fc1af3c merged into master
Platform: darwin

python2.6: OK (libraries not available: matplotlib pymongo qt tornado wx wx.aui)
python2.7: OK
python3.2: Failed, log at https://gist.github.com/2639226

Not available for testing: python3.1

takluyver · 2012-05-08T22:47:17Z

I don't know what's happening with that test failure - it looks like the file was installed without 2to3 having run.

minrk · 2012-05-08T22:56:08Z

Yes, I will make a PR against test_pr to fix that. Python 3.2 is re-using the build dir from 2.7, so it skips a bunch of 2to3 and I have to rm -rf build between builds.

takluyver · 2012-05-08T23:03:28Z

OK - I wonder why I don't see that. Maybe a Linux/Mac difference, but I'm not sure quite how.

minrk · 2012-05-08T23:07:03Z

Yes, I think it is. The build dir gets a 3.2 suffix on Linux, but not on OS X. Only seems to affect pure-Python packages.

minrk · 2012-05-09T01:27:50Z

Test results for commit 386ef09 merged into master
Platform: darwin

python2.6: OK (libraries not available: matplotlib pymongo qt tornado wx wx.aui)
python2.7: OK (libraries not available: matplotlib pymongo tornado wx wx.aui zmq)
python3.2: OK (libraries not available: matplotlib pymongo qt tornado wx wx.aui)

Not available for testing:

first step in merging kernel/streamkernel

prevent unnecessary formatting of log messages not displayed.

some eventloop integration updated to match

rather than specifically IPKernelApp (%pylab now works in Engines)

and update %autopx tests, now that %autopx uses silent=False

allows classes to be passed log attribute, avoiding duplicate setup and occasional errors if things are done in the wrong order.

for using an Engine's Kernel with connecting frontends.

instead of [sys.exe, module_path], which resulted in the modules being `__main__`, which caused problems.

minrk · 2012-05-23T05:32:19Z

Okay, it seems to be working now. To confirm, you can do:

$> ipcluster start -n 2

from IPython import parallel
rc = parallel.Client()
dv = rc[:]
dv.block = True
dv.activate()

%px from IPython.parallel import bind_kernel; bind_kernel()
%px %qtconsole

Which sets up your engines as full kernels, and then launches a QtConsole attached to each one.

ellisonbg · 2012-05-23T06:11:28Z

This is fantastic! Do we want to push this one step further and have
a parallel magic that does this automatically for an engine on the
active direct view?

On Tue, May 22, 2012 at 10:32 PM, Min RK
reply@reply.github.com
wrote:

Okay, it seems to be working now. To confirm, you can do:

$> ipcluster start -n 2
from IPython import parallel
rc = parallel.Client()
dv = rc[:]
dv.block = True
dv.activate()

%px from IPython.parallel import bind_kernel; bind_kernel()
%px %qtconsole
Which sets up your engines as full kernels, and then launches a QtConsole attached to each one.

Reply to this email directly or view it on GitHub:
#1630 (comment)

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

fperez · 2012-05-23T07:19:56Z

This is indeed all kinds of awesome. Great job, Min!!

minrk · 2012-05-25T00:29:10Z

This is fantastic! Do we want to push this one step further and have
a parallel magic that does this automatically for an engine on the
active direct view?

I'm not sure. It's a lot more common for engines to be remote than regular IPython kernels, and running %qtconsole on a remote engine (or 100 of them!) would not be desirable. If I were to do this, I would want to open a console on the calling machine, connected to the engine, even if it is remote.

In any case, adding to the parallel magics should probably be done after this and #1732 are both merged.

ellisonbg · 2012-05-25T19:01:39Z

OK, so I think we can merge this PR right?

On Thu, May 24, 2012 at 5:29 PM, Min RK
reply@reply.github.com
wrote:

This is fantastic! Do we want to push this one step further and have
a parallel magic that does this automatically for an engine on the
active direct view?

I'm not sure. It's a lot more common for engines to be remote than regular IPython kernels, and running %qtconsole on a remote engine (or 100 of them!) would not be desirable. If I were to do this, I would want to open a console on the calling machine, connected to the engine, even if it is remote.

In any case, adding to the parallel magics should probably be done after this and #1732 are both merged.

Reply to this email directly or view it on GitHub:
#1630 (comment)

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

minrk · 2012-05-25T22:16:03Z

Test results for commit 45a4d04 (can't merge cleanly)
Platform: linux2

python2.7: OK (libraries not available: pymongo wx wx.aui)
python3.2: OK (libraries not available: matplotlib pymongo qt wx wx.aui)

Not available for testing:

minrk · 2012-05-25T22:26:12Z

Test results for commit 45a4d04 merged into master
Platform: darwin

python2.6: OK (libraries not available: matplotlib pymongo qt tornado wx wx.aui)
python2.7: OK (libraries not available: wx wx.aui)
python3.2: OK (libraries not available: matplotlib pymongo qt wx wx.aui)

Not available for testing:

also increase timeout for kernel start to 10 seconds

minrk · 2012-05-25T23:32:35Z

the test_pr script doesn't work on Windows, but I was able to do what it does manually, and I was able to run the parallel test suite on Python2.7 and 3.2 on Windows, as well as basic fiddling with the notebook and qtconsole, so I think we are good to go.

ellisonbg · 2012-05-25T23:43:24Z

OK, merging!

Merge divergent Kernel implementations

fperez · 2012-05-25T23:51:16Z

Awesome job, @minrk! And thanks to everyone who pitched in. This is a major milestone for the project.

Merge divergent Kernel implementations

minrk mentioned this pull request Apr 24, 2012

Aborted parallel tasks can't be resubmitted #1647

Closed

minrk mentioned this pull request Apr 26, 2012

Delay flushing softspace until after cell finishes #1662

Merged

minrk added 13 commits May 13, 2012 15:09

drop in apply_request handler in zmq.ipkernel

ff8a701

first step in merging kernel/streamkernel

lazy-formatting logs in ipkernel

09cd42a

prevent unnecessary formatting of log messages not displayed.

allow multiple shell_sockets in ipkernel

1901b79

allow aborting tasks

1587a7f

respect copy flag in Session.recv

fbf9e2e

Don't copy shell requests

d362026

move apply serialization into zmq.serialize

ed57daf

add ids to ipkernel

6174cb0

use IOLoop in ipkernel

9440f9e

some eventloop integration updated to match

use IPython.zmq.kernel in parallel Engines

24fdfb1

allow exceptions in handlers without crashing

5c5118d

use execute_request for parallel execute, instead of apply

7f6086f

scrub: ident on iopub

08b95f8

minrk added 15 commits May 22, 2012 22:07

check for any Application with Kernel in zmq.eventloop

e9a0b8e

rather than specifically IPKernelApp (%pylab now works in Engines)

test %pylab on engines

7576fb6

and update %autopx tests, now that %autopx uses silent=False

demote enter/exit eventloop messages to INFO

85e05e1

move default log setup to _log_default from init_logging

d2f71ea

allows classes to be passed log attribute, avoiding duplicate setup and occasional errors if things are done in the wrong order.

Application.log_format is a configurable

9fc6597

split init_sockets into a few steps, so they can be better reused

f24fe97

reduce parentless IOPub message to warning level

3b1700a

add listen_kernel method to IPEngineApp

23d0ff7

allow embed_kernel to be called from engine

83b4b12

kernel app cleans up connection files that it wrote

3fab994

relax some timing constraints in parallel tests

100a5e3

store whole content, instead of just display data in metadata.outputs

46db02d

revert embed_kernel changes that implied bind_kernel in engine

e0b0d5e

add IPython.parallel.bind_kernel

64f2f8d

for using an Engine's Kernel with connecting frontends.

use [sys.exe, "-c", "…launch_new_instance()"] in launchers

45a4d04

instead of [sys.exe, module_path], which resulted in the modules being `__main__`, which caused problems.

switch order of failed-to-start errors in test_embed_kernel

87af741

also increase timeout for kernel start to 10 seconds

ellisonbg added a commit that referenced this pull request May 25, 2012

Merge pull request #1630 from minrk/mergekernel

6cb2026

Merge divergent Kernel implementations

ellisonbg merged commit 6cb2026 into ipython:master May 25, 2012

minrk deleted the mergekernel branch March 31, 2014 23:36

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this pull request Nov 3, 2014

Merge pull request ipython#1630 from minrk/mergekernel

d915736

Merge divergent Kernel implementations

Uh oh!

Conversation

minrk commented Apr 19, 2012

Uh oh!

fperez commented Apr 19, 2012

Uh oh!

takluyver commented Apr 19, 2012

Uh oh!

minrk commented Apr 19, 2012

Uh oh!

takluyver commented Apr 19, 2012

Uh oh!

minrk commented Apr 23, 2012

Uh oh!

minrk commented Apr 24, 2012

Uh oh!

minrk commented May 8, 2012

Uh oh!

takluyver commented May 8, 2012

Uh oh!

minrk commented May 8, 2012

Uh oh!

minrk commented May 8, 2012

Uh oh!

takluyver commented May 8, 2012

Uh oh!

minrk commented May 8, 2012

Uh oh!

takluyver commented May 8, 2012

Uh oh!

minrk commented May 8, 2012

Uh oh!

minrk commented May 9, 2012

Uh oh!

minrk commented May 23, 2012

Uh oh!

ellisonbg commented May 23, 2012

Uh oh!

fperez commented May 23, 2012

Uh oh!

minrk commented May 25, 2012

Uh oh!

ellisonbg commented May 25, 2012

Uh oh!

minrk commented May 25, 2012

Uh oh!

minrk commented May 25, 2012

Uh oh!

minrk commented May 25, 2012

Uh oh!

ellisonbg commented May 25, 2012

Uh oh!

fperez commented May 25, 2012

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants