Skip to content
This repository

In-process kernel support (take 2) #2397

Closed
wants to merge 14 commits into from

13 participants

Evan Patterson Min RK Carlos Cordoba Thomas Kluyver hsyed Takafumi Arakaki Jonathan March Fernando Perez Brian E. Granger Bradley M. Froehle tomcraw00 Pietro Berkes Robert Kern
Evan Patterson
Collaborator

This PR is functionally equivalent to #2382 but involves substantially less refactoring. Note, however, that ZMQ is now required.

The gist referenced in the previous PR contains usage examples: https://gist.github.com/3659874.

Evan Patterson
Collaborator

@ccordoba12, I'm now responding to your questions in #2382 (comment). Sorry for the delay.

  1. The EmbeddedKernel is really only an abstraction layer to let frontends talk to the underlying InteractiveShell. If you want to get object info from elsewhere in Spyder, you should bypass the kernel and call kernel.shell.object_inspect() directly. This is an example of how the in-process kernel is extremely convenient.

  2. Here's an example:

kernel = EmbeddedKernel()
km1 = QtEmbeddedKernelManager(kernel=kernel)
km2 = QtEmbeddedKernelManager(kernel=kernel)
kernel.frontends = [km1, km2]
[assign km1 and km2 to two different frontend widgets]

This is just like the out-of-process situation: each frontend should have its own kernel manager, but the kernel managers may be connected to the same kernel. However, unlike the out-of-process situation, you must explicitly tell the kernel about the frontends.

  1. No JSON connection files are used.

  2. By default, stdout/stderr will be directed to the frontend only during execution. If you want stderr to always appear in the frontend, you can manually set sys.stderr = kernel.stderr.

IPython/zmq/displayhook.py
@@ -37,7 +37,7 @@ class ZMQShellDisplayHook(DisplayHook):
37 37
     topic=None
38 38
 
39 39
     session = Instance(Session)
40  
-    pub_socket = Instance('zmq.Socket')
  40
+    pub_socket = Any()
3
Min RK Owner
minrk added a note September 10, 2012

Instead of these Any() traits, do you think it's worthwhile to do a simple trait validator that checks for the necessary interfaces (just send_multipart, I believe)?

Robert Kern Collaborator
rkern added a note September 11, 2012

IPython requires 2.6 now, doesn't it? We could make an ABC that checks for the necessary methods, and then use a plain Instance trait.

http://docs.python.org/library/abc

Evan Patterson Collaborator

Thanks, Robert. I'll try this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Min RK
Owner

Thanks for doing this, it does make it easier to see what's actually going on. Ultimately, we certainly will want to make it work without zmq imports, because once it is up to snuff, plain single-process terminal IPython should be using the embedded Kernel/KernelManager code, so we finally have a consolidated codebase.

Evan Patterson
Collaborator

@fperez, @minrk, any word on this? Thanks.

Min RK
Owner

I don't have time for a full review, but most of it looks good. One concern is that the embedded kernel manager is almost entirely a copy/paste from the original with no inheritance. That doesn't seem right to me, and the cost of large duplicate code, particularly in a relatively unused path like this (e.g. the old pykernel), has shown itself to be quite high. Can you make the case for not reusing any code?

Also, a minor naming consideration: We have a notion of 'embedding', which is IPython running in a particular namespace. This is largely unrelated to that, where embedded means the kernel is in the same process as the frontend. Perhaps there is a better name that we are not already using to mean something else?

Evan Patterson
Collaborator

I think you should look at the kernel managers a little more carefully. In contrast to the EmbeddedKernel in my original approach, the EmbeddedKernelManager here duplicates only interface, not implementation. This is because the implementation of the (base) kernel manager is almost entirely devoted to IO loops and kernel process management, which are irrelevant to the embedded case. Were we writing in, say, Java, it would be proper to define a kernel manager interface, and have two separate implementations for in and out of process kernels. In Python, we are settling (perhaps incorrectly) for duck typing in this case. The maintenance problem is only one of maintaining a consistent kernel manager interface, which you must do in any case to avoid breaking users' code.

Also, notice that for the kernel manager subclasses (both blocking and Qt), where it does make sense to share code, I have done so through the (somewhat kludgey) technique of mixin classes.

As to the name I don't particularly care. Originally, I eschewed InProcess for being two words and hence awkward to type, but if you prefer that or something else I would be happy to change it.

Carlos Cordoba

Thanks @epatters for your explanation. It seems this really is the right approach for us. One more thing: it's not entirely clear for me if EmbeddedKernel is based or not on ZMQ. I'm asking this because we would like our users could enjoy the benefits of IPython.parallel too.

Besides, I downloaded your branch to try to play with it but it's giving me a segmentation fault if I try to run your gist or even when I'm just trying to start qtconsole. I don't know where that could be coming from, because master is working just fine.

Evan Patterson
Collaborator

EmbeddedKernel doesn't use ZMQ (though at the present time, it does import ZMQ). For the sake of frontends, it sends messages conforming to the messaging spec, but through "dummy" sockets that don't do anything. As far as you're concerned, this is just an implementation detail.

I'm afraid I know very little about IPython's parallel infrastructure. What did you have in mind?

As for the segfault---that's not good. What platform are running? What version of PySide or PyQt? Could you give a stack trace? Thanks.

Thomas Kluyver
Collaborator

Could you add an example like that in the gist to the examples directory (at least for Qt - I wouldn't promote the terminal one at present)?

Also, one key use case for this is 'embed a shell with access to Python objects already in the process' - what's the most efficient way to start a kernel with specific objects in the namespace?

hsyed

Hello, Thomas K guided me to this branch from stack overflow. I am trying to get the example code to run (https://gist.github.com/3659874#file_embedded_qtconsole.py) using this branch. But I am getting the following error :

Traceback (most recent call last):
File "ipqt.py", line 31, in
main()
File "ipqt.py", line 14, in main
km.start_kernel()
File "../ipython/IPython/frontend/qt/base_kernelmanager.py", line 223, in start_kernel
self.started_kernel.emit()
AttributeError: signal was not defined in the first super-class of class 'QtEmbeddedKernelManager'

I'm not a python expert, or a QT expert (C/C++ is my Forté).

I'm also interested in what Thomas is talking about in the previous comment --i.e., passing objects from my pyqt app into the ipython embedded shell.

Evan Patterson
Collaborator

@takluyver, I'll work on this. I have a question for you, though. What is the best way to modify/replace the user namespace using InteractiveShell? Because that's basically all you would do.

@hsyed, are you running PyQt? If so, what version? I only tested the code under PySide---probably a terrible mistake.

Thomas Kluyver
Collaborator

@epatters : If possible, the simplest way is to pass either user_ns or user_module when InteractiveShell is instantiated. If not, I think this should work:

shell.init_create_namespaces(user_ns={'foo': 1})
# You could alternatively pass `user_module`, which should be
# a module-like object, so user_module.foo works

# To add IPython features like `exit`, `In`, `get_ipython` etc.:
shell.init_user_ns()
hsyed

@epatters Yes I am using PyQt. I suppose I could install pyside. But it will probably go pear shaped since my code runs PyQt ? PyQt is at 4.9.4.

Evan Patterson
Collaborator

@hsyed, PyQt should absolutely work. I'm going to test it out right now.

Evan Patterson
Collaborator

Or, what I mean to say, is that PyQt is supposed to work.

Evan Patterson
Collaborator

@hsyed, this should fix the PyQt4 problems (including yours, @ccordoba12).

Evan Patterson
Collaborator

@takluyver, the reason I ask is that EmbeddedKernel and EmbeddedKernelManager really only exist to present a consistent interface to the frontends. If you want to do anything programatically--like modify the user namespace--you should just use the underlying InteractiveShell (accessible from the shell attribute of EmbeddedKernel). There is no magic here, which is what makes the in-process kernel very convenient (if unsafe).

I assumed that updating the user namespace would be as simple as

kernel.shell.user_ns.update({...})

Is that right?

Thomas Kluyver
Collaborator

Good point, thanks. In fact, there's a shell.push(dict) method which does that and takes care of hiding or unhiding variables. Could we add that to the example from your gist and put it in docs/examples/?

Evan Patterson
Collaborator

Great, that's exactly what I wanted to know. I'll add an updated example to the docs.

Evan Patterson
Collaborator

I wasn't sure exactly where to put this, so I created a 'frontend' directory.

hsyed

I haven't tried the code yet. But I was wondering if I could share references between the two interpreters ? Say if I wanted to pass a reference to some functions between a GUI app and an embedded interpreter, would this work ?

Evan Patterson
Collaborator

@hsyed, you could indeed share references between the two interpreters, provided that everything remains in process.

Thomas Kluyver
Collaborator

@hsyed: That's what I was just discussing with @epatters. Have a look at the example that's now in the pull request, specifically the kernel.shell.push() line. You can simply push a dictionary referring to whatever functions you want to expose in the shell.

As a technical detail, there's only one interpreter, used by both the PyQt application and the in-process kernel. So you just need to add the relevant objects to the namespace that the IPython shell is using. That's what the push method does.

hsyed

@takluyver fantastic, Was a bit confused since technically one could always serialize a dictionary across using pass-by-copy :D

hsyed

@epatters The two examples of starting a embedded kernel with a Qt Controll differ a bit. one of them does not have a start_kernel() call. I looked into the call and it makes sense for the new example not to have that call since it creates a kernel implicitly. Whereas the new example creates the kernel and pushes arguments onto it.

perhaps start_kernel() should be renamed initialise_default_kernel or setup_default_kernel, or start_kernel() be made mandatory in both examples regardless of weather a kernel was user suplied or default constructed ?

Evan Patterson
Collaborator

@hsyed, the embedded kernel manager implements an interface largely identical to IPython.zmq.kernelmanager.KernelManager. I had to give start_kernel() an implementation, so I made it do the obvious thing. Likewise, stop_kernel() and kill_kernel() don't really do much. The only advantage to doing this is that it makes in- and out-of-process kernels trivially interchangeable (under idealized circumstances), as in my original gist.

I wouldn't worry about this too much. You can create a Kernel however you please.

hsyed

@epatters does app = guisupport.get_app_qt4() get the app object of the embedding app's Qt runloop ? I'm wondering if I need to change my apps initialization (standard Qt initialization).

Edit
After looking at the implementation code, it seems that it plays nicely with QT's existing runloop machinery.

Evan Patterson
Collaborator

Right, no event loop trickery is required.

hsyed

@epatters I am trying to embed the RichIPythonWidget into a QFrame (since I want to be able to resize the console and perhaps add some sibling widgets) but I am not getting a prompt, just an empty box. Am I doing something wrong ?

http://gist.github.com/3747364

Evan Patterson
Collaborator

Yes, you're setting the private attribute _kernel_manager instead of the public property kernel_manager on the frontend widget.

hsyed

@epatters yep that solved it (sorry for the silly mistake). I tried the following approach as well before you replied:

class Widget2(RichIPythonWidget):
    def __init__(self):
        self._kernel = EmbeddedKernel()
        self._kernel_manager = QtEmbeddedKernelManager(kernel = self._kernel)
        self._kernel_manager.start_channels()
        self._kernel.frontends.append(self._kernel_manager)
        app = guisupport.get_app_qt4()
        self.exit_requested.connect(app.quit)
        self.kernel_manager = self._kernel_manager

app = QApplication(sys.argv)
iew = Widget2()
iew.show()
app.exec_()
sys.exit()

This threw the following exception :

TypeError: pyqtSignal must be bound to a QObject, not 'Widget2'

This might be another PyQt4 - PySide issue ?

edit

Sorry, I forgot to initialize the super class the correct code is :

class Widget2(RichIPythonWidget):
    def __init__(self):
        RichIPythonWidget.__init__(self)
        self._kernel = EmbeddedKernel()
        self._kernel_manager = QtEmbeddedKernelManager(kernel = self._kernel)
        self._kernel_manager.start_channels()
        self._kernel.frontends.append(self._kernel_manager)
        app = guisupport.get_app_qt4()
        self.exit_requested.connect(app.quit)
        self.kernel_manager = self._kernel_manager

but this gives :

  File "../../ipython/IPython/frontend/qt/base_frontend_mixin.py", line 28, in _set_kernel_manager
    old_manager.started_kernel.disconnect(self._started_kernel)
TypeError: 'instancemethod' object is not connected
hsyed

@epatters Another Issue now that I am trying to embed the interpreter into my main application. I get the following error:

Traceback (most recent call last):
  File "MainApp.py", line 10, in <module>
    from IpythonEmbeddedWidget import IpythonEmbeddedWidget
  File "/Users/hassan/code/research/analasys gui/IpythonEmbeddedWidget.py", line 5, in <module>
    from IPython.frontend.qt.console.rich_ipython_widget import RichIPythonWidget
  File "../../ipython/IPython/frontend/qt/console/rich_ipython_widget.py", line 15, in <module>
    from IPython.external.qt import QtCore, QtGui
  File "../../ipython/IPython/external/qt.py", line 35, in <module>
    prepare_pyqt4()
  File "../../ipython/IPython/external/qt.py", line 20, in prepare_pyqt4
    sip.setapi('QString', 2)
ValueError: API 'QString' has already been set to version 1
Evan Patterson
Collaborator

Both errors are really the same error, I think. The problem is that PyQt4 has two APIs: an old-style API (version 1) and a new-style API (version 2). On Python 2, the default is old-style, but the IPython client-side machinery requires version 2 (which is the version you're trying to use in your first snippet).

The easiest fix is to just import Qt from IPython.external.qt, which will automatically set the SIP API. Otherwise, you must set the SIP API to version 2 before importing anything from Qt. For more info, see

http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/html/incompatible_apis.html#ref-incompat-apis

Evan Patterson
Collaborator

Specifically, you need to set the API for QString and QVariant:

https://github.com/ipython/ipython/blob/master/IPython/external/qt.py

hsyed

Ok, The shell is integrated, and it has been quite painless. I'll start gluing an interface to my app tomorrow.

Amazing work Evan. Previously I was trying to embed a python interpreter into a QTextEdit and I got it working, but it was no where near as nice as IPython.

Evan Patterson
Collaborator

I'm glad everything is working. (We actually use a QTextEdit under the hood, but it was a very large development effort to make it work so well.)

hsyed

@epatters It seems I have to kernel.shell.push() a namespace dictionary if I make a change to it. If I do not the state of the shell is not updated (code below self["active_topic"] remains referring to the empty_topic).

Am I reasoning about things correctly, or is there something about namedtuple's I am missing ? I understand that named tuple's aren't mutable but shouldn't self["active_topic"] have reference semantics ? Hopefully I am not making silly mistakes again.

from collections import namedtuple

nt_active_topic = namedtuple("active_topic",['name','posts','views','active_page','db_interface'])
empty_active_topic = nt_active_topic(None,None,None,None,None)

class GuiToPythonInterface(dict):
    def __init__(self):
        self["active_topic"] = empty_active_topic

    def set_active_topic(self, active_topic):
        self["active_topic"] = active_topic
        self._kernel.shell.push(self)


    def set_kernel(self,kernel):
        self._kernel = kernel
Thomas Kluyver
Collaborator

Hi @hsyed . Your trouble is that IPython is maintaining a separate namespace, so to change a non-mutable value, you need to update the namespace. You can interact with it more directly by setting kernel.shell.user_ns['active_topic']. You can keep a reference to kernel.shell.user_ns (which is an ordinary dict) if you like.

Alternatively, if your application already constructs a dictionary that you want to expose as the namespace, you can make IPython use it with this code snippet from one of my earlier comments. This is slightly more likely to break things, because the change is more complex, but it should work:

shell.init_create_namespaces(user_ns=mydict)
# You could alternatively pass `user_module`, which should be
# a module-like object, so user_module.foo works

# To add IPython features like `exit`, `In`, `get_ipython` etc.:
shell.init_user_ns()
Carlos Cordoba

Sorry @epatters for the long delay in answering, I've been quite busy. Your bug fix commit solved the problem for me, so many thanks for that. I also tested some very simple IPython.parallel examples and they work as expected.

But I've noticed one unfortunate thing: while the widget is running some computations, it's completely blocked, i.e. it can't be scrolled or accept Ctrl+C. @epatters, is this to be expected? My guess is that it is, because kernel and app live in the same process. And so, we couldn't use this approach in Spyder, because that would block the entire app while a console is running. Even worst, if the user sends an infinite loop to the console, he'd have to hard kill the app and restart all over again.

Thomas Kluyver
Collaborator

What do you do to avoid blocking the UI at present in Spyder? I think that was one of the drivers for the two-process approach in the first place.

Evan Patterson
Collaborator

@ccordoba12, you have identified the fundamental tradeoff between the in- and out-of-process models. It is inevitable that the in-process kernel will block. That is the price you pay for the convenience of having it in process. That is why the out-of-process model is preferable if you can make it work for your application.

As long as you don't need deep integration with the kernel's namespace (like @hsyed seems to), there should be no real obstacles to using an out-of-process kernel.

hsyed

@takluyver yep that is indeed a sound enough approach to the problem, fits in nicely with the way I am doing things (in terms of efficiency and non-hackyness).

Carlos Cordoba

@epatters, thanks for the confirmation. It's far more simple to manage in-process kernels than out-ones, but the blocking aspect of it represent a no go for us.

@takluyver, Pierre also developed a two process architecture (with raw sockets), and he used it to embedded a non-blocking IPython 0.10 console, which was perhaps one of the most impressive features of Spyder 2.0 (when it was launched 2 years ago :-). Now we are using that same architecture as an intermediate layer to communicate with IPython kernels. Problem is this has become quite complex and I was looking for a way to reduce that complexity. Since I don't know very well that part of Spyder code, I wasn't sure if @epatters approach would work for us or not.

Thomas Kluyver
Collaborator

Hmm, I must have misunderstood. I thought Spyder wanted an in-process console so it could easily work alongside things like the variable explorer. If you already use out-of-process kernels, what's stopping you using the kernelmanager interface and Qt console widget that existed before this PR?

Evan Patterson
Collaborator

Yes, I don't understand that either.

Carlos Cordoba
Carlos Cordoba
Takafumi Arakaki

You can send raw python object from kernel to frontend, using data_pub
http://ipython.org/ipython-doc/dev/development/messaging.html#raw-data-publication

I used it to send pickled matplotlib figure to frontend
#2322 (comment)

I don't know if IPython supports the other direction. (Well, you can base64-encode pickled string then send it as a JSON object...)

Takafumi Arakaki

Ah, as IPython parallel sends some code to engines, I suppose the protocol to send pickled object already exists. I don't know where it is though.

Thomas Kluyver
Collaborator
Carlos Cordoba

Thanks a lot @tkf for the references. I didn't know that was already possible.

Carlos Cordoba

Ok @takluyver, I'll do that to not disrupt this PR.

Evan Patterson
Collaborator

Speaking of this PR, I think it's ready to be merged. I responded to @minrk's comments up thread. @minrk, did you find my reply satisfactory?

Jonathan March
Collaborator

@minrk, @takluyver, @fperez - what would be an appropriate way to proceed to move this PR toward merging, or toward resolving any problems with it? I recognize that this is non-trivial to review. Thanks!

Fernando Perez
Owner

Sorry for having been totally derelict on this one, despite my best intentions (and private promises to @minrk...). Let me try to go over it, I know it's hugely important and we're super thankful to you guys for the amount of feedback and rework you've put in.

I'm now publicly putting myself on the shame block if I don't review it by this weekend...

docs/examples/frontend/embedded_qtconsole.py
... ...
@@ -0,0 +1,41 @@
1
Fernando Perez Owner
fperez added a note September 24, 2012

This example should have a reasonable module-level docstring that explains what it does. Examples are meant to be used as starting points by others for their own projects, so they should be overly verbose in comments and useful information. The rest of the code is well commented, but a top-level docstring is necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
docs/examples/frontend/embedded_qtconsole.py
((17 lines not shown))
  17
+    kernel.frontends.append(km)
  18
+
  19
+    # Create the Qt console frontend.
  20
+    control = RichIPythonWidget()
  21
+    control.exit_requested.connect(app.quit)
  22
+    control.kernel_manager = km
  23
+    control.show()
  24
+
  25
+    # Execute some code directly. Note where the output appears.
  26
+    kernel.shell.run_cell('print "x=%r, y=%r, z=%r" % (x,y,z)')
  27
+
  28
+    # Execute some code through the frontend (once the event loop is
  29
+    # running). Again, note where the output appears.
  30
+    do_later(control.execute, '%who')
  31
+
  32
+    guisupport.start_event_loop_qt4(app)
1
Fernando Perez Owner
fperez added a note September 24, 2012

Any particular reason why once it's running, I can't call %pylab without getting an error? I was expecting that would work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Fernando Perez
Owner

Picking up the discussion above, I'm not sure I like the name 'embedded' for this module for the same reasons that @minrk brought up. InProcess is a bit of a mouthful, but perhaps a more faithful representation of what's going on... Thoughts?

As for the inheritance/mixin business, it sounds to me like the right solution would be to create a python abstract base class for the main interface, and then have in/out of process kernel managers implement that interface.

But I'm not saying that needs to be done for this PR, just thinking out loud for the future. We can land this now, and refine it in the future after we beat on it a bit more in the wild.

The code is very clean, as we've come to expect from @epatters's excellent contributions. Thanks a lot for tackling this, one of the long-standing 'big ticket' items we've had! I'm very excited to see this go in.

Remaining points:

  • see a few inline comments I made. I'm particularly worried as to why %pylab doesn't work, as I think it should. That kernel should behave like any other ipython...

  • this is important enough that it needs an entry in the what's new file in the docs.

  • Crucially, there are no tests at all for this entire new submodule, and that just won't fly. We can't put in an entire new subdirectory with this much new code and zero testing. This is fortunately code that is easier to test, as it can be managed purely programatically (no GUI code mess to worry about like we have with the Qt console). I don't know how easy it would be to wire this kernel and hit it with part/all of the existing testing machinery, so that we're always sure that kernels managed this way behave normally. I say this due to my surprise with the %pylab failure. But at least, we need some tests for the actual methods that are here in this new code.

So I think that we have a few very easy to address issues, and the slightly more involved, but crucial, issue of getting some testing for this code. Hopefully it won't be too hard to get that done, as the contribution is certainly a great one.

Evan Patterson
Collaborator

@fperez, thanks for the detailed review. These are all reasonable criticisms and I will be working to incorporate them over the next day or so.

Evan Patterson
Collaborator

@fperez, I have addressed all of the issues except for testing, which I am about to begin. I intend to write some basic tests invoking the in-process kernel and kernel managers, but I'm not sure what you meant by "hit it with part/all of the existing testing machinery". As far as I can tell, none of the code in IPython.zmq is particularly well tested. Am I missing something?

Fernando Perez
Owner

@epatters, I meant that since this can give a fresh ipython kernel in a process, one could use it to run a lot of the tests we already have for 'the real ipython'. If you look through our tests, in many places they access a global ipython either by explicitly requesting it via ip = get_ipython() or by relying on the fact that we inject that same variable into the global namespace as _ip for convenience in tests.

So what I meant to say was that we should consider rerunning entire batches of tests that already exist and use this global _ip to instead use a freshly created object like this, which is now easy to instantiate in isolation.

But I don't necessarily think that must happen in this PR, as I suspect the practicalities of that would require some cleanup and retooling of the test machinery. I'm just thinking about where our design moves forward thanks to this work.

As long as we can get solid test coverage of the new module by itself, we're good. I would encourage you to run with the tests using the --with-coverage --cover-package=IPython flag so that you get a sense of what you're missing. Ideally we'd hit 100% coverage for new code; in practice that may be impossible, but it's not unrealistic to shoot for high 90's at least (and in my experience, I've often found that last 1% of missing coverage to reveal bugs once I add a test for it, so don't dismiss the idea outright).

I don't want to sound like I'm piling on you with the test requirements, in fact I'd like to update our dev guide to be more explicit and strict about improving test coverage for all new code. At the very least new code should never regress our test coverage, and I want to see it always improve it.

Evan Patterson epatters closed this September 26, 2012
Evan Patterson
Collaborator

Thanks for the explanation, @fperez. I'm not very familiar with IPython's testing machinery, so I'll leave that task for someone else.

I have, however, added some more tests. Code coverage is now at 93%:

epatters$ nosetests --with-coverage --cover-package=IPython.inprocess IPython/inprocess/
........
Name                                      Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------
IPython.inprocess                             0      0   100%   
IPython.inprocess.blockingkernelmanager      40      1    98%   56
IPython.inprocess.ipkernel                   81      6    93%   78, 82, 98-99, 103, 165
IPython.inprocess.kernelmanager             149     11    93%   50, 71, 221, 232, 252, 263, 278, 282, 286, 339, 410
IPython.inprocess.socket                     23      2    91%   33, 37
-----------------------------------------------------------------------
TOTAL                                       293     20    93%   
----------------------------------------------------------------------
Ran 8 tests in 0.956s

OK
Evan Patterson epatters reopened this September 26, 2012
Evan Patterson
Collaborator

Blast, GitHub! Anyway, @fperez, please let me know if anything remains to be done here.

Evan Patterson
Collaborator

Oh, and my last day before heading back to school is Friday, so we need to get this wrapped up soon.

Fernando Perez
Owner

Mmh, something's gone awry with those tests. While running them via nosetests as you did above I get the same results, if I use the official test runner, iptest, they crash catastrophically. iptest is a thin wrapper around nosetests that sets a few variables, in particular the global ipython instance. It's a bit clunky to do so, but we've found it in the past to be necessary to have an ipython instance each group of tests can rely on, so for now we're stuck with that.

So I'm afraid that before we merge this one, we'll absolutely have to figure out what's going on here and get the 'real' test suite passing. For more details on how to run our tests, see the testing section of our docs.

This is what I get:

longs[junk]> iptest IPython.inprocess
EEEEEEEE
======================================================================
ERROR: Does pylab work in the in-process kernel?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/tests/test_kernel.py", line 34, in test_pylab
    km.start_kernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/kernelmanager.py", line 369, in start_kernel
    self.kernel = InProcessKernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/ipkernel.py", line 66, in __init__
    super(InProcessKernel, self).__init__(**traits)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/zmq/ipkernel.py", line 147, in __init__
    user_ns     = self.user_ns,
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 329, in instance
    '%s are being created.' % cls.__name__
MultipleInstanceError: Multiple incompatible subclass instances of InProcessInteractiveShell are being created.

======================================================================
ERROR: Does the in-process kernel handle raw_input correctly?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/tests/test_kernel.py", line 43, in test_raw_input
    km.start_kernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/kernelmanager.py", line 369, in start_kernel
    self.kernel = InProcessKernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/ipkernel.py", line 66, in __init__
    super(InProcessKernel, self).__init__(**traits)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/zmq/ipkernel.py", line 147, in __init__
    user_ns     = self.user_ns,
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 329, in instance
    '%s are being created.' % cls.__name__
MultipleInstanceError: Multiple incompatible subclass instances of InProcessInteractiveShell are being created.

======================================================================
ERROR: Does the in-process kernel correctly capture IO?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/tests/test_kernel.py", line 57, in test_stdout
    kernel = InProcessKernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/ipkernel.py", line 66, in __init__
    super(InProcessKernel, self).__init__(**traits)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/zmq/ipkernel.py", line 147, in __init__
    user_ns     = self.user_ns,
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 329, in instance
    '%s are being created.' % cls.__name__
MultipleInstanceError: Multiple incompatible subclass instances of InProcessInteractiveShell are being created.

======================================================================
ERROR: Does requesting completion from an in-process kernel work?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/tests/test_kernelmanager.py", line 67, in test_complete
    km.start_kernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/kernelmanager.py", line 369, in start_kernel
    self.kernel = InProcessKernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/ipkernel.py", line 66, in __init__
    super(InProcessKernel, self).__init__(**traits)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/zmq/ipkernel.py", line 147, in __init__
    user_ns     = self.user_ns,
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 329, in instance
    '%s are being created.' % cls.__name__
MultipleInstanceError: Multiple incompatible subclass instances of InProcessInteractiveShell are being created.

======================================================================
ERROR: Does executing code in an in-process kernel work?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/tests/test_kernelmanager.py", line 59, in test_execute
    km.start_kernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/kernelmanager.py", line 369, in start_kernel
    self.kernel = InProcessKernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/ipkernel.py", line 66, in __init__
    super(InProcessKernel, self).__init__(**traits)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/zmq/ipkernel.py", line 147, in __init__
    user_ns     = self.user_ns,
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 329, in instance
    '%s are being created.' % cls.__name__
MultipleInstanceError: Multiple incompatible subclass instances of InProcessInteractiveShell are being created.

======================================================================
ERROR: Does requesting history from an in-process kernel work?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/tests/test_kernelmanager.py", line 91, in test_history
    km.start_kernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/kernelmanager.py", line 369, in start_kernel
    self.kernel = InProcessKernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/ipkernel.py", line 66, in __init__
    super(InProcessKernel, self).__init__(**traits)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/zmq/ipkernel.py", line 147, in __init__
    user_ns     = self.user_ns,
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 329, in instance
    '%s are being created.' % cls.__name__
MultipleInstanceError: Multiple incompatible subclass instances of InProcessInteractiveShell are being created.

======================================================================
ERROR: Does the in-process kernel manager implement the basic KM interface?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/tests/test_kernelmanager.py", line 37, in test_inteface
    km.start_kernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/kernelmanager.py", line 369, in start_kernel
    self.kernel = InProcessKernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/ipkernel.py", line 66, in __init__
    super(InProcessKernel, self).__init__(**traits)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/zmq/ipkernel.py", line 147, in __init__
    user_ns     = self.user_ns,
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 329, in instance
    '%s are being created.' % cls.__name__
MultipleInstanceError: Multiple incompatible subclass instances of InProcessInteractiveShell are being created.

======================================================================
ERROR: Does requesting object information from an in-process kernel work?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/tests/test_kernelmanager.py", line 79, in test_object_info
    km.start_kernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/kernelmanager.py", line 369, in start_kernel
    self.kernel = InProcessKernel()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/inprocess/ipkernel.py", line 66, in __init__
    super(InProcessKernel, self).__init__(**traits)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/zmq/ipkernel.py", line 147, in __init__
    user_ns     = self.user_ns,
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 329, in instance
    '%s are being created.' % cls.__name__
MultipleInstanceError: Multiple incompatible subclass instances of InProcessInteractiveShell are being created.

----------------------------------------------------------------------
Ran 8 tests in 0.019s

FAILED (errors=8)
Evan Patterson
Collaborator

@fperez, I think this is related to something I was worried about, but which I forgot to bring up here, namely that InteractiveShell seems to be a singleton. This assumption doesn't work so well for the single-process model, in which it is perfectly reasonable to have multiple kernels, each with their own interactive shell. At least, that has been my mental model and it's something that people will expect to work.

Here's the code which is surely problematic:

https://github.com/epatters/ipython/blob/d32bdb0b1a8916079540dda4e798272e45c41b4a/IPython/zmq/ipkernel.py#L143

I know very little about the motivation and uses for the singleton interactive shell, so you are probably better placed than I am to decide what to do about this. What do you recommend?

Brian E. Granger
Owner
Evan Patterson
Collaborator

Thanks for the information, Brian.

What do you recommend we do right now? I certainly can't undertake any serious project-wide refactor.

Brian E. Granger
Owner
Fernando Perez
Owner

@epatters, I just made a single-commit PR against your branch that fixes the problem for now. The long-term solution is to cleanly rework our assumptions about where we truly need singletons, but that's indeed far too much of a refactor to undertake in the context of this PR.

Now, because of these issues, you are likely only going to be able to use one kernel at a time per-process, but that's still progress. And with these tests in-place, we can then gradually refactor things out until we confine our singleton assumptions only to where they make sense (at the application level, which is where they match the OS singleton - the process).

Fernando Perez Ensure that in-process test group doesn't create global IPython singl…
…eton.

Since the in-process group makes its own shells, it should avoid
creating the global singleton.
e51dadb
Evan Patterson
Collaborator

Thanks, @fperez. I cherry-picked the commit.

I can confirm that iptest IPython.inprocess works, but I get the singleton errors when I run iptest on the whole project. Is this the behavior you expected?

Fernando Perez
Owner

Mmh, it works for fine for me whether I run it on the whole project or part of it... That's very strange...

Fernando Perez
Owner

BTW, the error reported by the Travis build is b/c your pylab test isn't protected by a decorator to skip it if matplotlib isn't present. See this for an example on how to do it.

Evan Patterson
Collaborator

Thanks. Hopefully the tests start passing on the build bot now.

Fernando Perez
Owner

Did you sort out the problem on your machine? I don't understand what's going on for you...

I added this line:

print(sys.argv[1:]); sys.exit(0)  # dbg

at line 502 of iptest.py, right before the if statement. That showed me that indeed, during the full run, IPython.inprocess is passed to sys.argv and things behave as expected.

Why don't you try debugging it there until we sort it out... What platform are you on?

Evan Patterson
Collaborator

@fperez, I will look at this first thing tomorrow. Unfortunately, I have a Friday deadline for some unrelated work and will be offline for the rest of the day.

Evan Patterson
Collaborator

Oh, and my work machine is OS X.

Fernando Perez
Owner

OK; @minrk, @ellisonbg if you guys could run test_pr on this one from your OSX boxes, it would help us to have an extra data point, as it's passing fine for me on linux.

@epatters, the current failure on Travis looks like a legitimate issue on 3.2 only (the 2.x builds are now passing). Could you look into that? And that log shows one thing we need to fix: the in-process kernel should be initialized with tracebacks in no-color mode so that we can read them without all the ANSI gunk on the screen.

Min RK
Owner

passed on 2.6/7, failed on 3.2.

Fernando Perez
Owner

Thanks @minrk. @epatters that does indeed look like a legitimate 3.2 failure, I see it on linux too; I hadn't tested 3.2 earlier when I reported success.

Bradley M. Froehle
Collaborator

I believe the failure is coming because raw_input was renamed to input in Python 3. So the test needs to either call a compatibility function like IPython.utils.py3compat.input() or you'll need to do something like:

km.shell_channel.execute('try:\n    x = raw_input()\nexcept NameError:\n   x = input()')

or maybe

if py3compat.PY3:
    km.shell_channel.execute('x = input()')
else:
    km.shell_channel.execute('x = raw_input()')

Edit: hmf, maybe not. I should try these suggestions out first. Anyway, I second @fperez's suggestion to turn off the colorization (how?) so the traceback is readable.

hsyed

Hello again, I'm wondering if there is a way of getting the widget to get keyboard focus when I click on it ? I have my ipython widget in a tab layout and when I click on the tab I have to click again into the widget, it would be nice if the currsor would move to its position when the widget is activated.

Also, do you know how I would configure the embedded shell to have --pylab=inline functionality ?

Thomas Kluyver
Collaborator

@epatters I'm just checking the pull request queue. Will you have time to sort out the input/raw_input issue any time soon? Thanks.

tomcraw00

Any update on this? I've been messing around with putting this into a project but I don't want to get left behind by the main trunk of ipython.

Pietro Berkes

Hi, I just wanted to bump this pull request: using an in-process shell for application scripting is a scenario that I see quite frequently and it would be great to have it in IPython.

What is left to be done for the PR to be merged? I may be able to spend some time on it.

Jonathan March
Collaborator

Closing because superseded by PR #2724

Jonathan March jdmarch closed this December 27, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.