Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

In-process kernel support (take 3) #2724

Merged
merged 17 commits into from

12 participants

Pietro Berkes Thomas Kluyver Jonathan March Evan Patterson Min RK Brian E. Granger tomcraw00 Nicholaus Halecky Charl P. Botha Cyrille Rossant Hassan Syed Fernando Perez
Pietro Berkes

This PR continues PR #2397 now that @epatters is offline. The previous PR has been closed and this one will take its place.

I fixed the Python3 issues that got PR #2397 stuck.

epatters and others added some commits
Evan Patterson epatters Refactor kernel managers in preparation for the EmbeddedKernel. e367f3e
Evan Patterson epatters Implement EmbeddedKernel. 8d53a55
Evan Patterson epatters Add abstract base class (ABC) for sockets used in kernel. 40e9f49
Evan Patterson epatters BUG: QtKernelManagerMixin does not define PyQt4-safe signals. 346e7ab
Evan Patterson epatters DOC: Add an example showing off the embedded Qt console. 20b2162
Evan Patterson epatters REFACTOR: Terminology change: 'embedded' -> 'in-process'. bd40426
Evan Patterson epatters DOC: Add module-level docstring to in-process kernel example. 28bcbaa
Evan Patterson epatters DOC: Add information about in-process kernels to What's New. d2ab68e
Evan Patterson epatters BUG: GUI integration broken in the in-process kernel. 6aaeb87
Evan Patterson epatters TST: Add some unit tests for in-process kernel and kernel manager. fc75ceb
Evan Patterson epatters TST: Add more unit tests for in-process kernel manager. 6632e6c
Evan Patterson epatters BUG: raw_input logic incorrect for in-process terminal frontend. d32bdb0
Fernando Perez fperez Ensure that in-process test group doesn't create global IPython singl…
…eton.

Since the in-process group makes its own shells, it should avoid
creating the global singleton.
e51dadb
Evan Patterson epatters TST: Skip pylab test for in-process kernel when matplotlib is missing. 38e8da0
Pietro Berkes pberkes BUG: Logging a rare error condition would have failed. 270e9a5
Pietro Berkes pberkes BUG: Solve 2to3 conversion error.
One of the methods in the inprocess kernel objects used
a local variable called `raw_input` that was mistakenly
converted to `input` during the 2to3 conversion.
36f98b6
Pietro Berkes pberkes BUG: In inprocess tests, use input/raw_input depending on Python vers…
…ion.
775571c
Pietro Berkes

As I mentioned before, this PR covers a frequent use case for me and I think it would be great to have it in IPython. Based on comments on the previous PR by @fperez , it also sounds like it would be useful for IPython in general. I have some time to spend on it at the moment if there is any other issue to be fixed...

BTW, I tried merging the PR to master and all tests stil pass.

Thomas Kluyver
Owner

Thanks for tackling this. It might be a few days before anyone looks at it in detail, but we're definitely interested in supporting this case.

Jonathan March
Collaborator

Who would be plausible reviewers for this PR?

Thomas Kluyver
Owner

Maybe @epatters and @minrk ?

Evan Patterson
Collaborator

@takluyver, I have reviewed the Python 3 compatibility fixes. The fixes look good (thanks, @pberkes!).

At the time of my departure, these were the only outstanding issues of which I was aware. I hope that this PR can now move forward.

Thomas Kluyver
Owner

Thanks @epatters .

@minrk : are you happy for this to be merged?

Jonathan March
Collaborator

Thanks, @epatters.
@ellisonbg, in #2643, also expressed willingness to help review this ticket, if more review is needed.

Min RK
Owner

I think this looks pretty solid now. My only thought is that I would put these new base classes in IPython.kernel instead of IPython.inprocess.

Brian E. Granger ellisonbg commented on the diff
IPython/inprocess/blockingkernelmanager.py
((16 lines not shown))
+
+# Standard library imports.
+import Queue
+from threading import Event
+
+# Local imports.
+from IPython.utils.io import raw_print
+from IPython.utils.traitlets import Type
+from kernelmanager import InProcessKernelManager, ShellInProcessChannel, \
+ SubInProcessChannel, StdInInProcessChannel
+
+#-----------------------------------------------------------------------------
+# Utility classes
+#-----------------------------------------------------------------------------
+
+class BlockingChannelMixin(object):
Brian E. Granger Owner

I am not sure it makes sense for these base classes to in in IPython.inprocess, I view the main kernel code to be more fundamental.

Evan Patterson Collaborator

This seems reasonable and @minrk evidently shares this view. Are BlockingChannelMixin and SocketABC the only classes that need to be moved?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brian E. Granger ellisonbg commented on the diff
IPython/zmq/ipkernel.py
@@ -739,36 +753,6 @@ def _complete(self, msg):
cpos = len(c['line'])
return self.shell.complete(c['text'], c['line'], cpos)
- def _object_info(self, context):
Brian E. Granger Owner

Where did the implementation of _object_info go?

Evan Patterson Collaborator

This was dead code, left over from the very early days of the kernel. I took the liberty of removing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brian E. Granger ellisonbg commented on the diff
IPython/inprocess/kernelmanager.py
((278 lines not shown))
+ self._pause = True
+
+ def unpause(self):
+ """ Unpause the heartbeat. """
+ self._pause = False
+
+ def is_beating(self):
+ """ Is the heartbeat running and responsive (and not paused). """
+ return not self._pause
+
+
+#-----------------------------------------------------------------------------
+# Main kernel manager class
+#-----------------------------------------------------------------------------
+
+class InProcessKernelManager(HasTraits):
Brian E. Granger Owner

After this PR is merged, I am going to do some cleanup work on the zmq.KernelManager. I am going to make it a Configurable. Do you want to use that base class here? Does this class have the same public API as zmq.KernelManager. Would it make sense to share any of the implementation with a common base class? Also, do the channel classes above have the same API?

Evan Patterson Collaborator

I have tried to make the in-process kernel manager API-compatible with the standard kernel manager. Under ideal circumstances, they should be drop-in replacements for each other. Certain methods, like signal_kernel, do not make sense for the in-process kernel manager and therefore raise exceptions; similarly, certain attributes, like the ZMQ context, are not present on the in-process channel objects. Apart from this, the interfaces should be identical, including for the channel classes. Ultimately, I would like to see both kernel managers inherit from some abstract base class to clarify which subset of the interface is guaranteed to be common.

Sharing implementation was not useful for the kernel managers, though I did share implementation as appropriate in the blocking and Qt channel classes.

Min RK Owner
minrk added a note

Ah, I somehow missed this when reading it most recently. I think the shared base class for inprocess and zmq KernelManagers should be a criterion for merge.

Evan Patterson Collaborator

To be clear, the shared base class is useful only for indicating shared interface; there is no implementation to be shared here. (If I were writing ETS code, I would use an Interface for this). Given that, the issue does not seem especially pressing. Besides, @ellisonbg has already indicated his desire to work on this after the merge.

Brian E. Granger Owner
Min RK Owner
minrk added a note
Brian E. Granger Owner

OK let's merge this then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brian E. Granger
Owner

Overall, this look like really clean code. I just left a few inline comments.

Brian E. Granger ellisonbg merged commit 98e3fdd into from
tomcraw00

This is great, but is there any documentation or example code to actually implement this? The old gist that epatters wrote does not work. I am guessing there were some name changes somewhere in here?

Nicholaus Halecky

I am also excited about this feature, but can't seem to find any documentation about how to use it except for one thing! I did see the attached .ipynb in the PR stack, checked it out on nbviewer, but couldn't get it working. I'm obviously missing something. Perhaps a special way to start ipython notebook from command line? Any pointers would be appreciated. TY!

Charl P. Botha

I would like to double-check: This work has not been backported to the 0.13.x branch, and is currently only available on master, right?

Min RK
Owner

Correct

Cyrille Rossant

embedded_qtconsole.py (from here) does not appear to work on master:

ImportError: No module named kernelmanager
Min RK
Owner

That's correct, some things have been renamed. There is an example in IPython now.

Cyrille Rossant

Great, I missed that. Thanks!

Hassan Syed

Could someone please help with my SO question :
http://stackoverflow.com/questions/16737323/embedding-ipython-into-a-pyqt4-app

Hmm, I am wondering if this code even made it into 13.2

Many thanks.

Hassan Syed

Ok, so I modified the code fragment from SO (see above). but I am getting the same error. I am on OS X and using QT (4.8.4) and PyQt4 (4.10.1) The git version of ipython is whatever is current today,

Min RK
Owner

There is an example in the IPython tree. Your error indicates that you have imported PyQt prior to importing any IPython objects, and the IPython qt code requires PyQt API v2, which is not the default for PyQt in Python 2.

Hassan Syed

Many thanks, that solved it. I would suggest that the error message mentions the potential import sequence issue.

Min RK
Owner

that's a good idea, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 10, 2012
  1. Evan Patterson
  2. Evan Patterson

    Implement EmbeddedKernel.

    epatters authored
Commits on Sep 11, 2012
  1. Evan Patterson
Commits on Sep 17, 2012
  1. Evan Patterson
  2. Evan Patterson
Commits on Sep 24, 2012
  1. Evan Patterson
  2. Evan Patterson
  3. Evan Patterson
  4. Evan Patterson
  5. Evan Patterson
Commits on Sep 26, 2012
  1. Evan Patterson
  2. Evan Patterson
Commits on Sep 27, 2012
  1. Fernando Perez Evan Patterson

    Ensure that in-process test group doesn't create global IPython singl…

    fperez authored epatters committed
    …eton.
    
    Since the in-process group makes its own shells, it should avoid
    creating the global singleton.
  2. Evan Patterson
Commits on Dec 27, 2012
  1. Pietro Berkes
  2. Pietro Berkes

    BUG: Solve 2to3 conversion error.

    pberkes authored
    One of the methods in the inprocess kernel objects used
    a local variable called `raw_input` that was mistakenly
    converted to `input` during the 2to3 conversion.
  3. Pietro Berkes
Something went wrong with that request. Please try again.