Skip to content
This repository

add IPython.embed_kernel() #1357

Merged
merged 9 commits into from about 2 years ago

5 participants

Scott Tsai Min RK Fernando Perez Thomas Kluyver Bradley M. Froehle
Scott Tsai

This is the second submission of this patch, revised to address concerns raised in:
#1351 (comment)
1. Allow IPython.zmq import to fail since IPython does not require zmq to exist.
This disables the feature.
2. Pass the namespace of the function calling embed_kernel() down as user_ns

This patch adds IPython.embed_kernel() as a public API.
Embedding an IPython kernel in an application is useful when you want to
use IPython.embed() but don't have a terminal attached on stdin and stdout.

My use case is a modern gdb with Python API support
#!/usr/bin/gdb --python
import IPython; IPython.embed_kernel()
this way I get to use ipython to explore the GDB API without
the readline librarry in gdb and ipython fighting over the terminal settings.

A Google search revealed that other people were interetsted in this use
case as well:
http://mail.scipy.org/pipermail/ipython-dev/2011-July/007928.html

Scott Tsai scottt add IPython.embed_kernel()
This patch adds IPython.embed_kernel() as a public API.
Embedding an IPython kernel in an application is useful when you want to
use IPython.embed() but don't have a terminal attached on stdin and stdout.

My use case is a modern gdb with Python API support
 #!/usr/bin/gdb --python
 import IPython; IPython.embed_kernel()
this way I get to use ipython to explore the GDB API without
the readline librarry in gdb and ipython fighting over the terminal settings.

A Google search revealed that other people were interetsted in this use
case as well:
http://mail.scipy.org/pipermail/ipython-dev/2011-July/007928.html
25e21eb
Min RK
Owner

Now the default behavior is to embed the Kernel into the calling application scope, which is incorrect for the basic IPython kernel. This is why the embed entry point and the regular entry point in terminal IPython are not the same. One is not a subset of the other.

Scott Tsai

I'm reading up on traitlets.

Scott Tsai

@minrk, I think I addressed the concerns raised so far.

Min RK
Owner

nice! Thanks. Pinging @fperez, who was asking about exactly this feature last week.

One thing you might change from a user perspective is slightly different handling of the failed import. Right now, if pyzmq is not installed, trying IPython.embed_kernel() will raise an AttributeError. Perhaps it would be better to fallback on something with a more informative error (ImportError("IPython.zmq requires pyzmq ≥ 2.1.4") being the most logical error to raise).

Fernando Perez
Owner

Awesome, @scottt, many thanks! I have a long trip tomorrow so I'll be offline for a day or two, but if nobody beats me to it I'll definitely play with this and review it before the end of the week.

Scott Tsai

@minrk, I tweaked the error message to be ImportError("IPython.embed_kernel requires pyzmq >= 2.14") as refering to IPython.embed_kernel instead of IPython.zmq seems clearer to me.
Thanks a bunch for the fast responses and pointers.

@fperez, Looking forward for the review :)
My last patch to IPython was 6+ years ago, thanks for all your work on IPython. I love this software!

IPython/__init__.py
@@ -44,6 +44,12 @@
44 44 from .core import release
45 45 from .core.application import Application
46 46 from .frontend.terminal.embed import embed
  47 +try:
  48 + from .zmq.ipkernel import embed_kernel
  49 +except ImportError:
  50 + def embed_kernel(*args, **kwargs):
  51 + raise ImportError("IPython.embed_kernel requires pyzmq >= 2.14")
3
Min RK Owner
minrk added a note

This should be 2.1.4, not 2.14

Thomas Kluyver Collaborator
takluyver added a note

I'm not sure about catching ImportError here. It could mask a real problem with another import this triggers. Also, having this import here will load the ZMQ code even if it's not needed (i.e. in the plain terminal client). I think we can work round both issues by doing something like this:

def embed_kernel(*args, **kwargs):
    # Lazy import
    from .zmq.ipkernel import embed_kernel as real_embed_kernel
    real_embed_kernel(*args, **kwargs)
Scott Tsai
scottt added a note

@takluyver, I'm sympathetic to the two issues you raised:
1. Catching all ImportError's could mask the module really causing it.
2. Always importing ZMQ is unnecessary for normal IPython use.

For functions that examine the local variables of their callers you can't just add another layer that wraps them though. i.e.real_embed_kernel() would end up inspecting the locals and module globals of embed_kernel() instead of the caller of embed_kernel() in your snippet. I ended up changing the code like: scottt@087e9c3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
IPython/zmq/ipkernel.py
((4 lines not shown))
  648 +def caller_module_and_locals():
  649 + """Returns (module, locals) of the caller"""
  650 + caller = sys._getframe(1).f_back
  651 + global_ns = caller.f_globals
  652 + module = sys.modules[global_ns['__name__']]
  653 + return (module, caller.f_locals)
  654 +
  655 +def embed_kernel(module=None, local_ns=None):
  656 + """Call this to embed an IPython kernel at the current point in your program. """
  657 + (caller_module, caller_locals) = caller_module_and_locals()
  658 + if module is None:
  659 + module = caller_module
  660 + if local_ns is None:
  661 + local_ns = caller_locals
  662 + app = IPKernelApp.instance(user_module=module, user_ns=local_ns)
  663 + app.initialize()
1
Min RK Owner
minrk added a note

You should pass initialize an empty list here (app.initialize([])), so that it doesn't try to parse sys.argv, which could be problematic if called from an environment that used its own command-line args.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
scottt added some commits
Scott Tsai scottt embed_kernel: pass [] to IPKernelApp.initialize to prevent argv parsing
https://github.com/ipython/ipython/pull/1357/files#r400153
Stop embed_kernel() from parsing sys.argv since to support being called from
a environments with their own command line args.
75b852d
Scott Tsai scottt embed_kernel: only import zmq when needed 087e9c3
Scott Tsai

After reading #1357 (comment), I changed the code slightly in scottt@087e9c3

embed_kernel() now only imports IPython.zmq.ipkernel when called. So as not to cause unnecessary I/O and slow down IPython startup for an infrequently used feature.

I also stopped catching ImportError for IPython.zmq.ipkernel. I found that IPython/zmq/init.py would catch and replace the ImportError message with "IPython.zmq requires pyzmq >= 2.1.4" which seems clear enough. Not doing the import check twice also means the minimum ZMQ version required can stay written only in one place.

Thomas Kluyver

I think this function should be in IPython.utils.frame - this top level file should have as little code in as possible. The embed_kernel wrapper can then import it.

IPython.utils.frame seems like a good place indeed. I did the move in 030fc46

IPython/utils/frame.py
@@ -85,3 +85,10 @@ def debugx(expr,pre_msg=''):
85 85 # deactivate it by uncommenting the following line, which makes it a no-op
86 86 #def debugx(expr,pre_msg=''): pass
87 87
  88 +def caller_module_and_locals():
  89 + """Returns (module, locals) of the caller"""
3
Thomas Kluyver Collaborator
takluyver added a note

We should clarify this a bit - 'the caller' would normally mean the frame in which this is called, whereas this actually looks at the frame above that.

Possibly the function should take an optional depth argument similar to extract_vars.

Scott Tsai
scottt added a note

@takluyver,I didn't want to add a depth argument to `caller_module_and_locals() without renaming the function. See if you like scottt@a61f445

Thomas Kluyver Collaborator
takluyver added a note

Thanks, that looks good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Thomas Kluyver takluyver commented on the diff
IPython/__init__.py
@@ -55,3 +57,14 @@
55 57 __author__ += author + ' <' + email + '>\n'
56 58 __license__ = release.license
57 59 __version__ = release.version
  60 +
  61 +def embed_kernel(module=None, local_ns=None):
  62 + """Call this to embed an IPython kernel at the current point in your program. """
  63 + (caller_module, caller_locals) = extract_module_locals_above()
  64 + if module is None:
2
Thomas Kluyver Collaborator
takluyver added a note

I think this logic to get the calling scope should be inside the embed_kernel function in ipkernel, so that it has exactly the same API if it's imported from there.

Fernando Perez Owner
fperez added a note

I agree with @takluyver here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Fernando Perez fperez commented on the diff
IPython/zmq/ipkernel.py
@@ -639,6 +645,10 @@ def launch_kernel(*args, **kwargs):
639 645 return base_launch_kernel('from IPython.zmq.ipkernel import main; main()',
640 646 *args, **kwargs)
641 647
  648 +def embed_kernel(module, local_ns):
1
Fernando Perez Owner
fperez added a note

This function needs a docstring. @scottt: our dev guide describes our doc guidelines (which basically amount to pep8 and using the numpy docstring standard).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Fernando Perez
Owner

@scottt, this is great! One thing I'd like to have before we merge it, is at least one or two tests. Because of the kernel embedding, they might need to go into a new test group, so let us know if you need a hand. But I don't want to build too much more technical debt without new tests.

In this case the test should be pretty simple to write: just start a new process with an embedded kernel that has a namespace configured with one or two variables, check those values from the test, run a piece of code in there, check the result, done.

Let us know if you need a hand with the test, shouldn't be hard.

Fernando Perez
Owner

Finally, this new capability should also be mentioned in the manual, in the embedding section.

Bradley M. Froehle bfroehle commented on the diff
IPython/utils/frame.py
@@ -85,3 +85,16 @@ def debugx(expr,pre_msg=''):
85 85 # deactivate it by uncommenting the following line, which makes it a no-op
86 86 #def debugx(expr,pre_msg=''): pass
87 87
  88 +def extract_module_locals(depth=0):
  89 + """Returns (module, locals) of the funciton `depth` frames away from the caller"""
  90 + f = sys._getframe(depth + 1)
  91 + global_ns = f.f_globals
  92 + module = sys.modules[global_ns['__name__']]
  93 + return (module, f.f_locals)
  94 +
  95 +def extract_module_locals_above():
2
Bradley M. Froehle Collaborator
bfroehle added a note

This function seems overly specific. Why not just replace its one use with extract_module_locals(1)? I see that there is an extract_vars_above function, but I don't think it's ever actually used.

Maybe I just bristle at the word above 'above'. extract_caller_module_locals, similar to a previous patch of yours, makes more sense to me.

Thomas Kluyver Collaborator
takluyver added a note

I don't think caller properly conveys what this is doing: that would imply it looks at the frame which calls this function, whereas its looking at the frame above that. above seems less ambiguous.

I agree that the more general function is probably sufficient for the limited use of this, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Min RK
Owner
minrk commented

This currently breaks all regular usage of the KernelApp, with:

AttributeError: 'IPKernelApp' object has no attribute 'user_module'

Which should be fixed by adding the user_module/user_ns to the IPKernelApp as traitlets.

Fernando Perez
Owner

@scottt, what's your take on this one? This is kind of a high-priority/value PR for us, so if you're swamped and don't think you'll be able to work on it, let us know and we'll do our best to pitch in and give you a hand. It seems there's a bit of fixing still needed before we can merge it...

Min RK
Owner
minrk commented

This is super close, so I'm happy to take it over the line, if you want.

The only changes requested by review seem to be:

  • add user_module/ns to IPKernelApp
  • remove unnecessarily specific extract_module_locals_above()
  • add a docstring and basic test and docs

Those seem quite straightforward, and I'm happy to do them.

One thing we might want to add is passing config to the embedded Kernel, changing the sig to:

def embed_kernel(module=None, local_ns=None, **kwargs):
    KernelApp = IPKernelApp.instance(**kwargs) 
    ...

This better matches the existing IPython.embed()

Min RK
Owner
minrk commented

Actually, come to think of it, user_module/_ns should not be part of IPKernelApp. The embedding should just set them directly on the kernel.

Fernando Perez
Owner

@minrk, are you sure? I seem to recall that we need to 'prepare' a namespace that will be used by the kernel by adding a few names to it (like __file__, get_ipython, etc). In that scenario, it's better to get the dict the user wants in the constructor, so we can manipulate it before the interactive loop starts.

Min RK
Owner
minrk commented
Min RK
Owner
minrk commented

@fperez I've got a working version here, where the user_module/ns are attached only to the Kernel, and behave essentially the same as EmbeddedShell (changing user_ns triggers init_user_ns() which does the setup).

Fernando Perez
Owner

I certainly agree that the two should match, I just wasn't sure the embedded terminal was fully correct :) Is the init_user_ns() step triggered by traitlets event handling?

Min RK
Owner
minrk commented

it is not in EmbeddedShell, it is in the Kernel

Min RK minrk referenced this pull request
Merged

Finish up embed_kernel #1640

Fernando Perez fperez merged commit a61f445 into from
Fernando Perez fperez closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 9 unique commits by 1 author.

Jan 31, 2012
Scott Tsai scottt add IPython.embed_kernel()
This patch adds IPython.embed_kernel() as a public API.
Embedding an IPython kernel in an application is useful when you want to
use IPython.embed() but don't have a terminal attached on stdin and stdout.

My use case is a modern gdb with Python API support
 #!/usr/bin/gdb --python
 import IPython; IPython.embed_kernel()
this way I get to use ipython to explore the GDB API without
the readline librarry in gdb and ipython fighting over the terminal settings.

A Google search revealed that other people were interetsted in this use
case as well:
http://mail.scipy.org/pipermail/ipython-dev/2011-July/007928.html
25e21eb
Scott Tsai scottt zmq.ipkernel: don't call embed_kernel() in main() 8f53ec7
Scott Tsai scottt zmq.Kernel: turn user_{module,ns} into traitlets a92b812
Scott Tsai scottt embed_kernel: give a clear error message on pyzmq ImportError a96b676
Scott Tsai scottt embed_kernel: fix pyzmq version requirement error message a2ec8bd
Scott Tsai scottt embed_kernel: pass [] to IPKernelApp.initialize to prevent argv parsing
https://github.com/ipython/ipython/pull/1357/files#r400153
Stop embed_kernel() from parsing sys.argv since to support being called from
a environments with their own command line args.
75b852d
Mar 11, 2012
Scott Tsai scottt embed_kernel: only import zmq when needed 087e9c3
Mar 12, 2012
Scott Tsai scottt Move caller_module_and_locals() to IPython.util.frame 030fc46
Scott Tsai scottt Refactor caller_module_locals() into extract_module_locals() a61f445
This page is out of date. Refresh to see the latest.
13 IPython/__init__.py
@@ -44,10 +44,12 @@
44 44 from .core import release
45 45 from .core.application import Application
46 46 from .frontend.terminal.embed import embed
  47 +
47 48 from .core.error import TryNext
48 49 from .core.interactiveshell import InteractiveShell
49 50 from .testing import test
50 51 from .utils.sysinfo import sys_info
  52 +from .utils.frame import extract_module_locals_above
51 53
52 54 # Release data
53 55 __author__ = ''
@@ -55,3 +57,14 @@
55 57 __author__ += author + ' <' + email + '>\n'
56 58 __license__ = release.license
57 59 __version__ = release.version
  60 +
  61 +def embed_kernel(module=None, local_ns=None):
  62 + """Call this to embed an IPython kernel at the current point in your program. """
  63 + (caller_module, caller_locals) = extract_module_locals_above()
  64 + if module is None:
  65 + module = caller_module
  66 + if local_ns is None:
  67 + local_ns = caller_locals
  68 + # Only import .zmq when we really need it
  69 + from .zmq.ipkernel import embed_kernel as real_embed_kernel
  70 + real_embed_kernel(module, local_ns)
13 IPython/utils/frame.py
@@ -85,3 +85,16 @@ def debugx(expr,pre_msg=''):
85 85 # deactivate it by uncommenting the following line, which makes it a no-op
86 86 #def debugx(expr,pre_msg=''): pass
87 87
  88 +def extract_module_locals(depth=0):
  89 + """Returns (module, locals) of the funciton `depth` frames away from the caller"""
  90 + f = sys._getframe(depth + 1)
  91 + global_ns = f.f_globals
  92 + module = sys.modules[global_ns['__name__']]
  93 + return (module, f.f_locals)
  94 +
  95 +def extract_module_locals_above():
  96 + """Returns (module, locals) of the funciton calling the caller.
  97 + Like extract_module_locals() with a specified depth of 1."""
  98 + # we're one frame away from the target, the function we call would be two frames away
  99 + return extract_module_locals(1 + 1)
  100 +
10 IPython/zmq/ipkernel.py
@@ -70,6 +70,8 @@ class Kernel(Configurable):
70 70 iopub_socket = Instance('zmq.Socket')
71 71 stdin_socket = Instance('zmq.Socket')
72 72 log = Instance(logging.Logger)
  73 + user_module = Instance('types.ModuleType')
  74 + user_ns = Dict(default_value=None)
73 75
74 76 # Private interface
75 77
@@ -110,6 +112,8 @@ def __init__(self, **kwargs):
110 112 # Initialize the InteractiveShell subclass
111 113 self.shell = ZMQInteractiveShell.instance(config=self.config,
112 114 profile_dir = self.profile_dir,
  115 + user_module = self.user_module,
  116 + user_ns = self.user_ns,
113 117 )
114 118 self.shell.displayhook.session = self.session
115 119 self.shell.displayhook.pub_socket = self.iopub_socket
@@ -585,6 +589,8 @@ def init_kernel(self):
585 589 stdin_socket=self.stdin_socket,
586 590 log=self.log,
587 591 profile_dir=self.profile_dir,
  592 + user_module=self.user_module,
  593 + user_ns=self.user_ns,
588 594 )
589 595 self.kernel = kernel
590 596 kernel.record_ports(self.ports)
@@ -639,6 +645,10 @@ def launch_kernel(*args, **kwargs):
639 645 return base_launch_kernel('from IPython.zmq.ipkernel import main; main()',
640 646 *args, **kwargs)
641 647
  648 +def embed_kernel(module, local_ns):
  649 + app = IPKernelApp.instance(user_module=module, user_ns=local_ns)
  650 + app.initialize([])
  651 + app.start()
642 652
643 653 def main():
644 654 """Run an IPKernel as an application"""

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.