-
-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow connection to kernels by files #847
Conversation
HMAC signatures are now on by default in all situations, including notebook, kernel, and qtconsole. |
Also add For reference, this function may be useful to folks like @fperez, who like to connect a qtconsole to their current notebook: def connect_qtconsole():
from subprocess import Popen, PIPE
from IPython.zmq.kernelapp import KernelApp
app = KernelApp.instance()
return Popen(['ipython', 'qtconsole', '--existing', app.connection_file]) Which, when called, will launch a qtconsole connected to the kernel that called it. This applies to kernels started with a notebook, other qtconsole, etc. |
@minrk, that's great, and I'd actually vote for wrapping it in a magic also, that we expose interactively for all kernels. Havig the function api is useful for scripted control, but users would simply type |
|
@ivanov can you check if any of the changes here cause problems for you with vim-ipython? That's the only third party client I know of. The change of ip/port pairs to just ports might mean you have to make an adjustment, but in the long run it should be simpler, because you shouldn't have to keep track of ports at all. |
Just to say, |
looking into it now... |
I peeked into vim-ipython a bit, and it shouldn't be a big deal. You just need to specify one argument: if 'connection_file' in BlockingKernelManager.class_trait_names():
# 0.12
km = BlockingKernelManager(connection_file=s)
else:
# parse line as before for 0.11 |
It did require a small change to be convenient, but I think it should be working now: ivanov/vim-ipython#13 |
Just a note: this will certainly conflict with #813 in qtconsoleapp.py, so let's pick merge order carefully, as that one has already been rebased at least once. |
as usual, Min fixed the issues i was going to complain about, even before I had the chance to verbalize the complaint :) My only concern is related to profiles, since, as I understand it - clients aren't apps, but the connection_file get stored in the active profile's security/ folder. I suppose it's not a big issue, since the connection_info magic prints the files contents |
We do aim to please.
Note that you can always specify the full path of the connection file, regardless of profiles, etc. The profile |
yes, true, but you have to work a bit to get that full path, since it isn't what's printed, but that's ok.
right - so i guess that's the trade off - before, i didn't have to know anything about profiles, just hookup to the right ports and go. Now it seems I have to look in the profile directory in order to get to the file (though, i suppose, there's nothing stopping me from using the contents of the file as printed by the connection_info magic in the same manner as i used to in 0.11) |
The full path is only not printed if it's in the default location, so if the output is just
Yes, it's a tradeoff. An alternate approach would be if I made the KernelManager itself profile-aware (it's not currently), which would mean that clients using the KernelManager would generally not need to be. An example of the KernelManager not being profile aware - if you create a KernelManager unconfigured and use it to start a Kernel, the connection file will go in a temp dir: >>> from IPython.zmq.blockingkernelmanager import BlockingKernelManager
>>> km = BlockingKernelManager()
>>> km.start_kernel()
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing /var/folders/zz/v3fz17cd2nz0bqvpd2s7btq00000gn/T/tmpMBJMP6.json
>>> km.start_channels() So it's only non-IPython apps that want to start clients connected to IPython-started kernels under default config that are at any disadvantage. We could also add an API like: get_security_file(fname, profile='default'):
"""returns security file from IPython profile directory"""
return os.path.join(get_ipython_dir(), 'profile_%s' % profile, 'security', fname) So that apps would not need to know about our directory structure, only the more abstract idea of profiles. |
Oh, I didn't realize this - that's actually very good!
I think it's great as is right now - I can't think of a reason why the
This would be great - it was exactly the kind of thing I couldn't |
get_security_file() added to utils.path |
# alias passed with arg via space | ||
swallow_next = True | ||
|
||
def init_connection_file(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A small docstring here summarizing what this puppy does would be good...
I only have some trivial cosmetic comments on most of it (made inline), and one of substance on the magics: let's make both magics be trivial wrappers around regular Python library functions. Testing magics is kind of annoying, and using them from other code also awkward, b/c you have to call them via the magic system. And for example, the Same thing for the |
Other than the above trivial cosmetic fixes and the slightly more substantial (but easy) fixing of the magics, I'm +100 on merging this ASAP. Unless anyone objects, let's merge it once it's fixed up; the improvements it brings are really major. Fantastic work! |
I did the cleanup on qtconsoleapp. The library functions make perfect sense. What would you call them, and where would you put them? |
all foo_address traits became foo_port, adding one 'ip' trait for all This matches the Kernel, which does not allow specifying multiple IPs for each channel.
$> ipqt --shell 12345 would not scrub the port, previously
this addresses security issues (ipython#688) and ease-of-multi-frontend
This means that proper restart is now available, rather than killing and starting a new kernel, breaking connections to secondary frontends.
add session_aliases and session_flags to consolidate these settings use new consolidated flags in parallel apps as well add default_secure() function for use in apps where the default behavior should be to enable HMAC signatures.
This is separate from previous, because it is more likely to be rejected. It requires the the Session objects in Handlers get a reference all the way back up to the IPython App that started the environment.
for loading connection info from its connection_file
ensure str_to_bytes and bytes_to_str are used to transform Session.key to/from JSON
This makes it easier for third party tools to retrieve security files (e.g. json connection files) by name and [optional] profile name alone, without knowledge of IPython's directory structure. For example, whenever ipkernel outputs: ` --existing kernel-12345.json --profile foo` that file can be found with: get_security_file('kernel-12345.json', profile='foo')
not sure if i'm OT, but I'd like to also see init_ssh functionality grafted out of qtconsoleapp and placed wherever other general utilities go |
@ivanov argh! That's a good idea. I will never be done with this. |
@ivanov, tunnel_to_kernel should now be a generic function for tunneling connections to a kernel, given a connection file or its unpacked contents. |
On Wed, Oct 12, 2011 at 9:00 PM, Min RK
|
principally adding big docstring to complicated init_ssh method
get_connection_info and connect_qtconsole are now implemented as library functions in the new module. The zmqshell magics that call them are now simple wrappers.
added to new IPython.lib.kernel with other kernel connection utilities.
I also split the connection file search into a library function, so that will be easier to reuse as well. |
* fileglob-based searching for connection file added as IPython.lib.kernel.find_connection_file() * also use this search in other lib.kernel methods, to make them more useful when run outside an IPython kernel.
@minrk, this is looking pretty solid, and big enough we shouldn't let it linger much further. I can't think of much more to do on it at this point, can you? Otherwise, let's push it through. Awesome job! |
I'm just playing with various use cases to make sure it works, then I'll go ahead and merge. @ivanov, I also updated the vim-ipython PR, to use the new glob-based file search |
* JSON connection files are now used to connect files * HMAC message signing is now on by default in all IPython apps * Adds IPython.lib.kernel, which contains utility functions for connecting clients. These were mostly copied out of qtconsoleapp in order to be more general. * Adds %connection_info and %qtconsole magics to zmqshell closes gh-688 closes gh-806 closes gh-691
merged |
* JSON connection files are now used to connect files * HMAC message signing is now on by default in all IPython apps * Adds IPython.lib.kernel, which contains utility functions for connecting clients. These were mostly copied out of qtconsoleapp in order to be more general. * Adds %connection_info and %qtconsole magics to zmqshell closes ipythongh-688 closes ipythongh-806 closes ipythongh-691
This moves all the connection info for a kernel into JSON files like the parallel code, so there's no need to specify 4 ports to connect to an existing kernel. In fact, if you just specify
ipython qtconsole --existing
, it will try to connect to the most recent kernel, including those started by the notebook.It also cleans up some KernelManager code in the notebook, so that changes to the base KernelManager will be inherited in the notebook. The main immediate benefit being that restarting a notebook kernel really does restart it, so attached qtconsoles will remain attached.
I will also turn HMAC signatures on by default before merging.
This should address #688, #806, and even probably #691.