remote kernel support

[jzthree]

This was in the original to-do list ("support remote kernels"). I think this feature would be extremely helpful for people who need to run the code on remote servers. Is this feature still part of the roadmap for the near future :)?

[TKCen]

Maybe remote kernel can be used/integrated to support this feature?

[willwhitney]

I would still like to support this.

@rgbkrk, I think you had some concerns about zmq on random ports making it through people's networks?

@TKCen it looks like rk is helping write kernelspecs for remote systems. If they work with Jupyter, I would expect them to work with Hydrogen. The only caveat is that if you have a local kernel for a particular language, Hydrogen may use that instead (it uses the first matching kernel it finds).

Would someone be willing to give rk that a try and let us know if it works?

[willwhitney]

Just opened #39 to address switching between multiple kernels for a language.

[rgbkrk]

rk appears to be over SSH which works for a lot of folks, myself included. The "template" kernel would probably work just fine as is. Might as well try it out. I wouldn't want to manage dependencies this way, but it's at least simple for most users. It doesn't seem like it would require any special requirements on Hydrogen at all.

As for my concerns, these are my observations from teaching classes, running a few tmpnb deployments, JupyterHub deployments, and my own security perspectives. Feel free to take it all with a grain of salt.

Remotes over ZeroMQ

It's all about the kernels communication layer. Not every ZeroMQ installation includes encryption by default (libsodium is an optional dependency for ØMQ). ØMQ4 does have curve, which is wire-compatible with ØMQ3, but won't necessarily have libsodium. One can build it that way for themselves, certainly. This tends to mean either using websockets (through a notebook server) or tunneling through SSH.

Remotes over websockets

The most reliable way I've seen for getting websockets through is making sure they're served over HTTPS on the standard port (443). That boils down to corporate (MITM) proxies totally mucking up the sticky connection with websockets, interfering with the flow, and generally being nasty. As for port, that's about stringent controls (no high ports getting through, etc.) Use HTTPS and you're fine!

Upstream!

That being said, I think the remote kernel issue should be addressed with upstream Jupyter projects to work for the notebook server, etc. As long as we're adhering to the same protocol, it should help multiple projects at once.

I see a near future where folks can consume kernels remotely from wherever they have them deployed, whether that means O'Reilly Media providing them for static content (e.g. https://beta.oreilly.com/learning/an-illustrated-introduction-to-the-t-sne-algorithm, through the use of tmpnb + thebe), a JupyterHub deployment (complete with auth), or some other spawning mechanism. There's not a current authentication construct/config around providing support for remote kernels (from a project level) other than IPython parallel and full notebooks with the JupyterHub spawners, but I'd like to see it soon.

[willwhitney]

Interesting — if something like rk works, it's probably an OK stopgap measure. We can revisit this when Jupyter adds more features for remotes that will let this be a good experience (and not leave the kernel server sitting wide open).

[JavierMares]

I tried rk, Jupyter connects to the kernel momentarily, then loops between "Dead Kernel" and "Connected" every twenty seconds or so.

[dchapsky]

Forgive my ignorance, is the security issue with the remote kernel only relevant in open networks? Lots of use cases involve only talking to closed networks through a vpn or some such.

[rgbkrk]

Forgive my ignorance, is the security issue with the remote kernel only relevant in open networks? Lots of use cases involve only talking to closed networks through a vpn or some such.

For someone advanced enough to do that, they can already use remote kernels just fine. It's the launcher that has to perform it. The kernel.json spec file could specify whatever IP you bind to:

{
  "stdin_port": 57985,
  "ip": "127.0.0.1",
  "control_port": 61088,
  "hb_port": 55956,
  "signature_scheme": "hmac-sha256",
  "key": "1fab9810-6e36-40df-a400-ceb196fbf9a7",
  "shell_port": 54211,
  "transport": "tcp",
  "iopub_port": 56874
}

The default in the notebook is to do localhost so that we're not exposing typical users to remote execution of their laptop while, e.g. at a cafe.

[razcore-rad]

Can someone please give us some example on how to set this up? I am a total noob when it comes to ipython kernnel-ing, and a bit noob with ssh as well (I am connecting remotely through VPN). I'm trying to figure out where this kernelspec should go... I see a folder Hydrogen/kernel-configs but something tells me that those are created each time a kernel is run through atom. How do you do this thing more concretely? A page in the wiki would be nice! Thanks

edit:
OK, so I successfully followed the recipe here to run a kernel on a remote server and connect to it using ipython console through ssh, but I have no idea how to instruct Hydrogen to use the server... if someone could clarify this would be great!

[willwhitney]

There's no documentation because this is definitely not supported behavior right now. Since Hydrogen is designed to start the kernel itself rather than connecting to an existing kernel, it's going to be weird if it works at all. Hydrogen currently connects via local ZeroMQ ports. Furthermore, Hydrogen does not use any security at all, since it's designed for local access, meaning that the server and probably also your local machine would have a huge security hole.

Honestly, I think you're going to be better off using ipython itself for now. I definitely want to support this usage at some point, but there are a lot of changes that need to be made between now and then (both to Hydrogen and to Jupyter). Building remote systems is hard!

I'm sorry I don't have a better answer for you right now, but keep an eye out for changes to come.

[razcore-rad]

I will post shortly... I have just made this to work (without changing anything in hydrogen). So thank you guys! now I have interactivity with my remote kernel which is awesome!

[willwhitney]

Right on!

On Wed, Jul 1, 2015 at 12:30 PM Răzvan Rădulescu notifications@github.com
wrote:

I will post shortly... I have just made this work (without changing
anything in hydrogen)

—
Reply to this email directly or view it on GitHub
#38 (comment)
.

[rgbkrk]

My major issue with not using message signing, even though local, is that anything else that can reach those ZMQ sockets can get remote execution. They have to know the port, but it's not like they can't port scan.

If someone got it to work through the flash binding (zmqsocket.js) you would have a XSS attack with full remote execution from random sites on the internet. That's not good. There's a reason the key and signature scheme is provided.

[razcore-rad]

But it sounds to me that there's a super easy solution to this... You just build the ability to add remote machines in hydrogen, where you specify the remote path to the kernel-1234.json file that the ipython creates on the remote kernel. Read that file and include the key information in the configuration, then do the HMAC serialization... Obviously, this would mean that you need to start the ipython kernel yourself on the remote machine, but I don't see this as a problem.

[willwhitney]

@rgbkrk Ugh, yeah. Let's get signing working.

@razvanc87 Yep — signing and remote kernels are two different steps.

I agree that getting it just to the point of running in Hydrogen isn't a ton of work. The harder steps about making that use case work as smoothly as possible for people.

[willwhitney]

If you're interested in working on this, pull requests are very much appreciated :)

[razcore-rad]

Yep, I definitely am, but I have a lot of ground to cover... I have no experience with this coffeescript stuff, and even less with zmq!... but anyway, until then, I have put this gist together explaining how to set the remote communication up... maybe you guys can take a look and let me know if it makes sense, if I need to change anything or complete/modify. Thank you!

oh right... I forgot to explain what the bash script does... oh well :))... details

[willwhitney]

Awesome work on this — I'll direct people this way in the future!

[oxinabox]

Any news on this?

[n-riesco]

@oxinabox I haven't used them myself, but I'm aware of, at least, three projects to provide remote kernels. See:

• https://bitbucket.org/tdaff/remote_ikernel
• https://github.com/korniichuk/rk
• https://github.com/danielballan/remotekernel

@rgbkrk @lgeiger Currently, the only way to start up Hydrogen is to make a run request. Functionality like switching kernels is not available before that. I think we could provide two more commands:

• Start kernel (this would make the current switch command obsolete)
• Connect to running kernel (this would prompt for a connection file)

[rgbkrk]

We've got a couple remote kernel setups working as part of enchannel, though they're not ready for utility consumption.

[plieningerweb]

old link form @razvanc-r was not working anymore, link to his gist is: https://gist.github.com/razvanc-r/a365b921ba4bf575b6a5

[n-riesco]

Closed via #295