IPython notebook can't connect to existing kernel

[bbirand]

Isn't there a way to have the IPython notebook to connect to an already-running kernel? It seems like the --existing flag is not supported for the notebook subcommand.

I usually start a single kernel on the console when I begin my development, but may want to switch to the notebook once I have written and tested some functions. It would be useful to boot up the notebook from the same kernel, so that I don't have to duplicate everything. Is there a reason why this is not supported?

[ivanov]

this is correct and is a known issue: a notebook MUST start its own kernel at the moment, though other clients can connect to that kernel. It's a limitation of the way we communicate with the webserver (via websockets), and some assumptions that the javasciript side, but this will be refactored and will probably be possible to connect a notebook to another kernel in the next version of IPython (2.0 around Decemeber of this year, and note the capitalization 😉 )

[bbirand]

Great, looking forward to IPython 2.0!

[bbirand]

Another very useful addition to spawning a notebook that connects to a running console would be to let the notebook command paste the content of the current history directly in the notebook. From the usability perspective, this will mesh well with the current role of the command. With perhaps a switch to paste it all in a single cell instead of breaking it down to many..

[minrk]

I don't think we have any plans to support --existing in the standard notebook webapp. That's not to say that you (or we) can't create another webapp reusing part of the notebook machinery (and all of the javascript) that only connects to running kernels. This would be relatively simple. But it becomes extremely complicated when you include kernels that the notebook server does not own in the existing framework (kernels cannot be interrupted or restarted, etc.), so it will likely not happen in IPython itself for quite some time.

[toobaz]

Still, it would be great to have, in the "Kernel" menu, an option "Attach to current", which would allow the user to pick any of the currently unused notebooks, or a new one, and then open it in a new page, connected to the current kernel.

The result would be analogous to what one already gets by clicking twice on the same notebook name in the Dashboard (and which I find great when I want to have two tabs side by side, one for playing with the variables, the other for the final code - everything apparently works flawlessly, including kernel restarts/interrupts/deaths), but without the mess of the two pages saving to the same file.

As long as the Dashboard does not assume/enforce somewhere a one-to-one mapping between notebooks and kernels, this should not be so complicated to implement, right?

[ellisonbg]

I remain -1 on this idea.

The difficulty is that the abstraction of a one-to-one mapping between
kernels and notebook is hardwired into the entire frontend/backend
architecture. Implementing this would be very non-trivial.

Also, separating these two things from each other introduces a new
abstraction for users that has to be carefully managed. I think that most
of our users don't have any idea of what a kernel is - or even that they
exist. All they know is that notebooks exist and you can type code into
them. I know this from working extensively with students. These users don't
even have a clear sense of the process abstraction that would allow them to
cleanly separate the different processes (browser, server, kernel) in their
mind.

On Sat, Jan 11, 2014 at 1:11 PM, Pietro Battiston
notifications@github.comwrote:

Still, it would be great to have, in the "Kernel" menu, an option "Attach
to current", which would allow the user to pick any of the currently unused
notebooks, or a new one, and then open it in a new page, connected to the
current kernel.

The result would be analogous to what one already gets by clicking twice
on the same notebook name in the Dashboard (and which I find great when I
want to have two tabs side by side, one for playing with the variables, the
other for the final code - everything apparently works flawlessly,
including kernel restarts/interrupts/deaths), but without the mess of the
two pages saving to the same file.

As long as the Dashboard does not assume/enforce somewhere a one-to-one
mapping between notebooks and kernels, this should not be so complicated to
implement, right?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4066#issuecomment-32107204
.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

[toobaz]

Sure, I do understand it would probably not be a mainstream future, and that's why I would hide it in the "Kernel" menu, rather than expose it in the Dashboard.

But that said, I just had no idea about the implementation difficulties, so thanks for the clarification.

[minrk]

Just to clarify that it not a limitation of any individual component of the notebook, it is an assumption made in the current design of the notebook application as a whole that this 1:1 relationship exists. You can easily make a different web application that changes the relationship of notebooks and kernels (e.g. all notebooks share one kernel, or every visit to a notebook page gets a new kernel, etc.). What is difficult, and probably out of scope at this point, is supporting different kinds of relationships in the existing application.

[jasongrout]

Just FYI, the sage cell design (which uses the IPython kernel javascript and architecture) does instantiate a new kernel for every cell on a page. It is definitely possible.

[minrk]

and the singlecell demo is an example of an application where everything shares a single kernel.

[erxnmedia]

It's no problem, here's a fix: http://stackoverflow.com/questions/20091154/sharing-a-namespace-across-multiple-ipython-notebooks

[orientalperil]

This is highly confusing because of all the talk in the documentation about the difference between kernels and frontends. Essentially the point is that the notebook frontend is NOT decoupled from kernels. This is not written in the documentation. At the very least it should be noted somewhere that the notebook must run its own kernel. The documentation explicitly says that the QT frontend can connect to existing kernels. Why doesn't it say that the notebook frontend does not support this?

[milindsmart]

The documentation explicitly says that the QT frontend can connect to existing kernels. Why doesn't it say that the notebook frontend does not support this?

Agreed! For all that talk of decoupling....

[ihnorton]

Bump? I've seen probably a dozen or so versions of this how-to/feature request in various github repos and stackoverflow posts along the way to this issue... including:

https://groups.google.com/forum/#!topic/jupyter/lVhTvgUbkXU (exactly my use-case)
https://groups.google.com/d/msg/jupyter/qamkem52Xn0/YqrK7pgrAgAJ
jupyter/help#298
jupyter/jupyter#106
jupyter/notebook#727
jupyterlab/jupyterlab#2044
https://stackoverflow.com/questions/40371187/execute-jupyter-notebook-and-keep-attached-to-existing-kernel/40611297
https://stackoverflow.com/questions/31382405/ipython-notebook-how-to-connect-to-existing-kernel?rq=1

[suewonjp]

I think it's a reasonable decision not to disrupt your existing design by not adding the feature like this.
However, this feature would definitely help some use cases. (including mine)
In that situation, you can provide some example code that users can extend or customize for their own use cases.

Is it possible you provide an example bridge kernel? (just like echo_kernel example, but a little more sophisticated.)
What I mean by a "bridge kernel" is that it can do the following:

• At startup, the kernel tries to find an existing kernel. ( e.g. A kernel created by IPython.embed_kernel )
  • In that case, it just delegates communications to that existing kernel.
  • You may use the approach suggested here
    • ( I don't know it's technically a best approach. I'm just learning how jupyter_client works, but only knows basics of basics for now. )
• Otherwise, the kernel behaves like a regular ipykernel.

After writing that bridge kernel as an example, you simply announce you won't do technical support for that example. (except bug fix)
In that way, you can be free from any future burdens, and users who really need the feature can write their kernels based on that example.