Persistent Jupyter Kernels - Restore/Re-connect to an existing local/remote kernel (do not shutdown kernel upon closing/reloading vscode/vscode.dev)

[DonJayamanne]

Problem

Local

• User starts a notebook, the kernel is now running on the local machine
• Assume the computer goes to sleep,
• After a while if we go back into the notebook, the Notebook is unable to re-connect to the same kernel (kernel state is lost)
• Similarly if user re-loads VS Code, the notebook is unable to re-connect to the same kernel (kernel state is lost)
• Similarly if a user is using Remote SSH, and connection is reset and user re-connects and opens the same notebook, then the user is unable to re-connect to the same kernel (kernel state is lost)

Remote

• User opens a notebook and runs a cell against a remote kernel
• Similarly if user re-loads VS Code (or vscode.dev), the notebook is unable to re-connect to the same kernel (kernel state is lost)

Investigation Running Server & JupyterLab API for extensibility
Goals:

• Long running kernels
  • User can open a notebook with a cell thats still running & see the output being generated
  • Same as User can open notebooks related to kernels that are still running.
• Extensibility for extension authors
  This is a by-product of the long running kernels (i.e. you get this for free - almost)

Planned (related) Prototypes

• Long running kernels
  Solve problems related to kernel/session being lost due to :
  • VS Code Shutdown
  • VS Code Restart
  • Computer sleeping
  • SSH connection issues
  • AML Compute will benefit
• By product of extensiblity
  • Julia Widgets (i might end up doing this first, might be easier)
  • IPyWidget outside notebooks
  • Variable viewer using the new api
  • Data Frame viewer using the new api

Technical details

• Server
  Background process
  Manages kernels & sessions
  Expose kernel socket connection over this connection (we already have the code/technology for this) - proxy socket (dummy kernel in UI layer, by creating a dummy socket connection)
  Security - how do we secure this web server (will need to be addressed, but i'm leaving that for later)
• Expose Jupyter extension extensibility over Jupyter Lab API
  I wont be exposing a connection, instead will just expose the SessionManager, KernelManager & other class instances from extension API

Also related #300

[matifali]

Is there a milestone issue to see the progress of the update?

[DonJayamanne]

Unfortunately this issue has not yet been prioritized at our end, please do vote on this issue though

[matifali]

What do you suggest as a workaround if one wants to run long 10+ hours sessions using Jupyter notebooks in vscode when connected to a remote kernel over SSH (using vscode remote extension)?
after some hours the connection gets disconnected and there is no way to see the progress or output of running cells.

[DonJayamanne]

@matifali unfortunately at this stage we have no work around for this, let me see if i can get an udpate within a week.

[DonJayamanne]

@matifali
I'm trying to understand your expectations, hence the following questions

• Assume you have 1 cell
  *Code in this cell prints numbers from 1 to 100, printing a number every hour
• Assume you run this cell and saw the number 1 printed out.
• Now you run this cell, and close vscode and come back tomorrow and open vscode and open this asme notebook
• Would you expect to see the numbers 1, 2, 3, 4 and then slowly the number going up to 100 while vscode is open (as the execution is still in progress)
• Or would you expect to see 1, 80, 81, 82 and then the number will keep going up while vscode is open (as the execution is still in progress)
• Assume you have opened vscode after a few hours and you know all 100 would have been printed out and vscode was closed. Would you expect to see all 1, 2, ... 100 in the output or just expect to be able to connect to the kernel and see the fact that execution has completed

I ask this because the easiest thing to get working is:

• if the cell is still running then we display 1, 80, 81, 82 (where 1 was from the first instance of vscode and 80, 81 and so on after vscode is opened again. I.e. all of the output generated while vscode was closed will not becaptured and not stored in teh notebook)
• I.e. we will only allow connecting to a kernel and you can see whether exeuction has comlpeted {or not, and if it is still going on then the data will be appened to what was stored previously

Thanks

[matifali]

• Would you expect to see the numbers 1, 2, 3, 4 and then slowly the number going up to 100 while vscode is open (as the execution is still in progress)

I would prefer this output as my use case is to train deep learning models and its better if we can see the full history.

Assume you have opened vscode after a few hours and you know all 100 would have been printed out and vscode was closed. Would you expect to see all 1, 2, ... 100 in the output or just expect to be able to connect to the kernel and see the fact that execution has completed

This is preferred,

Or would you expect to see 1, 80, 81, 82 and then the number will keep going up while vscode is open (as the execution is still in progress)

This is also OK but the problem is vscode is unable to connect to a running remote kernel and show any outputs. Yes, the process is running but we do not see anything printed. There is no indication if the losses are actually decreasing.

[metya]

It feature would be very useful for many of users. Because it is just simply common sense, that If the ssh connection is closed for some reason, we want to able to after reconnect have the same state of the kernel and cells, cause even after reloading VS Code or just reconnecting ssh I can just lose all of my work and code that I made in the cells, because the kernel went down and I forgot to do ctrl+s every 5 minutes.

I think it is not so difficult - just create some kernel in a remote fashion that is not relying on the current ssh connection, and after reloading ssh or entire vs code just propose to choose existing running kernels.

[DonJayamanne]

Another requirement for this
#14446 (comment)

[AnakinShieh]

I have to say it should be a crucial feature for visual studio code now. Currently losing connection to the remote tunnels means losing all of your work/progress makes it hard to do almost all important work.

[rsargent]

I'd love this. This is my biggest pain point with vscode.

[mkarikom]

One more thing to note:

In practice, many of us are running/testing/benchmarking research code, whose various levels of maintenance (I pulled a python 2 repo the other day) mean that project-specific dev containers are pretty common.

The upshot is that the remote kernel for any given notebook is running inside the dev container for that project so that it can make use of the relevant environment.

This results in the following workflow:

1. set up a project in a dev container on some workstation or possibly hpc allocation
2. open up laptop and remote-ssh to the workstation
3. open project folder in container
4. open notebook.ipynb in project
5. start kernel in that project environment
6. be able to connect and reconnect to that kernel as above

I don't know if that makes implementing this insanely important feature more or less complicated....

Last of all thanks @DonJayamanne (and everyone else) for your awesome work making vscode better every day for python!

[TTTPOB]

Here I am having some similar work scenario like @mkarikom. I have to deal with some nasty python environments whose setup might only be possible via container (which is quite common in academia), which results that I can only use remote kernels. But for now the pylance support for remote kernel is broken so the dev experience is not optimal.

I used to mount the container image and point the python extension interpreter path setting to the interpreter inside the container mount. but now this is impossible as python interpreter path setting can influnce the behaviour of jupyter extension is considered a bug and has been fixed.

[ando600]

I'd like to bump this issue. For me this is a breaking feature, and I use jupyterlab over vscode for this reason, despite vscode having a better linter, copilot, and better vim keybindings; I suspect many people who have any kind of remote data science/machine learning workflow feel similarly. I have had this issue for the past 2 years, but only just found this thread.

For what it's worth, I am willing to volunteer to help address this. I am not sure what the policy is for accepting pull requests from those outside the core team, but I thought I'd put that out there.

[andreimargeloiu]

+1, this is a breaking feature for anyone doing research and quantitative work where we need to rapidly experiment until we find what works well so that we can port it into a standalone script.

[man-shu]

Maybe most people are already aware of this workaround, but here's what I do:

1. open a bash terminal session on the remote machine
2. run tmux on that
3. run ipython inside that tmux session

[andytwigg]

+1 for this

[jucor]

From https://xkcd.com/2881/ ... We'll have a fantastic trip full of machine learners, research engineers, and data scientists 😅

[Abrackadabra]

IIUC if I want to have a ipykernel running remotely, I have two choices:

• Run the kernel through a remote jupyter instance, lose pylance functionality because it does not support remote kernels
• Connect as a Remote, run the kernel through VSCode itself by pointing it to an interpreter. Then kernel state does not survive a VSCode window reload.

Extending the second approach should be straightforward

[HeatPhoenix]

Wanted to give my +1 every way possible.

[arkaprava08]

+1 to this issue

[chirpio76]

+1 having this would be great!

[skambha6]

+1 to this issue

[han-steve]

+1 Is there any initiative to start this feature?

[doronbl]

+1 to this issue

[aflag]

It would be nice if, overall, the language server didn't die when restarting vscode. It's not just notebooks

[jasoncausey]

+1 to this issue.

[rsargent]

Consider supporting the new Jupyter kernel API to allow server side execution to continue with disconnected clients, and for clients to pull updates when they reconnect. jupyterlab/jupyterlab#2833

…

-- Randy

On Tue, Apr 9, 2024 at 4:16 PM Jason L Causey ***@***.***> wrote: +1 to this issue. — Reply to this email directly, view it on GitHub <#3998 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACPJ44HL72OZJB5EYHVZITY4REBHAVCNFSM4T5KLQE2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBUGU4TOOJTGEYQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[troubadour-hell]

+1 to this issue.

[ntoshev]

+1

[Draxran]

+1

[alephpi]

+1

[saultshaker]

+1