Remote jupyter doesn't reconnect after computer sleeps

[8enmann]

Maybe related: #822

Bug: Notebook Editor, Interactive Window, Editor cells

Steps to Reproduce:

1. Connect to a remote jupyter server
2. Sleep laptop
3. Try to run a cell
   Error: Cannot execute code, session has been disposed. Source: Python (Extension)

The workaround I use is close the notebook editor and reopen it, but this loses all my cells. Remote server still has the session.

Your Jupyter and/or Python environment

Version: 1.47.0
Commit: d5e9aa0227e057a60c82568bf31c04730dc15dcd
Date: 2020-07-09T08:01:54.115Z (2 days ago)
Electron: 7.3.2
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Darwin x64 19.5.0

• Jupyter server running: Remote
• Extension version: v2020.6.91350
• Setting python.languageServer: pylance
• Python and/or Anaconda version: Python 3.7.3
• Virtual environment: conda

Developer Tools Console Output

Error Python Extension: 2020-07-11 21:30:39: [Error: Kernel is dead at f._sendKernelShellControl (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/node_modules/@jupyterlab/services.js:3:510258) at f.sendShellMessage (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/node_modules/@jupyterlab/services.js:3:510027) at f.requestExecute (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/node_modules/@jupyterlab/services.js:3:512579) at _.requestExecute (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/extension.js:1:535559) at _.requestExecute (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/extension.js:48:460661) at f.P.generateRequest (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/extension.js:48:436320) at f.P.handleCodeRequest (/Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/extension.js:48:437313) at /Users/ben/.vscode/extensions/ms-python.python-2020.6.91350/out/client/extension.js:48:450604 at runMicrotasks () at processTicksAndRejections (internal/process/task_queues.js:85:5)]
t.log @ workbench.desktop.main.js:sourcemap:266
workbench.desktop.main.js:sourcemap:2947 Error: Cannot execute code, session has been disposed.

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

[DavidKutu]

Thanks for the issue @8enmann.

[8enmann]

Any updates?

[DavidKutu]

Hi @8enmann, this issue is parked for now, we'll come back to it if other people run into it.

In the mean time I have a question. You mentioned you loose your cells, does that happen even if you save the notebook?

[8enmann]

The problem is not that the cell goes away, it's that I need to restart the notebook in order to run cells since the remote session is disposed. Saving or not doesn't make a difference.

[DavidKutu]

I see, I'm glad there's a workaround. We're not planning to fix this one on our next release. We'll let the issue open for people to upvote it and will revisit it on triage after the august release.

[aray-openai]

I've got the same problem. Is there any way to get at the debug logs for reconnection -- it feels like I don't have good access to what the jupyter plugin is doing to try to reconnect.

[DavidKutu]

@aray-openai those should be on the webview developer tools:
ctrl + shift + P -> Developer: Open Webview Developer Tools

[shikiponn]

Hi vscode team,

I've run into this porblem too.
Resuming from sleep I see my vscode saying Cannot reconnect. Please reload the window. and I can no more access to python interactive where I was running my code.

[DavidKutu]

Hi @shikiponn, please share with us what your Python output tab on the Panel says when the issue happens.

[clayms]

Issue Type: Bug

Same basic problem. It happens when there is even the slightest interruption to my corporate VPN connection. It is so bad for my overseas colleagues, that despite me trying to get them to use the Python extension in VS Code, they refuse, as it is too frustrating to have the session get "disposed" every few minutes. Instead, they choose to use either Jupyter Labs or Notebooks, as those do not have any issue picking up right where it was before the network disruption.

The reported VS Code Python Extension error is: Error: Cannot execute code, session has been disposed.

Why is it that Jupyter Labs or Notebooks do not have any problems reconnecting to the Jupyter Session running on a remote computer after a slight network disruption and now the Python extension for VS Code cannot handle it?

This was not a problem a year or so ago. With the VS Code Python Extension, I used to be able to:

1. start a Jupyter session on a remote computer in VS Code at the office,
2. put my computer to sleep,
3. go home,
4. wake my computer up,
5. log into my corp VPN,
6. start running my code right where I left off in the VS Code Python Extension, everything in memory exactly as I left it.

Why and how was this broken? Why does this still work in Jupyter Labs or Notebooks?

Might be related to what IanMatthewHuff said in this issue comment:

"Jupyter Notebooks don't connect in the same way that we do so it is possible for them to make the connection and for us to not be able to."

Extension version: 2020.9.114305
VS Code version: Code 1.50.0 (93c2f0fbf16c5a4b10e4d5f89737d9c2c25488a3, 2020-10-07T06:10:52.432Z)
OS version: Windows_NT x64 10.0.18363

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz (4 x 2496)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: disabled_off opengl: enabled_on protected_video_decode: unavailable_off rasterization: enabled skia_renderer: disabled_off_ok video_decode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.88GB (6.92GB free)
Process Argv	--crash-reporter-id f652528b-5678-4b12-9ca0-3de6b1e1135f
Screen Reader	no
VM	0%

[rchiodo]

Why is it that Jupyter Labs or Notebooks do not have any problems reconnecting to the Jupyter Session running on a remote computer after a slight network disruption and now the Python extension for VS Code cannot handle it?

This is because of how we manage kernels. When our UI is disposed, we close the kernel connection. Whereas with jupyter you start a server and close kernels when you shutdown the server.

Likely what's happening when you put your computer to sleep, VS code is disposing of our Webview UI, causing our dispose method to be called.

In order to accommodate this we'd need some sort of kernel management UI (other than just closing windows). Something like this bug here:
https://github.com/microsoft/vscode-python/issues/14087

[clayms]

@rchiodo that is exactly the issue. If those features were added, I think that would solve this issue.

Concise description of my experience:
https://github.com/microsoft/vscode-python/issues/14087#issuecomment-705055809

I believe this issue could be rolled up into that issue/feature request.

[clayms]

I should note the usual way I run python.

From PowerShell on local computer:

ssh clay@100.100.100.100

jupyter lab --ip=100.100.100.100

Then copy the displayed link into .code-workspace file:

	"settings":  {
		"python.dataScience.jupyterServerURI": "http://100.100.100.100:8888/?token=5555555555555555555555555"
	 }

Then the VS Code Python Extension works great, at least until a slight network interruption or my local computer goes to sleep.

[8enmann]

Any updates on this? Blocking me from using it entirely.

[rchiodo]

Which part is blocking you? Your computer going to sleep and then reconnecting? Does starting the jupyter server yourself (using a remote connection to a localhost) resolve that problem?

(I haven't tried to see what happens with jupyter when a computer goes to sleep, not sure if the server stays running or not)

We're actively changing the way we connect to remote machines. Part of this will be some form of kernel management. We may also have the same kernel management for local kernels (although not sure how we decide when to kill the kernels).

[8enmann]

Yep, the blocker is that when I have a remote jupyter, when I sleep and wake, I lose all my jupyter state. This means I can't have a persistent session and have to rerun all the cells every time my laptop sleeps.

I just tested with a jupyter on localhost and it worked fine. Unfortunately I do a lot of stuff with GPUs, so I have to run on a remote connection.

Really appreciate your work here. Love VSCode!!

[rchiodo]

Remote kernels should be left running though. If you're using notebooks (and not the interactive window), it should reconnect to the original kernel when you reopen the notebook.

Unless you mean remote in the fact that you're using SSH (then the jupyter connection is actually local because it's started on the SSH machine next to where our extension is running).

Using SSH would require us to support not shutting down local kernels as well.

[8enmann]

I'm not sure what the interactive window is. I use Python: Create New Blank Jupyter Notebook, which I assume is a notebook. I never explicitly close the window on my machine, maybe electron is telling it I closed the window when I sleep my mac? When I unsleep, the window's still there with all the state in it.

The kernel does continue to run on my remote machine. I connect to it using HTTPS over VPN. For some reason VSCode can't reconnect to it.

Maybe it'd work better if I used eternal-terminal port forward from remote so VSCode thinks it's actually a local jupyter?

[rchiodo]

The kernel does continue to run on my remote machine. I connect to it using HTTPS over VPN. For some reason VSCode can't reconnect to it.

If you close the notebook and reopen it, does it reconnect? There's no logic for handling sleep (not sure VS code even has an API to tell us about that). However when reopening a notebook we attempt to reconnect to the original kernel. Or at least we should.

[rchiodo]

Maybe it'd work better if I used eternal-terminal port forward from remote so VSCode thinks it's actually a local jupyter?

Not sure what eternal-terminal port forward is. We don't need the server URI to be localhost. It should just work as long as you're connecting through the 'Specify local or remote Jupyter server for connections'

[michaelsanchez-pingidentity]

I am not using jupyter or python extensions in VSCode and I have the exact same issue. All my extensions are Javascript and React related. Steps to reproduce are mostly identical.

1. log on to Macbook Pro.
2. Start VPN client.
3. Launch VSCode
4. Open SSH connection with VSCode remote to dev server.
5. work work work.
6. Walk away for lunch or other and laptop goes to sleep.
7. When I return and wake it up, laptop still connected to VPN and everything else works.
8. VSCode lost connection. Any attempt to "Reconnect Now", "Reload window", "Retry", or even closing the remote connection and restarting all fail at connecting to any SSH host.

I have to kill my VPN connection and reconnect new to get VSCode to connect to any SSH host. I'm mostly just supplying further "evidence" to help out. A solution would be great but I've decided it's just part of my job description now. shrug

Version: 1.51.1
Commit: e5a624b788d92b8d34d1392e4c4d9789406efe8f
Date: 2020-11-11T01:11:34.018Z
Electron: 9.3.3
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Darwin x64 19.6.0

[joyceerhl]

@michaelsanchez-pingidentity I believe you are encountering a separate issue pertaining to the Remote-SSH extension. You might like to check out the Remote-SSH output tab in VSCode when trying to reconnect and see what the logs say, and please do consider filing a bug on their repo so their team can assist you.

[8enmann]

I believe it tries to reconnect after sleep and then gets
workbench.desktop.main.js:sourcemap:2947 Error: Cannot execute code, session has been disposed.

[zpincus]

I mentioned this in a separate bug (#1378 (comment)) but its relevant here: it seems extremely weird for the notebook session management to be tied to when the UI is unloaded/reloaded.

To be explicit, I here refer to how vscode manages the "local" juptyer servers that it spins up on its own, not the management of kernels on external/existing/"remote" servers that the user has started separately.

In most other cases, vscode tries really hard to maintain user state: e.g. for the embedded terminal session, if you reload the window, the same terminal session is still present. But if you do that for a notebook session, the kernel is killed and restarted. Ideally, the lifecycle rules for notebook sessions should be the same as for terminal sessions -- i.e. notebook sessions should persist through UI reloads, but be killed when the user explicitly closes the notebook tab or the workspace window (perhaps with an alert). Contra #1378, "kill the kernel when and only when I close the tab/workspace" might be sufficient UI for managing the lifecycle of kernels, at least on vscode's "local" servers.

If UI reloads no longer dispose of running notebooks, this would probably also solve the issues everyone raises above, where disconnects over remote-ssh (which can require reloading the UI to resolve if the disconnect has lasted a while) nuke long-running notebook sessions on the remote-ssh host. It's crazy that a jupyter kernel (the "local" kind, auto-started by the vscode-remote extension on the remote machine) would be happy and alive on the remote machine through the whole disconnect, right up until the vscode client manages to reconnect, and then the very first thing the client does as part of the UI reload is to kill the kernel instead of reconnecting to it.

This seems like a major misfeature... should I file it as a separate bug, e.g. "notebook kernels should not be disposed of on UI reload" or something?

[rchiodo]

It is an interesting idea to keep the kernel around until an explicit close of a tab. Meaning closing VS code wouldn't kill anything. You'd have to explicitly close the notebook while VS code was open.

Although this would have to be behind a setting. I would hazard a guess a lot of people that are used to kernels just being cleaned up would be upset if they didn't go away on VS code shutdown.

[zpincus]

I think it would be reasonable to kill the kernels in any circumstance where a terminal session is also killed. So e.g. quitting vscode would reasonably be expected to kill any local jupyter servers that it spun up internally.

It would also be reasonable to add a setting where the kernel would be left running when vscode is quit but the workspace window is left open, since restarting vscode could be expected to restore all state. But I don't think that's necessary either -- again, terminal sessions are killed in such cases.

The main thing though is that it seems (to me at least) unreasonable for reloading the UI to kill local notebook kernels. That is contrary to how the terminal works, and appears to be the root cause of everyone's network-disconnect woes with notebooks running local to remote-ssh hosts...

[rchiodo]

I need to double check but I don't think we (as an extension) can tell the difference between quitting VS code and reloading the UI.

AFAIK, the extension host (the process our extension is running in) is shutdown the same each time. We just get our dispose method called.

Terminals are handled by VS code itself, so it may be able to treat reload differently.

I think we'd need a new API from VS code to distinguish this case.

[zpincus]

I don't think we (as an extension) can tell the difference between quitting VS code and reloading the UI.

Hmm, that is a pickle then!

Will the new "native notebook" feature have this limitation too? Or is that part of the core vscode, more like how terminals are handled?

If the latter, perhaps just waiting for that to roll out (and filing bugs on it if needed) would be the best path forward long-term. But if this issue will carry over into the native-notebook world, do let me know if it would be helpful to file an explicit bug to track the root issue about losing user state on UI reload...

[rchiodo]

Native notebooks will have the same issue. Kernels are controlled by us.

I don't think we need a new bug. Maybe just some more visibility for this one. We can ask VS code for the ability to tell when reload happens versus shutdown and then it would be possible to not kill kernels on reload.

[IanMatthewHuff]

Terminals are handled by VS code itself, so it may be able to treat reload differently.

Pretty sure this is the case, they landed some code semi-recently for supporting this I believe:
microsoft/vscode#20013

[rchiodo]

Reading through the PR for the VS code issue, gave me an idea. Might be too much work, but we could have a heartbeat in the kernel. Say every 10 minutes or something. If VS code doesn't respond, it kills itself.

That should work in the sleep case (well because the kernel will go to sleep too) and would keep the kernel alive for 10 minutes after a shutdown.

[zpincus]

Clever idea!

Might not even need a heartbeat per se? Instead of tearing down the Jupiter server directly on ui reload, set an e.g. 10 minute timer in the server process, after which it tears itself down unless the timer is canceled by a reconnect from vscode. Not sure if that’s dramatically simpler than a full heartbeat system, but perhaps?

[rchiodo]

Related to this.
#3998

[Serebryanskiy]

I've recently started using VS code as a Data Scientist and it helps me a lot with autocompletion and UI. From my point of view, it's much better than the Pycharm. Before VS code I usually set up the docker container on the remote host with project environment inside and jupyter notebook listening on some port. This way I can launch GPU training and safely close the notebook. But with VS code I can't close the notebook or interrupt the connection, since the state of the notebook and the training progress will fail, which is a great issue for anyone who is using jupyter for GPU training. I really love VS code UI and hope there is a way to resolve this issue.

[zpincus]

@Serebryanskiy -- IMO the best current workaround is to use a separate jupyter server with vscode, just like you did before. (It's called "Remote Jupyter" in the vscode lingo, to differentiate it from the "Local Jupyter" server that vscode will start for you, but which goes away when you close vscode etc.)

Note that with vscode-remote, if you run the jupyter server on the same machine that you're connected to remotely, you connect to your "Remote" Jupyter server on localhost (i.e. local to where vscode-remote is running). To connect to a "Remote" Jupyter server, just click on the "Jupyter Server" button at the lower right of the window and enter the correct URL with token...

It's not perfect; there are a few rough edges compared to using the "Local Jupyter" (e.g. #8285 and #7610) but they're getting fixed over time.

[vpatil131]

@Serebryanskiy -- IMO the best current workaround is to use a separate jupyter server with vscode

Thanks, this seems to be the best workaround so far. Manually started a Jupyter server on remote machine, connected VSCode to it using "Jupyter Server: Remote" option and verified through Jupyter logs (on remote machine) that VSCode window reload, VPN disconnect or quitting and restarting VSCode doesn't kill the current kernel which means all Jupyter state variables are retained.

[DonJayamanne]

@vpatil131 @zpincus @8enmann @michaelsanchez-pingidentity
Its my understanding that this issue is focused on re-connecting to a remote computer
By Remote I mean the Jupyter server is running on a remote Machine and VSCode is locally running and you connect to the remote Jupyter server using the remote Jupyter Connection feature in the Jupyter extension.
If however you launch VS Code on the remote maching using Remote SSH, then VS Code is running on the remote machine, hence all kernels are also started on the same remote machine, i.e. everything is local with respect to where VS Code is running. Thus when using Remote SSH, the kernels are started locally on that machine.

If anyone is running into any issues with re-connecting to remote jupyter server, please do feel free to comment here and I'll re-open this issue, or please feel free to create a new issue.
It is my understanding that there are no problems (at least anymore) with regards to re-connecting to remove jupyter server kernels, i.e. this works.

If on the other hand someone is starting a local kernel, and the connection dies after the computer sleeps, then please upvote or comment on the issue #3998

hard to maintain user state: e.g. for the embedded terminal session, if you reload the window, the same terminal session is still present. But if you do that for a notebook session, the kernel is killed and restarted. Ideally, the lifecycle rules for notebook sessions should be the same as for terminal sessions -- i.e. notebook sessions should persist through UI reloads, but be killed when the user explicitly closes the notebook tab or t

The issue that would address this is #3998

I am not using jupyter or python extensions in VSCode and I have the exact same issue. All my extensions are Javascript and React related. Steps to reproduce are mostly identical.

log on to Macbook Pro.
Start VPN client.
Launch VSCode
Open SSH connection with VSCode remote to dev server.
work work work.

Here too, everything is local in the context of VS Code, hence #3998 would address this problem

Closing this issue for now, as I believe there has been some confusion around what is local vs remote hence we have duplicate issues.