[Triage] WebSocket Errors on AML compute instance with Jupyter Notebooks

[YnnamTenob]

Issue Type: Bug

After the July update 4 days ago connections to compute instances become unstable very quickly. VSCode reports socket Errors.
[Error - 2021-08-08 22:49:35.458] Websocket 6 error sending data: Error: WebSocket is not open: readyState 3 (CLOSED)
[Error - 2021-08-08 22:49:41.032] Websocket 7 error sending data: Error: WebSocket is not open: readyState 3 (CLOSED)

These errors appear to render VSCode unusable du to slowness. during this episode VSCode exhibits High Memory and CPU usage. The only way to get VSCode working again is to esstablish a new connection with the remote compute resource; however after only a few minutes that connection experiences the same error.

VS Code version: Code 1.59.0 (379476f0e13988d90fab105c5c19e7abc8b1dea8, 2021-08-04T23:13:12.822Z)
OS version: Windows_NT x64 10.0.17134
Restricted Mode: No
Remote OS version: Linux x64 5.4.0-1043-azure

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz (6 x 2592)
GPU Status	2d_canvas: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: enabled opengl: enabled_on rasterization: enabled skia_renderer: enabled_on video_decode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.74GB (6.12GB free)
Process Argv	--crash-reporter-id 8af295f0-b048-43c3-a501-083ca0c08aa2
Screen Reader	no
VM	0%

Item	Value
Remote	Azure ML: cpu-ds3-v2-tenob-m
OS	Linux x64 5.4.0-1043-azure
CPUs	Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz (4 x 2397)
Memory (System)	13.67GB (1.76GB free)
VM	0%

Extensions (9)

Extension	Author (truncated)	Version
vscode-ai-remote	ms-	0.4.0
vscode-azurecognitivesearch	ms-	0.2.0
python	ms-	2021.8.1105858891
vscode-pylance	ms-	2021.8.0
jupyter	ms-	2021.8.1195043623
vscode-ai	ms-	0.6.24
vscode-ai-remote	ms-	0.4.0
azure-account	ms-	0.9.8
vscode-sanddance	msr	3.3.0

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
pythonvspyt700cf:30270857
pythonvspyt602:30300191
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstes627:30244334
pythonvspyt639:30300192
pythontb:30283811
pythonvspyt551:30345470
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
pythonvsuse255:30340121
vscod805cf:30301675
pythonvspyt200:30340761
vscextlang:30333561
binariesv615:30325510
vsccppwt:30329788
pythonvssor306:30344512
bridge0708:30335490
vstre464cf:30346473

[roblourens]

I don't know of anything in this area on vscode's end that changed in this release, @SiddhanthUnnithan any idea?

[YnnamTenob]

Thanks @roblourens. Perhaps nothing changed; however, the experience is less than stellar. When does @alexdima return frm vacation?

[SiddhanthUnnithan]

Thanks for letting us know @YnnamTenob -- adding @tbombach to this thread to help triage.

[YnnamTenob]

started again a few ahours ago. no more than one minute after connecting the WebSocket errors out

erverStartTime====
578d1df7-8231-4991-b2e9-039a3e469e1b: end
[Info - 2021-08-10 03:32:21.018] Websocket server listening on: 8890
[Info - 2021-08-10 03:32:21.019] VSCode url: cpu-ds3-v2-tenob-m-8890.eastus.instances.azureml.ms/ws
[Error - 2021-08-10 03:33:29.813] Websocket 1 error sending data: Error: WebSocket is not open: readyState 3 (CLOSED)
[Error - 2021-08-10 03:33:49.351] Websocket 2 error sending data: Error: WebSocket is not open: readyState 3 (CLOSED)
[Error - 2021-08-10 03:34:17.931] Websocket 3 error sending data: Error: WebSocket is not open: readyState 3 (CLOSED)

[eduedix]

I have been having the same issue. Please let us know what I can share to provide more details on this.

[alexdima]

@YnnamTenob Looking at logs shared in #1234 , it appears to me that the remote extension host becomes unresponsive. Could you perhaps try to use extension bisect to see if one of your extensions is causing very high CPU usage of the remote extension host?

• run F1 > Help: Start Extension Bisect
• (this will restart VS Code with half of your extensions disabled)
• try to do the steps which reproduce the problem
• if the problem reproduces, click "This is bad" (this narrows down the problematic extension to the half which was enabled)
• if the problem does not reproduce, click "Good now" (this narrows down the problematic extension to the half which was disabled)
• these steps will repeat until the problematic extension is found

initial	step	final

[YnnamTenob]

@alexdima I'm not sure how this might work when bisect starts it disables everything I need to do work to detect the problem so I can't determine if the problem exists because I cannot run the notebook. There is no way for me to say if its good or bad. I said it was bad bisect said it was done and it was a problem in code.

[alexdima]

@YnnamTenob Could you perhaps try to reduce the extensions you are using to the bare minimum manually using Disable Extension? I understand that you cannot do what you intended to, but by disabling the extensions and then doing something simple, like editing a plain text file, it would help troubleshoot where a problem is occurring.

When I have seen this in the past, it was often the case that one of the extensions would run up the extension host process to 100% CPU, which would trigger our UI to believe connection was lost to the remote extension host.

(could you perhaps take a look at ps to check if there is any process running at 100% CPU)?

[YnnamTenob]

@alexdima when I am connected to the Azure ML Remote all extensionsions are disabled with the exception of:
Azure Account v0.9.8
Azure Cognitive Search v0.2.0
Azure Machine Learning v0.6.25
Azure Machine Learning - Remote v0.4.0
Jupyter v2021.8.1236758218
Pylance v2021.8.3
Python v2021.8.1159798656
SandDance for VSCode v3.3.0
.NET Interactive Notebooks v1.0.2409020

I think I can only turn off Sandance and Azure Cognitive Search I need all the rest to do my work. I have done that.

[YnnamTenob]

Yesterday was a good day VSCode behaved well did not give me any socket issues and did not hog the memory - cpu. Today VSCode is once again up to no good. I have determined that the problem is in how VSCode reacts when it's websockets connections are failing in this remote mode. VSCode is not able to gracefully cope with this and in turn makes development difficult if not impossible if the connection to Azure is weak or intermittent. I don't know if the intermittent behavior is caused by Azure itself or if it is a function of my network connectivity. I am always connected via gigabit fibre but that is not sufficient evidence that my network is not the issue. In a nutshell I think you guys need to rethink how VSCode behaves under these conditions. I think I have a solid connection but in any case VSCode should not go into fits if the connection fails on either end.

[eduedix]

I have determined that the problem is in how VSCode reacts when it's websockets connections are failing in this remote mode. VSCode is not able to gracefully cope with this and in turn makes development difficult if not impossible if the connection to Azure is weak or intermittent. I don't know if the intermittent behavior is caused by Azure itself or if it is a function of my network connectivity.

I had the same experience so far. The problem occurs due to the failing websockets in the remote mode. Though I got to mention that my network connection was not stable, though development in Jupyter notebook in the meantime was smooth and developing in VSCode was not possible for me.

[alexdima]

Thank you for following up.

@sevillal Given the information we have at this time, I don't believe this to be an issue with VS Code core, but with the the connection created by the ms-toolsai.vscode-ai-remote extension.

[edgBR]

Hi,

Same problem here. I have also the same situation than @eduedix mentions. Connection is amazing and jupyterlab works fine but VScode does not.

BR
E

[alexdima]

@sevillal @greazer should we move this issue somewhere else for better tracking it? It is about the connection created by the ms-toolsai.vscode-ai-remote extension.

[eduedix]

I started to get websocket errors again with stable internet connection.

[Info - 2021-11-24 10:11:04.264] Local TCP port: 58203
[Error - 2021-11-24 10:44:43.875] Websocket 16 error sending data: Error: WebSocket is not open: readyState 3 (CLOSED)
[Error - 2021-11-24 10:45:28.429] Websocket 17 error sending data: Error: write EPIPE
[Error - 2021-11-24 10:45:30.636] Websocket 18 error sending data: Error: write EPIPE
[Error - 2021-11-24 10:45:44.747] Websocket 19 error sending data: Error: write EPIPE
[Error - 2021-11-24 10:46:48.234] Websocket 20 error sending data: Error: write EPIPE

I guess this has to do with the amount of data being transferred via the socket. The failures occur when I create plots with many data points in vscode notebook. You can try it out by line charting 500k datapoints.

[sevillal]

@alexdima transfered to vscode-tools-for-ai

[sevillal]

@eduedix do you have a sample notebook to repro the issue? Thanks

[shsuman]

Closing this as there has been no response for over 7 days. Please reopen if need be.