Large jupyter cell output causes OOM crash

[LumpBloom7]

Type: Performance Issue

When opening a specific notebook file, which has a cell with a lot of image output, vscode would use up a lot of memory in an attempt to display it, before crashing with an OOM error.

Though oddly enough sometimes it recovers, and successfully renders the content of the cell but the probability of it happening is slim.

This happens on a clean installation of vscode and vscode insiders, both local and browser versions.

This doesn't seem to happen on the regular jupyter client.

Reproduction steps:

1. Download notebook file: https://1drv.ms/u/s!Aubk29qavf4c1UXHhRwxZnmvqnJ5?e=vM0sLQ
2. Open the notebook and scroll down to somewhere around the "part 2" or "Part 3" headers, if not lower

Results

Vscode will chew through all available system memory, before crashing.

Expectations

VSCode doesn't crash

VS Code version: Code 1.73.1 (6261075, 2022-11-09T04:27:29.066Z)
OS version: Windows_NT x64 10.0.22623
Modes:
Sandboxed: No

System Info

Item	Value
CPUs	AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (8 x 2096)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	undefined
Memory (System)	17.88GB (13.73GB free)
Process Argv	--crash-reporter-id 4b2ec309-8248-4027-b776-d43c905f0cb8
Screen Reader	no
VM	0%

Process Info

CPU %	Mem MB	   PID	Process
    0	    92	  9372	code main
    0	   191	  1324	   window (Visual Studio Code)
    0	   290	  3772	   extensionHost
    0	    76	 16700	     "C:\Users\Derrick\AppData\Local\Programs\Microsoft VS Code\Code.exe" --ms-enable-electron-run-as-node "c:\Users\Derrick\AppData\Local\Programs\Microsoft VS Code\resources\app\extensions\json-language-features\server\dist\node\jsonServerMain" --node-ipc --clientProcessId=3772
    0	    85	  4908	   issue-reporter
    0	    27	  9548	   crashpad-handler
    0	    39	 10432	   utility-network-service
    1	   117	 14128	   gpu-process
    0	   115	 15344	   shared-process
    0	    70	  8524	     ptyHost
    0	    73	  8956	     fileWatcher

Workspace Info

;

Extensions (37)

Extension	Author (truncated)	Version
beardedicons	Bea	1.2.0
catppuccin-vsc	Cat	2.4.0
vscode-markdownlint	Dav	0.48.1
dotnet-test-explorer	for	0.7.8
codespaces	Git	1.13.2
copilot	Git	1.62.7527
remotehub	Git	0.46.0
vscode-pull-request-github	Git	0.54.1
vscode-fix-checksums	leh	1.1.0
git-graph	mhu	1.30.0
fluent-icons	mig	0.0.18
csharp	ms-	1.25.2
jupyter-keymap	ms-	1.0.0
jupyter-renderers	ms-	1.0.12
remote-wsl	ms-	0.72.0
cpptools	ms-	1.12.4
hexeditor	ms-	1.9.9
powershell	ms-	2022.11.0
remote-repositories	ms-	0.24.0
test-adapter-converter	ms-	0.1.6
vs-keybindings	ms-	0.2.1
vsliveshare	ms-	1.0.5788
gradle-language	nac	0.2.3
indent-rainbow	ode	8.3.1
resxpress	Pra	4.9.0
java	red	1.13.0
errorlens	use	3.6.0
intellicode-api-usage-examples	Vis	0.2.6
vscodeintellicode	Vis	1.2.29
vscode-gradle	vsc	3.12.6
vscode-java-debug	vsc	0.47.0
vscode-java-dependency	vsc	0.21.1
vscode-java-pack	vsc	0.25.7
vscode-java-test	vsc	0.37.1
vscode-maven	vsc	0.40.1
vscode-icons	vsc	12.0.1
markdown-all-in-one	yzh	3.5.0

[Bill-M-DN]

I have exactly the same issue with a large amount of image output inside vscode running a notebook.

I am going to set up standard Jupyter and see if I can duplicate the problem there.

[rebornix]

I can reproduce this issue, sometimes it crashed but sometimes it becomes responsive after waiting for minutes.

Profiling this and it turns out that the process is stuck at sending output buffers to the webview

From the comment it seems we might be aware of this

vscode/src/vs/workbench/contrib/notebook/browser/view/renderers/backLayerWebView.ts

Lines 1341 to 1343 in 5cc938b

	// TODO@jrieken - the message can contain "bytes" and those are transferable
	// which improves IPC performance and therefore should be used. However, it does
	// means that the bytes cannot be used here anymore

though I'm not sure why it's smooth sending data from extension host to renderer, but so slow when sending to the webview.

[Bill-M-DN]

FWIW: I apologize, I meant to get back to this and post my results. I did as I said, I fired up a standard jupyter hub, and ran the same notebooks. No issues. I've been running that since on all work, and again no issues. Hope that helps.

[mjbvz]

Looks like there's a cell with thousands of output items. I created a smaller version of the notebook which slows down VS Code but does not crash it so we can investigate this: big-small.ipynb.zip

---

Profiling, it seems that no one call cause the the issue. Instead the issue appears to be that we're performing certain operations thousands of times as fast as possible

Here's sending a profile of where we're sending outputs to the webview one by one:

And here's us rendering outputs in the webview one by one:

The actual rendering is relatively fast by there's a lot of overhead to handling each message

---

Two proposals:

1. Limit the number of output items we try rendering here. We already have limits in some places

2. Batch operations. The createOutput method for example should let callers pass in a bunch of outputs instead of invoking it for each output

   This would let us pass all the outputs to the webview in a single request instead of rendering them one at a time

[mjbvz]

This seems fixed after 9fd77b4. Opening the original notebook, I no longer see an OOM crash or lengthy freeze

The notebook does still jump around a little while loading content and there's likely plenty of further optimization opportunities, but I'm closing this since the original issue seemed to be resolved