typescript.tsserver.maxTsServerMemory ignored if > 4 GB

[AdarLieberDembo-at]

Issue Type: Bug

I run VSCode on a very large typescript codebase, and have noticed that when opening certain files, the typescript extension service (tsserver.js) crashes with an OOM, restarts, and crashes again, ad nauseum.

I edited tsserver.js and added some instrumentation to inspect v8's heap limits, and saw that even though our .vscode/settings.json contains "typescript.tsserver.maxTsServerMemory": 8192, the heap_size_limit was never higher than 4 GB. But, the setting was being respected by tsserver.js, as if I dropped it below 4096, heap_size_limit shrank accordingly.

The first comment in #33524 contained an innocuous statement about the maximum heap size possible in Node (4096 MB). I investigated this, in the VSCode, Electron, Node.JS, and v8 codebases and I believe the root cause is the use of v8 pointer compression. If my understanding of the v8 heap management code is correct, when enabled, pointer compression caps the v8 heap at 4 GB.

Electron was modified to enable pointer compression in electron/electron#21464, which shipped as part of Electron 9.0.0. VSCode updated its Electron dependency from 8.4.0 to 9.1.0 in #102011, which shipped as part of 1.49.0 in September of 2020.

I think azmenak/tsserver-bridge, while ostensibly obselete due to the (now proper) plumbing of typescript.tsserver.maxTsServerMemory, can serve as a workaround for this as it runs tsserver.js in a proper Node.JS instance rather than in a Node.JS-like Electron instance (with the ELECTRON_RUN_AS_NODE environment variable set). I haven't tried this yet, though.

VS Code version: Code 1.57.1 (507ce72, 2021-06-17T13:28:32.912Z)
OS version: Darwin x64 20.5.0
Restricted Mode: No

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (12 x 2600)
GPU Status	2d_canvas: enabled gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on oop_rasterization: enabled opengl: enabled_on rasterization: enabled skia_renderer: disabled_off_ok video_decode: enabled webgl: enabled webgl2: enabled
Load (avg)	3, 2, 2
Memory (System)	32.00GB (3.98GB free)
Process Argv	--crash-reporter-id af659216-0094-4f0b-a191-b467110a2413
Screen Reader	no
VM	0%

Extensions (13)

Extension	Author (truncated)	Version
bracket-pair-colorizer-2	Coe	0.2.1
vscode-eslint	dba	2.1.23
gitlens	eam	11.5.1
prettier-vscode	esb	8.0.1
terraform	has	2.13.0
vscode-docker	ms-	1.14.0
python	ms-	2021.6.944021595
vscode-pylance	ms-	2021.6.3
jupyter	ms-	2021.6.999662501
remote-containers	ms-	0.183.0
vscode-groovy-lint	Nic	1.4.0
ejs-language-support	Qas	0.0.1
vscode-yaml	red	0.20.0

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
pythonvspyt602:30300191
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492cf:30256860
pythonvspyt639:30300192
pythontb:30283811
vspre833:30321513
pythonptprofiler:30281270
vsdfh931cf:30280410
vshan820:30294714
vscorecescf:30322572
pythondataviewer:30285071
vscus158:30321503
pythonvsuse255:30323308
vscorehov:30309549
vscod805cf:30301675
vscextlang:30322306
binariesv615:30325510

[AdarLieberDembo-at]

I think azmenak/tsserver-bridge, while ostensibly obselete due to the (now proper) plumbing of typescript.tsserver.maxTsServerMemory, can serve as a workaround for this as it runs tsserver.js in a proper Node.JS instance rather than in a Node.JS-like Electron instance (with the ELECTRON_RUN_AS_NODE environment variable set). I haven't tried this yet, though.

Quick update: I gave this a shot and it worked!

[mjbvz]

@deepak1556 Are you aware of any limits on v8's heap size related to electron/Chromium changes?

[mjbvz]

@AdarLieberDembo-at Are you seeing this issue in the latest VS Code build? If so, would you be willing to help investigate since it sounds like you already have some background in this area

[MarcCelani-at]

This is still an issue for us and it's going to get worse. Electron is having some discussion about releasing builds without pointer compression, but there isn't a commitment yet.

This is concerning because our tsserver-bridge workaround may become more difficult to maintain if we want to move to yarns plug'n'play linker (the node_modules directory will disappear, and it looks like tsserver-bridge may assume it exists).

[MarcCelani-at]

It looks like there is a little bit of traction in electron/electron#31330

[MarcCelani-at]

The electron folks suggest:

"It should work just fine to set the v8_enable_pointer_compression = false variable when compiling."

Can vscode build its own version of electron with pointer compression off?

[deepak1556]

Sorry for the delay in getting to this issue, @mjbvz the heap limit is due to the pointer compression feature from v8 https://v8.dev/blog/v8-release-92#shared-pointer-compression-cage and we cannot disable the limit at runtime, it has to be done by disabling this feature at build time. Disabling pointer compression is not an option, since it will deviate our runtime from upstream electron and also get rid of the memory savings, sandbox features that chromium is working on.

Having said that, we do have considerable workaround for the use cases mentioned in this issue. Since our remote server is based on vanilla node that comes with pointer compression disabled. The tsserver extension running under this environment shouldn't face the heap limit. So I would suggest to wait for the basis server launch and suggest that as an alternate path.

@MarcCelani-at @AdarLieberDembo-at we will have considerable workarounds for this issue in upcoming versions with certain new features of our remote server, I will paste more info once the features are available for use.

[MarcCelani-at]

Excellent! Do you have a rough timeline of when this will be available? This has become a blocker for adopting TS 4.6 as it is not working with tsserver-bridge. We are working for a solution to that problem, but if a new vscode workaround is coming soon we can just wait for that.

[MarcCelani-at]

Also when you say remote server...does that mean what I think it means? Currently we do not require internet access for development and we'd like to maintain that option if possible. We also patch typescript so that would need to be supported as well.

[MarcCelani-at]

@deepak1556 , do you mind taking a look at my last question? We could potentially patch tsserver-bridge but we need to know if it is worth it based on what you said above, and that depends on my question.

[jdelman-stripe]

we do have considerable workaround for the use cases mentioned in this issue. Since our remote server is based on vanilla node that comes with pointer compression disabled. The tsserver extension running under this environment shouldn't face the heap limit. So I would suggest to wait for the basis server launch and suggest that as an alternate path.

What are the workarounds you're currently using?

[deepak1556]

You should now be able to connect with any remote using our tunnels feature, the remote server runs with pointer compression disabled and the extensions should run fine with increased memory limit. As for the VSCode desktop, we won't be disabling pointer compression to align with upstream Electron.