Editor becomes unusable for a short while when stepping into VS Code api call

[mjbvz]

Issue Type: Bug

1. While developing an extension
2. Accidentially step into an VS Code api call, such as registerCommand

Bug
This steps into a minified copy of extensionHostProcess.js

The entire editor freezes for a few seconds (I believe while syntax highlighting is kicking in)

The debugger remains frozen for a longer period, so that clicking step out doesn't appear to do anything

Eventually everything becomes interactive again but I hit this issue pretty consistently and it makes debugging an extension kind of painful

VS Code version: Code - Insiders 1.61.0-insider (Universal) (aa93eef, 2021-09-09T05:13:32.970Z)
OS version: Darwin x64 20.6.0
Restricted Mode: No

Extensions (55)

Extension	Author (truncated)	Version
salesforce-soql-editor	all	1.8.0
comment-tagged-templates	bie	0.3.1
docs-view	bie	0.0.9
emojisense	bie	0.9.0
folder-source-actions	bie	0.2.1
jsdoc-markdown-highlighting	bie	0.0.1
markdown-checkbox	bie	0.3.2
markdown-emoji	bie	0.2.1
markdown-yaml-preamble	bie	0.1.0
test-ext	bie	0.0.1
vscode-eslint	dba	2.1.25
typescript-notebook	don	2.0.2
gitlens	eam	11.6.0
tsl-problem-matcher	eam	0.4.0
EditorConfig	Edi	0.16.4
codespaces	Git	1.0.5
vscode-pull-request-github-insiders	Git	2021.9.12069
vscode-drawio	hed	1.6.2
vscode-postfix-ts	ipa	1.9.4
latex-workshop	Jam	8.20.2
vscode-platform-specific-sample	joa	1.5.0
vscode-platform-specific-sample	joa	1.9.0
vscode-memfs	jri	0.0.3
template-string-converter	meg	0.5.3
vscode-azurefunctions	ms-	1.5.1
vscode-azureresourcegroups	ms-	0.4.0
vscode-docker	ms-	1.16.1
vscode-language-pack-de	MS-	1.60.3
vscode-edge-devtools	ms-	1.3.0
python	ms-	2021.10.1215685612-dev
vscode-pylance	ms-	2021.9.2-pre.1
jupyter	ms-	2021.9.1001216636
jupyter-keymap	ms-	1.0.0
remote-containers	ms-	0.194.0
remote-ssh	ms-	0.65.7
remote-ssh-edit	ms-	0.65.7
azure-account	ms-	0.9.8
hexeditor	ms-	1.8.2
js-debug-nightly	ms-	2021.9.417
live-server	ms-	0.2.8
vscode-github-issue-notebooks	ms-	0.0.106
vscode-typescript-next	ms-	4.5.20210908
vsliveshare	ms-	1.0.4761
debugger-for-chrome	msj	4.13.0
vetur	oct	0.34.1
java	red	0.81.0
code-spell-checker	str	2.0.4
rest-book	tan	4.2.0
ayu	tea	1.0.5
luna-paint	Tyr	0.9.0
sort-lines	Tyr	1.9.0
vscodeintellicode	Vis	1.2.14
vscode-java-debug	vsc	0.35.0
vscode-java-dependency	vsc	0.18.7
vscode-java-test	vsc	0.31.3

(2 theme extensions excluded)

A/B Experiments

vsliv695:30137379
vsins829:30139715
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
pythonvspyt602:30291494
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyt639:30291487
pythontb:30258533
pythonvspyt551cf:30291413
pythonptprofiler:30281269
vshan820:30294714
pythondataviewer:30285072
pythonvsuse255:30319630
vscod805cf:30301675
pythonvspyt200:30323110
vsccppwt:30364496
pythonvssor306:30340298
bridge0708:30335490
pygetstartedt2:30353727
dockerwalkthru:30364418
bridge0723:30353136
pythonrunftest32:30353181
pythonf5test824:30361779
javagetstartedt:30350119
pythonvspyt187:30361752
pydsgst2:30361790

[connor4312]

/dupliate #123257

[mjbvz]

@connor4312 I don't think it's the same issue. This is specific to stepping into minified files. Normal debugging works fine for me. Also some of this seems to be related to just opening a large minified file in the editor. I can reproduce some of the hangs just by opening that file without a debug session

I'm reopening but let me know if you still think this is a duplicate

[connor4312]

It looks like the editor is spending a lot of time doing tokenization -- even though the (single) line visible in the viewport is far longer than the tokenization limit. Maybe this can be optimized?

[hediet]

How can I reproduce this with Code OSS?
The extension host source does not seem to be minified.

[alexdima]

If I understand this correctly, this has nothing to do with debugging, and just opening the file exhibits slowness. I've taken an extensionHostProcess.js from a recent VS Code build (i.e. resources/app/out/vs/workbench/services/extensions/node).

extensionHostProcess.zip

It looks like the file contains 115 lines of code, with many long lines (16k characters, 12k characters, etc.). These lines fall short of our default 20k characters limit (editor.maxTokenizationLineLength), so the editor will try to tokenize these lines. I think the TS grammar is one of the most complex grammars we have, it includes some of the most complex regular expressions I've ever seen, and I believe it ends up stumbling on these lines. After adding logging to the TM tokenizer for lines which take longer than 10ms, here is what I see -- at most the UI thread is blocked for 1162ms:

tokenizing a line with 59 chars took 27 ms.
tokenizing a line with 16142 chars took 1095 ms.
tokenizing a line with 1722 chars took 19 ms.
tokenizing a line with 12212 chars took 43 ms.
tokenizing a line with 12479 chars took 45 ms.
tokenizing a line with 12212 chars took 43 ms.
tokenizing a line with 10289 chars took 338 ms.
tokenizing a line with 1209 chars took 48 ms.
tokenizing a line with 1470 chars took 94 ms.
tokenizing a line with 1054 chars took 73 ms.
tokenizing a line with 16872 chars took 1162 ms.
tokenizing a line with 5350 chars took 178 ms.
tokenizing a line with 725 chars took 12 ms.
tokenizing a line with 731 chars took 15 ms.
tokenizing a line with 741 chars took 12 ms.
tokenizing a line with 3431 chars took 344 ms.
tokenizing a line with 4212 chars took 395 ms.
tokenizing a line with 2512 chars took 236 ms.
tokenizing a line with 3278 chars took 310 ms.
tokenizing a line with 5858 chars took 717 ms.
tokenizing a line with 19157 chars took 722 ms.
tokenizing a line with 1067 chars took 69 ms.
tokenizing a line with 9784 chars took 1565 ms.
tokenizing a line with 4513 chars took 73 ms.
tokenizing a line with 6395 chars took 290 ms.
tokenizing a line with 4799 chars took 26 ms.
tokenizing a line with 10239 chars took 141 ms.
tokenizing a line with 5609 chars took 72 ms.
tokenizing a line with 1441 chars took 12 ms.

Given these numbers, I think we should consider reducing the editor.maxTokenizationLineLength for our TypeScript grammar.

For example, if I change the grammar to Go (I know it doesn't make sense, but just to see what the influence of the grammar is) -- at most the UI thread is blocked for 165ms:

tokenizing a line with 16142 chars took 101 ms.
tokenizing a line with 12479 chars took 155 ms.
tokenizing a line with 12212 chars took 165 ms.
tokenizing a line with 10289 chars took 78 ms.
tokenizing a line with 16872 chars took 17 ms.

[alexdima]

@hediet One way to tackle this would be to make this check be language dependent --

vscode/src/vs/workbench/services/textMate/browser/abstractTextMateService.ts

Lines 448 to 451 in ad9e300

	// Do not attempt to tokenize if a line is too long
	if (line.length >= this._maxTokenizationLineLength) {
	return nullTokenize2(this._languageId, line, state, offsetDelta);
	}

and have a different default for [typescript]: { "editor.maxTokenizationLineLength" } and [javascript]: { "editor.maxTokenizationLineLength" }, maybe something like 2000-3000 given the times I see with this grammar. (the JS grammar is basically the TS grammar)

[hediet]

Are default values per language supported?
How can [typescript]: { "editor.maxTokenizationLineLength" } be expressed as default value?

[alexdima]

Are default values per language supported?

Yes, here is an example from markdown configuring editor.wordWrap.

[mjbvz]

Much better performing now. Thanks for the fix!