color of semantic highlighting is not rendered after pasting code

[Eskibear]

I can stably reproduce it in both stable and insiders editions. See below screenshot, the color #4ec9b0 was not rendered. (There's another bug, if theme-selectors is too long, I cannot see the token type in the inspector window due to overflow, that's why I cut them in the screenshot)

I created the original issue in redhat-developer/vscode-java#1421 when we testing the RC build of vscode-java v0.61.0 .

Steps to reproduce:

• Open an empty folder and create a new file foo/bar/apiImpl/Hello.java:

package foo.bar.apiImpl;
public class Hello { }

• Semantic highlighting should be working well, you can observer apiImpl.
• Cut the first line (package.....), type some whitespaces, and paste it back.
• Boom! Now you see the color fallbacks to syntax highlighting.
• If you modify the content, e.g. appending a whitespace to any line, the color would get back to work.

VSCode version:

Version: 1.44.2
Commit: ff915844119ce9485abfe8aa9076ec76b5300ddd
Date: 2020-04-16T17:07:18.473Z
Electron: 7.1.11
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Darwin x64 19.4.0

and

Version: 1.45.0-insider
Commit: a0fe37870c42e0704a492cdc6b8550b4cf23f63c
Date: 2020-04-30T05:36:26.465Z
Electron: 7.2.2
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Darwin x64 19.4.0

[fbricon]

Similar issues occur after text edits are being applied (eg. organizing imports: redhat-developer/vscode-java#1423) or commenting code (redhat-developer/vscode-java#1424)

[aeschli]

I can reproduce in a Java container (Remote Container: Try a sample). I verified it's not a color lookup issue.

I see the following exception a lot in the log, not sure if related.

  ERR n.subarray is not a function: TypeError: n.subarray is not a function
	at Function._convertToEdits (/home/vscode/.vscode-server-insiders/bin/829230a5a83768a3494ebbc61144e7cde9105c73/out/vs/server/remoteExtensionHostProcess.js:695:355)
	at /home/vscode/.vscode-server-insiders/bin/829230a5a83768a3494ebbc61144e7cde9105c73/out/vs/server/remoteExtensionHostProcess.js:694:771
	at o (/home/vscode/.vscode-server-insiders/extensions/vscjava.vscode-java-test-0.22.3/dist/extension.bundle.js:1:255660)
	at /home/vscode/.vscode-server-insiders/extensions/vscjava.vscode-java-test-0.22.3/dist/extension.bundle.js:1:256157

vscode/src/vs/workbench/api/common/extHostLanguageFeatures.ts

Line 754 in ddff80b

data: newData.subarray(commonPrefixLength, newLength - commonSuffixLength)

[Eskibear]

FYI, I just disabled the extension vscjava.vscode-java-test and tried, the issue was still there. Adding @jdneo maintainer of extension for awareness.

[alexdima]

It looks like the SemanticTokens.data being returned from the extension is not a Uint32Array, but an Array. In the first request, by pure luck, the Array goes through without an exception, but on the second and all subsequent request, we invoke SemanticTokens.data.subarray, which exists on Uint32Array, but not on Array.

So this is an implementation problem. The question is if we should expand our API and accept Array as well as Uint32Array for the data.