Fast typing causes semantic highlights to flicker

[JacksonKearl]

Testing #89351.

Open, for instance, https://github.com/microsoft/vscode/blob/2ce17a576c278df08d07213c8cd5b42c605cc8c3/src/vs/workbench/contrib/search/browser/searchEditorInput.ts

Start a string below the import block. Type in that string. As you type faster, the import block will start flickering between semantic highlighting and textmate highlighting. This is distracting

(I use a string to show its not an issue of the document being semantically invalid)

[alexdima]

This is because the TS semantic tokens provider receives (offset,len) from TS and then, if the document changed in the meantime, cannot convert them to (pos,col) and then returns null.

On the API impl side, seeing null is equivalent to throwing away all the tokens.

[alexdima]

This reproduces the best in vscode because a large project is needed to get tsserver to be slower... Also useful:

"editor.tokenColorCustomizationsExperimental": {
	"type": {"fontStyle": "bold italic" },
},