[Bug] Setting maxTokenizationLineLength has no effect

[jonatanklosko]

Reproducible in vscode.dev or in VS Code Desktop?

• Not reproducible in vscode.dev or VS Code Desktop

Reproducible in the monaco editor playground?

• Not reproducible in the monaco editor playground

Monaco Editor Playground Code

monaco.editor.create(document.getElementById('container'), {
  value: "[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]",
  language: 'javascript',
  maxTokenizationLineLength: 4,
});

Actual Behavior

The line gets tokenized and highlighted.

Note: when hovering over the line it actually says that tokenization was skipped.

Expected Behavior

Tokenization should be skipped.

Additional Context

The default maxTokenizationLineLength value is 20_000. When I paste a longer line into https://vscode.dev it isn't highlighted, however it gets highlighted in the Monaco playground.

Since Monaco handles highlighting separately, my guess would be that this option is not taken into account.

The on-hover messages is added here (for both Monaco and VSCode) and the TextMate tokenization is skipped here.

[Lazyuki]

This is causing monaco-editor to freeze up the entire UI for several seconds making the browser unresponsive when it's tokenizing a log line containing 50k+ characters. I have a number of custom tokenizers that aren't cheap so ideally monac-editor should respect the maxTokenizationLineLength and skip lines that are too long.