No longer responding when opening a big file?

[slayerxj]

When I click the file (a 4M JSON file) in the file list panel in the left, the content of this file is correctly displayed but without syntax highlighting. And after that VS Code is no longer responding, I thought it's parsing the file, so I waited for a while. Then it alerts "Visual Studio Code is no longer responding: Exit or Keep waiting".

Windows Task Manager shows it's still "Running" (rather than "Not responding"), but even I select "Keep waiting" and wait for over half an hour, the editor is still no responding.

I thought the JSON file is common, and maybe using a smaller file will also reproduce this issue. If you can't reproduce this issue I'll do some study and try to provide sample.

Windows 7 64bit

[alexdima]

@slayerxj Can you please share the JSON file with me?

[slayerxj]

@alexandrudima I did some tests and found out this phenomenon seems to be related a special code.

fs.writeFile("record.js", "module.exports = " + JSON.stringify(javaScriptObject), function (err) {
    if (err) {
        throw err;
    }
});

And I can reproduce this phenomenon every time using
https://raw.githubusercontent.com/slayerxj/anicraw2/socket/record/result.js
This file can be correctly loaded by Node.js.

Now after the tests I no longer think this file is common, so it's OK if you don't see it as an issue.

[alexdima]

to investigate if this is a dup of #94 (long line containing multi byte characters)

[slayerxj]

@alexandrudima I can open that CSS file without any problem (and syntax highlight shown up very quickly), so I don't think they're same.

And I can reproduce this phenomenon both on the computer in the company which is a workstation and the computer at home which is also pretty fast. So you can just open the file I attached to have a try.

[alexdima]

TL;DR the file will now open, no hang, the text won't be colorized.

I looked into this for quite a while, logging all the way down to the C++ code in oniguruma, trying to find any possible mistakes or low hanging caching opportunities. The underlying problem is due to how TextMate syntaxes are specified, where a list of regular expressions must be searched for in the current string in the order specified in the grammar. Then the regular expression that matched closest to the current scanning position is chosen.

This often times leads to unnecessary regex matching at far-ahead positions. For example, in this specific file I could see the comment-line rule of the JavaScript tokenizer matching numerous times inside the strings containing "http://....". This is due to the comment-line rule being specified in the syntax file (via various state transitions) "before" the qstring-double which means that it will always be checked first. One can argue that strings should be checked first, but even if I change the syntax and reorder the rules such that qstring-double is before comment-line, I can easily produce the counter-example where there is a test file with numerous comments and unnecessary matches of qstring-double.

I have checked what others editors using TextMate grammars do and I found that Sublime refuses to colorize any line longer than 16384 (2^14) chars. From my testing, I have found acceptable times (below 50ms tokenization) for lines up to 20k, so I have set our limit to 20k.