Large file under git control encoding guess can freeze the editor

[ziggi]

• VSCode Version: 1.41 (same on 1.42 insiders build too)
• OS Version: Windows 10 Pro 1909, build 18363.535

Steps to Reproduce:

1. Open a large file (my file is 93k+ lines, 3.96 MiB) and a small one. The files should be in Windows 1251 encoding and VS Code settings.json should contains "files.autoGuessEncoding": true,.
2. Switch tab from small to large.

Does this issue occur when all extensions are disabled?: Yes

It looks like the problem is in jschardet, it was updated from 1.6.0 to 2.1.1.

CPU-20191218T010956.cpuprofile.txt
Profile-20191218T010523.json.txt

[bpasero]

@ziggi great analysis, wanna report it to jschardet maybe?

[ziggi]

I'm not 100% sure what the problem is in jschardet. The problem may be in vscode only.

[bpasero]

@ziggi looks like jschardet to me. We are simply calling this method:

vscode/src/vs/base/node/encoding.ts

Line 215 in b6ae21e

const guessed = jschardet.detect(buffer);

[ziggi]

Ok, done: aadsm/jschardet#53

[creative-acr]

I have the same issue while opening large JS file after updating to version 1.41.1

[bpasero]

@creative-acr do you have "files.autoGuessEncoding": true configured which this issue is about?

[bpasero]

@ziggi to clarify, you only see this with large files that are under git control?

[ziggi]

@bpasero yes

[bpasero]

I can reproduce and it looks like the dirty diff decorator is triggering this (probably ever since we moved to git-fs recently). Though I cannot explain why this performance issue does not show up when just opening the editor where we also guess the encoding. Maybe the diff decorator is not as optimized as the code editor model that reads the file in chunks and reads the file entirely?

@joaomoreno steps to reproduce: aadsm/jschardet#53 (comment)

[bpasero]

I pushed a fix that will limit the buffer for encoding guessing to 65kb. The dirty diff decorator was passing the entire buffer. Filed #88119 as follow up.