Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash on openning a json file ~7Mb size #42260

Closed
raphaelquati opened this issue Jan 27, 2018 · 8 comments
Closed

Crash on openning a json file ~7Mb size #42260

raphaelquati opened this issue Jan 27, 2018 · 8 comments
Assignees
@rebornix
Labels
bug Issue identified by VS Code Team member as probable bug editor-textbuffer Editor text buffer freeze-slow-crash-leak VS Code crashing, performance, freeze and memory leak issues important Issue identified as high-priority verification-needed Verification of issue is requested verified Verification succeeded
Milestone
February 2018

Comments

@raphaelquati
Copy link

  • VSCode Version: Code - Insiders 1.20.0-insider (84991c4, 2018-01-23T10:59:53.150Z)
  • OS Version: Windows_NT x64 10.0.17074
  • Extensions (18)
    Extension Author (truncated) Version
    jenkins-status ale 0.5.1
    jshint dba 0.10.15
    vscode-eslint dba 1.4.4
    xml Dot 1.9.2
    tslint eg2 1.0.24
    vs-code-xml-format fab 0.1.5
    beautify Hoo 1.3.0
    Go luk 0.6.73
    mssql ms- 1.3.0
    csharp ms- 1.13.1
    PowerShell ms- 1.5.1
    vscode-docker Pet 0.0.23
    java red 0.17.0
    vscode-icons rob 7.19.0
    jenkinsfile-support sec 0.1.0
    vscode-java-debug vsc 0.5.0
    vscode-java-pack vsc 0.2.0
    ascii-unicode-escape yar 0.1.0

Steps to Reproduce:

  1. Open VSCode (insider) - Open a json file (~7Mb)

Reproduces without extensions: Yes

image

image

image
@raphaelquati
Copy link
Author

More info:

The file is not formatted. So json file has 1 line.

"Formatting" the file, increases the size of file, but VSCode opens the file correctly.

image

@rebornix rebornix added the editor-textbuffer Editor text buffer label Jan 29, 2018
@rebornix rebornix modified the milestones: On Deck, Backlog Feb 12, 2018
@alexdima
Copy link
Member

@raphaelquati Can you please provide the file that causes the crash on open. Otherwise, a very similar one (or even a generated one) so we can reproduce the crash?

@alexdima alexdima added the info-needed Issue requires more information from poster label Feb 26, 2018
@raphaelquati
Copy link
Author

The file contains sensitive data. I need to obfuscate the file.

@alexdima
Copy link
Member

@raphaelquati Here is a nodejs script I put together which you might use:

const fs = require('fs');
let contents = fs.readFileSync('input.json').toString();

contents = contents.replace(/[a-z]/g, 'a');
contents = contents.replace(/[A-Z]/g, 'A');
contents = contents.replace(/[0-9]/g, '1');
contents = contents.replace(/[^aA1":,.\[\]{}\s\n]/g, 'x');

fs.writeFileSync('output.json', contents);

e.g.
input:

{"type":"FeatureCollection","crs":{"type":"name","properties":{"name":"urn:ogc:def:crs:OGC:1.3:CRS84"}}, "features": [
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[8.5068,47.2858],[8.5039,47.2731],[8.5021,47.2613],[8.5007,47.2613],[8.4724,47.2634],[8.4773,47.2889],[8.5068,47.2858]]]},"properties":{"gmdnr":1,"name":"Aeugst am Albis","kanton":"ZH","max_18":15.41}},
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[8.4445,47.2662],[8.4352,47.2476],[8.414,47.2437],[8.4101,47.2484],[8.4014,47.2682],[8.4261,47.2791],[8.4445,47.2662]]]},"properties":{"gmdnr":10,"name":"Obfelden","kanton":"ZH","max_18":15.41}},
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[8.5002,47.5429],[8.4904,47.528],[8.4615,47.5151],[8.4406,47.5381],[8.4561,47.5582],[8.5002,47.5429]]]},"properties":{"gmdnr":100,"name":"Stadel","kanton":"ZH","max_18":15.41}},
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[8.0675,47.0323],[8.06,46.9957],[8.035,46.9994],[8.0395,47.0211],[8.058,47.0288],[8.0675,47.0323]]]},"properties":{"gmdnr":1001,"name":"Doppleschwand","kanton":"LU","max_18":22.97}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[8.0228,46.9765],[8.0097,46.9804],[8.0169,46.9831],[8.0228,46.9765]]],[[[8.1292,47.0127],[8.1442,47.0021],[8.1462,46.9706],[8.1623,46.9635],[8.1569,46.9334],[8.1222,46.9092],[8.0858,46.9061],[8.081,46.9149],[8.12,46.9284],[8.1354,46.9435],[8.1192,46.9608],[8.0686,46.9727],[8.06,46.9957],[8.0675,47.0323],[8.0683,47.0335],[8.0905,47.0352],[8.1292,47.0127]]]]},"properties":{"gmdnr":1002,"name":"Entlebuch","kanton":"LU","max_18":0}},
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[8.0621,46.9096],[8.0698,46.8957],[8.0705,46.8776],[8.0657,46.8596],[8.0422,46.8365],[8.0644,46.8077],[8.0469,46.7871],[8.0314,46.7903],[7.9846,46.775],[7.954,46.7917],[7.9535,46.7942],[7.9508,46.7997],[7.9479,46.814],[7.973,46.8477],[7.9574,46.885],[7.9923,46.9146],[8.0151,46.9179],[8.0621,46.9096]]]},"properties":{"gmdnr":1004,"name":"Flühli","kanton":"LU","max_18":22.97}},
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[8.06,46.9957],[8.0686,46.9727],[8.1192,46.9608],[8.1354,46.9435],[8.12,46.9284],[8.081,46.9149],[8.0858,46.9061],[8.0698,46.8957],[8.0621,46.9096],[8.0649,46.9346],[8.0454,46.9355],[8.0476,46.963],[8.0228,46.9765],[8.0169,46.9831],[8.035,46.9994],[8.06,46.9957]]]},"properties":{"gmdnr":1005,"name":"Hasle (LU)","kanton":"LU","max_18":22.97}},
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[8.058,47.0288],[8.0395,47.0211],[8.035,46.9994],[8.0169,46.9831],[8.0097,46.9804],[7.9745,46.9698],[7.9596,46.9736],[7.9524,46.9813],[7.9501,47.007],[7.9694,47.0296],[8.0157,47.0255],[8.0337,47.036],[8.058,47.0288]]]},"properties":{"gmdnr":1007,"name":"Romoos","kanton":"LU","max_18":22.97}},
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[8.0228,46.9765],[8.0476,46.963],[8.0454,46.9355],[8.0649,46.9346],[8.0621,46.9096],[8.0151,46.9179],[7.998,46.9179],[7.9859,46.9403],[7.9564,46.9515],[7.9596,46.9736],[7.9745,46.9698],[8.0097,46.9804],[8.0228,46.9765]]]},"properties":{"gmdnr":1008,"name":"Schüpfheim","kanton":"LU","max_18":22.97}},

output:

{"aaaa":"AaaaaaaAaaaaaaaaa","aaa":{"aaaa":"aaaa","aaaaaaaaaa":{"aaaa":"aaa:aaa:aaa:aaa:AAA:1.1:AAA11"}}, "aaaaaaaa": [
{"aaaa":"Aaaaaaa","aaaaaaaa":{"aaaa":"Aaaaaaa","aaaaaaaaaaa":[[[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111]]]},"aaaaaaaaaa":{"aaaaa":1,"aaaa":"Aaaaaa aa Aaaaa","aaaaaa":"AA","aaax11":11.11}},
{"aaaa":"Aaaaaaa","aaaaaaaa":{"aaaa":"Aaaaaaa","aaaaaaaaaaa":[[[1.1111,11.1111],[1.1111,11.1111],[1.111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111]]]},"aaaaaaaaaa":{"aaaaa":11,"aaaa":"Aaaaaaaa","aaaaaa":"AA","aaax11":11.11}},
{"aaaa":"Aaaaaaa","aaaaaaaa":{"aaaa":"Aaaaaaa","aaaaaaaaaaa":[[[1.1111,11.1111],[1.1111,11.111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111]]]},"aaaaaaaaaa":{"aaaaa":111,"aaaa":"Aaaaaa","aaaaaa":"AA","aaax11":11.11}},
{"aaaa":"Aaaaaaa","aaaaaaaa":{"aaaa":"Aaaaaaa","aaaaaaaaaaa":[[[1.1111,11.1111],[1.11,11.1111],[1.111,11.1111],[1.1111,11.1111],[1.111,11.1111],[1.1111,11.1111]]]},"aaaaaaaaaa":{"aaaaa":1111,"aaaa":"Aaaaaaaaaaaaa","aaaaaa":"AA","aaax11":11.11}},
{"aaaa":"Aaaaaaa","aaaaaaaa":{"aaaa":"AaaaaAaaaaaa","aaaaaaaaaaa":[[[[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111]]],[[[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.111,11.1111],[1.11,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.11,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111]]]]},"aaaaaaaaaa":{"aaaaa":1111,"aaaa":"Aaaaaaaaa","aaaaaa":"AA","aaax11":1}},
{"aaaa":"Aaaaaaa","aaaaaaaa":{"aaaa":"Aaaaaaa","aaaaaaaaaaa":[[[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.111],[1.111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.111],[1.111,11.1111],[1.1111,11.111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111]]]},"aaaaaaaaaa":{"aaaaa":1111,"aaaa":"Aaxaaa","aaaaaa":"AA","aaax11":11.11}},
{"aaaa":"Aaaaaaa","aaaaaaaa":{"aaaa":"Aaaaaaa","aaaaaaaaaaa":[[[1.11,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.11,11.1111],[1.111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.111],[1.1111,11.1111],[1.1111,11.1111],[1.111,11.1111],[1.11,11.1111]]]},"aaaaaaaaaa":{"aaaaa":1111,"aaaa":"Aaaaa xAAx","aaaaaa":"AA","aaax11":11.11}},
{"aaaa":"Aaaaaaa","aaaaaaaa":{"aaaa":"Aaaaaaa","aaaaaaaaaaa":[[[1.111,11.1111],[1.1111,11.1111],[1.111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.111],[1.111,11.1111]]]},"aaaaaaaaaa":{"aaaaa":1111,"aaaa":"Aaaaaa","aaaaaa":"AA","aaax11":11.11}},
{"aaaa":"Aaaaaaa","aaaaaaaa":{"aaaa":"Aaaaaaa","aaaaaaaaaaa":[[[1.1111,11.1111],[1.1111,11.111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111],[1.1111,11.1111]]]},"aaaaaaaaaa":{"aaaaa":1111,"aaaa":"Aaaxaaaaaa","aaaaaa":"AA","aaax11":11.11}},

@raphaelquati
Copy link
Author

The file
dump2.zip

@alexdima
Copy link
Member

alexdima commented Feb 27, 2018 •
edited

Thank you @raphaelquati

I can reproduce the crash, even when renaming the file to .txt. i.e. it is not JSON grammar related.

@rebornix I believe this to be a regression from the new text buffer implementation. Every 3rd, 4th run I am able to open the file without crashing and I was able to record a profile run:

Here is opening the file in stable:
stable.cpuprofile.txt
image

And here is opening the file on master (running from sources):
master.cpuprofile.txt
image

@rebornix From a quick look, it appears getLineContent is called many times on the same line. Perhaps some easy caching will get us out of the perf cliff? I also notice that it runs OOM, and I cannot understand why that would happen, for a 11MB file...

A quick heap snapshot shows 70MB heap usage in stable and 2051MB heap usage on master!

It appears that each rendered view ends up pointing to a copy of the single 11MB line, so the taller the viewport, the more memory usage... I also believe this is made worse by having the minimap turned on...

@rebornix LMK if I can help mitigate this...

@alexdima alexdima removed their assignment Feb 27, 2018
@alexdima alexdima added important Issue identified as high-priority freeze-slow-crash-leak VS Code crashing, performance, freeze and memory leak issues bug Issue identified by VS Code Team member as probable bug and removed info-needed Issue requires more information from poster labels Feb 27, 2018
@alexdima alexdima modified the milestones: Backlog, February 2018 Feb 27, 2018
rebornix added a commit that referenced this issue Feb 27, 2018
@rebornix
re #42260. use getValueInRange in SplitLine.
020afb7
rebornix added a commit that referenced this issue Feb 27, 2018
@rebornix
re ##42260. cache for last visited line content.
1d5d103
@rebornix
Copy link
Member

Thanks @alexandrudima good catch here ;)

After the change to SplitLine.getLineContent/getLineData and adding a cache for line content, I opened this file, scrolled for 10 seconds, run --status command, the memory usage now became

OS Version:       Darwin x64 17.3.0
CPUs:             Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz (8 x 2200)
Memory (System):  16.00GB (0.69GB free)

CPU %	Mem MB	   PID	Process
    0	   115	 39686	code-oss main
    0	    82	 39694	   gpu-process
    0	    66	 39696	   shared-process
    0	   262	 43504	   window (dump2.txt)
    0	   131	 43506	     extensionHost

The renderer process memory usage increases by about 70 MB (a cold start uses 197 MB usually). It can be higher sometimes if the GC is not kicked in but it should be sane.

@rebornix rebornix added the verification-needed Verification of issue is requested label Feb 27, 2018
@alexdima
Copy link
Member

alexdima commented Feb 28, 2018 •
edited

Nice! Good work @rebornix

Also thanks @raphaelquati for the file !

@jrieken jrieken added the verified Verification succeeded label Feb 28, 2018
@vscodebot vscodebot bot locked and limited conversation to collaborators Apr 14, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@rebornix rebornix

Labels
bug Issue identified by VS Code Team member as probable bug editor-textbuffer Editor text buffer freeze-slow-crash-leak VS Code crashing, performance, freeze and memory leak issues important Issue identified as high-priority verification-needed Verification of issue is requested verified Verification succeeded
Projects
None yet
Milestone
February 2018
Development

No branches or pull requests
4 participants
@raphaelquati @rebornix @jrieken @alexdima