2.5gb on small project

[edunsouza]

I have a small project of Natural Language Processing and there are 2 files with 37k lines, which contains the datasets. The problem occurs even when i'm not using those files. Not even opening the files. I have to use other editor to run "node index.js" -my main fail.

I dont know how the problem starts, but when i open the vscode editor, it goes up to 2.5gb of ram on the startup. I dont know what is happening.

[chrmarti]

/needsMoreInfo

What is the file type of the data files?

[vscodebot]

Thanks for creating this issue! We figured it's missing some basic information or in some other way doesn't follow our issue reporting guidelines. Please take the time to review these and update the issue.

Happy Coding!

[edunsouza]

/needsMoreInfo

What is the file type of the data files?

They are all javascript and json files.
Vscode takes a while to shoot up the memory. When it does, the ram usage goes up to 2.5gb and starts to freeze some features.

[chrmarti]

Does it reproduce when starting with code --disable-extensions?

[edunsouza]

Yes, it does. I found out that the process “ts/js language features” is the responsible for the 2.5gb usage. When i manually disabled the feature, it starts to work normally. But only without JS intellisense features, and thats sad :/ Em ter, 13 de nov de 2018 às 07:33, Christof Marti <notifications@github.com> escreveu:

…

Does it reproduce when starting with code --disable-extensions? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/Microsoft/vscode/issues/62922#issuecomment-438198065>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQ_i6VoFW1gVR_7fKcElraQrdlp9Rbyzks5uupH7gaJpZM4YYn5e> .

[mjbvz]

@edunsouza Can you share those source files?

[edunsouza]

@mjbvz of course I can! And I'll be happy if I can help to find out the problem.

The files are available on the repository: https://github.com/edunsouza/communicator

[mjbvz]

Tested using typescript@3.2.0-dev.20181114

Switching between the js files in dataset/ I can still easily get up to over a GB. Seems mostly to happen after switching to another js file not under dataset.

Logs:
tsserver.log

[DanielRosenwasser]

@sandersn and @ahejlsberg, any ideas on how we could do better here?

[ahejlsberg]

The issue looks to be with the dataset/afinn/afinn.js file. It contains an array literal with 30K+ elements specified as object literals:

module.exports = [
    {texto: ";)", sentimento: 1, ";)": 1},
    {texto: ";@", sentimento: -1, ";@": -1},
    {texto: ";*", sentimento: -1, ";*": -1},
    {texto: ";**", sentimento: -1, ";**": -1},
    {texto: ";~", sentimento: -1, ";~": -1},
    ...
];

The reverse mapping pattern means that each object literal has a unique type. To make matters worse, object literal normalization (#19513) causes us to expand into 30K+ unique types with 30K+ properties each (!!!). We then attempt to perform subtype reduction on the set of types in order to make a union type. This ends up generating enormous amounts of work. If I reduce down to 2500 unique types we still take close to 30 seconds to type check, so with 30K+ types we'll effectively take forever.

I'm thinking we should consider a governor on union subtype reduction. Given that is an O(n^2) operation, perhaps we limit subtype reduction to unions with 500 or less constituents.

[ahejlsberg]

Oops, meant to close the old PR, not the issue.