Loading large files causes JavaScript heap out of memory

[we-wake-act21]

current my DSL is using two separate Langium based DSL (expression.langium & model.langium) to build ASTNode.

Model.langium refers to expression.langium as
Statement: models+=model*;
model infers Statement: exp=OxElement '=>' children+=Statement ;

where as the expression.langium consists of language based on ox-language.

when i am trying to load multiple large files (in my scenario 20) with more than 10000 LOC. The application crashes with error as JavaScript heap out of memory.
after making the following changes i was able to load the langium documents

model infers Statement: exp=STRING '=>'children+=Statement ;

So, in my analysis the object size with multiple statements and expression the ASTNode is becoming quite complex but after removing the dependency of Expression.langium the problem does not persists with same LOC and files.

So is there any way to approach this problem? or is there any way to load specific number of files at a time ?

currently, I am using langium as a DSL for Theia and GLSP based Application,
Versions:

1. Langium - 2.1.3
2. theia - 1.43.1

[msujew]

Hey @we-wake-act21,

It's pretty interesting that you hit the heap limit that early. Are you running on a memory restricted machine? We have a few projects where large workspaces (1 mil+ LoC) are common and we get to roughly 3GB of used memory on those. A repro example to see what's exactly the issue would be appreciated.

So is there any way to approach this problem? or is there any way to load specific number of files at a time ?

You can override the WorkspaceManager to decide on a file-by-file basis whether to load it, or not. Note that loading a file will automatically happen once it has been opened in the editor. Note that Langium cannot resolve cross references to unloaded files.

Aside from that, it probably makes sense for you to perform a bit of profiling on the language server process. Just to see which parts of Langium/your language are taking such a large amount of memory.

[we-wake-act21]

so this the profiler snapshot

[we-wake-act21]

hi @msujew,

I've analyzed and observed that during the parsing stage, there's a notable increase in memory utilization. This is evident in the screenshots, where memory usage rises from 537MB to 1.32GB after parsing workspace documents. The parsed output encompasses various objects, as depicted in the accompanying screenshot.

Is there a method to load documents in batches, whereby certain objects can be loaded after the validation of the initial batch? Additionally, I've noticed that within ChevrotainLexer, the positionTracking argument can be set to either onlyOffset or onlyStart, which parses the document from the beginning. Can this behavior be overridden to parse up to a certain depth and subsequently control whether to partially or fully load the document?

[msujew]

Hey @we-wake-act21,

Is there a method to load documents in batches, whereby certain objects can be loaded after the validation of the initial batch?

There is none yet. We thought about adding such a mechanism but didn't need it ourselves yet, so we didn't put in any work for that.

the positionTracking argument can be set to either onlyOffset or onlyStart, which parses the document from the beginning.

The position tracking only affects what kind of positional data will be tracked during the lexing process. I.e. token offset, token offset + token end offset or line + column tracking. Note that LSP features will stop working if you something different than 'full'.

Aside from that, the memory consumption is of course affected by the shape of the grammar. Though a consumption of 1.3GB sounds reasonable, depending on the size of the workspace of course.

[we-wake-act21]

@msujew ,
any suggestions or material i can refer on developing document load in batches? and is it possible to off-load certain data within the $cstNode object and only keep the actual modelNode

[msujew]

any suggestions or material i can refer on developing document load in batches? and is it possible to off-load certain data within the $cstNode object and only keep the actual modelNode

For the batching: I know that Xtext has been doing this in their version of our DocumentBuilder service. You should take a look there.

For the CST model unloading: I believe Xtext finally has now (after 16 years) added this recently: eclipse/xtext#2968. Since our CST works fairly similar, I think that it should be feasible to implement something similar in Langium as well.