indexing runs out of memory for large projects

[martinlippert]

The indexing infrastructure is running out of memory when indexing projects with a large number of source files (as reported in #1212).

We need to improve the implementation to reduce the overall memory consumption, especially to decouple the memory consumption from the size of the project or the number of projects being parsed.

Step 1: we need to chunk the set of source files into well-defined smaller chunks in order to allow the garbage collection to free up while indexing.

Step 2: we need to cleanup the lookup environment of the parser after each parsing attempt in order to avoid leaking memory or keeping zip files open.

[martinlippert]

Inviting @licam to this issue in order to provide additional feedback and test early builds, once available.

[martinlippert]

@licam The latest pre-release builds for VSCode should already contain a few early optimizations. Would be interesting to hear whether that runs any better in your environment and with your large projects. You can switch to the pre-release in VSCode directly when you click on the Spring Boot Tools entry in the list extensions, and then switch to pre-release.

[martinlippert]

Here are some early rough results, measuring the progress here (using my sample project):

Version 1.53.0 is able to:

• parse projects with 6.500 source code files
• generate 100.000 symbols

Version 1.54.0 is able to:

• parse projects with 65.000 source code files
• generate 1.000.000 symbols

Both measurements used the default max heap setting of 512m for the language server process.
This is a 10x improvement, so quite a good step forward here, I think.

The exact numbers will vary quite a bit, depending on the size of the individual source code files and the number of symbols generated for the concrete project, of course.

If you have larger projects that this, you have to increase the heap space for the language server.

[licam]

@martinlippert Sounds promising. We will test and adapt the new version once it will be released. Thank you!