High memory consumption of LSP process "PropertyLauncher" can lead to excessive garbage collection and subsequent slow down

[hbrands]

Eclipse 201812 on Windows 10 Prof, , STS 4.1.2 (and STS 3.9.7 Add on):

With our closed-source Eclipse projects and workspace we noticed that the background thread "PropertiesLauncher" quickly fills up its heap space. In particular, with the attached VisualGC it could be noticed that the old generation in the heap was full and a lot of full garbage collection cycles happened, causing CPU load:

[hbrands]

Leak suspects from MAT:

leak_suspects.pdf

[hbrands]

@martinlippert I'm not sure what this background process does. Does it analyse spring beans only or the whole project or the whole workspace?
Our workspace contains 47 Eclipse projects summing up to a total of 1,2 million LOC.
Of course, the two Spring Boot 1.5.x server projects depend on only a subset of these projects.

[kdvolder]

@hbrands asked:

Does it analyse spring beans only or the whole project or the whole workspace?

The workspace. It does try to avoid analyzing projects that don't look like they are using spring (decided by looking at the dependencies of the project).

[hbrands]

According to SpringProjectUtil.java a project is considered, if spring-core* and/or spring-boot* is on it's classpath?
Hmm, I'll check my workspace how many projects satisfy this condition tomorrow. (That could be a lot I assume)

[hbrands]

I can confirm that all projects in the workspace directly or indirectly reference spring-core in their classpath (by all using a base project that's dependent on spring).
So I guess the background process tries to analyse all java files in the workspace, which are a lot.

[martinlippert]

@hbrands very interesting numbers, even though I would have hoped that the language servers AST analysis is not that memory intensive and it seems to be in your case. What version of STS4 do you use? And what is max heap setting for your IDE itself and how much of that does it consume, if you do a full rebuild, for example? Would be extremely interesting to see that.

The leak suspects from the MAT analysis look not extremely suspicious, seems like the regular scanning of the ASTs for the projects (when the language server kicks off) and JDT internals showing up there. The main question will be if JDT is consuming so much memory for a good reason (and does basically the same than in the regular IDE) or do we do something wrong while using JDT or not freeing up that memory early enough for some reason? Sounds like we need to investigate further.

Lets first look at the other numbers (the one I asked above) and then decide how to move on here.

[hbrands]

@martinlippert I'm using Eclipse for Java Enterprise Developers with STS 4.1.2 and STS 3.9.7 Add-On installed.
Version: 2018-12 (4.10.0)
Build id: 20181214-0600

I'm using Eclipse with the following memory settings:
-XX:+UseG1GC
-XX:+UseStringDeduplication
-Xmn128m
-Xms1024m
-Xmx3G

I've attached a Visual GC screenshot for full workspace rebuild.

[hbrands]

Note that other plugins like m2e, Checkstyle and PMD were also running on the full rebuild.

Java-Version used is Oracle JDK 1.8.0_191.

[martinlippert]

@hbrands when you go to https://dist.springsource.com/snapshot/STS4/nightly-distributions.html, you can download a nightly CI build of STS4 (or use the update site mentioned at the top to update an existing install to the latest nightly build).

This new build contains two things:

• an option to set specific VM arguments for the language server process
• additional performance-related logging for the symbol creation

To set specific VM arguments for the language server, please add a specific system property to your eclipse.ini file (or SpringToolSuite4.ini, depending on your install): -Dboot.ls.custom.vmargs

For example, you could add a line like this:

-Dboot.ls.custom.vmargs=-Xmx3G to enable 3G of max heap space for the language server JVM. If you want to set more than one argument, you can add them all at once, for example:

-Dboot.ls.custom.vmargs=-Xmx3048m,-Dsomething=super,-Xdebug,-Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=n

The individual arguments are comma separated and the value of this property is split into the individual pieces using ,- as the indicator for different arguments.

This should allow you to experiment with larger max heap spaces for the language server process. I very much hope that this will help a bit.

The second thing that I added creates additional performance-related log output from the language server. To enable the logging, you can enable the checkbox in the preferences for Spring Language Servers. It will open a console view section for the specific language server and show the log output. Usually you don't need that (and it is somewhat annoying to get the log output streamed to the console view), but for debugging purposes it can be quite helpful. So please enable that (at least once) and send me the output. If you don't want to paste the output publicly, let me know and we can share privately.

The lines that I am interested in are:
scan java files for symbols for project...

[martinlippert]

@hbrands The latest builds include a few additional options and improvements and it would be awesome if you could give them a try, so that we can learn about the effects.

The first option that you should set for the language server (via the mechanism described above) is:

-Dboot.ls.symbols.caching.enabled=false

This disables the newly introduced cache for symbols. Since the cache avoids re-parsing the whole workspace every time you started up the language server for the first time, it would not allow us to analyze the behavior for scanning the whole workspace anymore. Therefore please disable that when running from the latest builds.

The second option is:

-Dboot.ls.symbols.ignoreMethodBodies=true

This skips certain parts of the parsing while scanning the whole workspace. This is not particularly useful for real world usage (since we need to NOT ignore method bodies for certain symbols), I would be very interested to see how that affects the memory consumption of the language server process.

So it would be great if you could give the latest CI build a try, set both options in the ini file via:

-Dboot.ls.custom.vmargs=-Dboot.ls.symbols.ignoreMethodBodies=true,-Dboot.ls.symbols.caching.enabled=false

and watch the memory consumption. For this test, please do not increase the max heap space yet. Once we have that data, you could try to increate the heap space (if still necessary), and remove the option to disable the caching. That should help a lot going forward.

Looking forward to hearing from you with the results.

[martinlippert]

As soon as build B1586 appears on this nightly download page, it should be ready for you:
https://dist.springsource.com/snapshot/STS4/nightly-distributions.html

[hbrands]

@martinlippert Attached are the log files for four runs:

1. ) boot_debug.txt :
   -Dboot.ls.symbols.caching.enabled=false

2.) boot_debug_ignoremethodbodies :
-Dboot.ls.symbols.ignoreMethodBodies=true,-Dboot.ls.symbols.caching.enabled=false

3.) boot_debug_2g :
-Xmx2G,-Dboot.ls.symbols.caching.enabled=false

4.) boot_debug_3g :
-Xmx3G,-Dboot.ls.symbols.caching.enabled=false

Notes:
The first run 1.) had at least one OutOfMemoryError parsing the biggest project edrewe-ui.
Alos there was an exception on the second biggest project edrewe-common.
The run with ignore method bodies completed without exception, max heap of 1G was enough, but parsing edrewe-ui took 45 seconds.
The third run was with 2G max heap (method bodies enabled), but that run into an OutOfMemoryError, too.
The fourth run was with 3G max heap (method bodies enabled) which went ok without exception.

I noted in the logs "o.s.i.v.c.boot.app.cli.SpringBootApp - check for spring jmx beans ..." with four retries, not sure if this is important. I triggered the parsing by opening a java file, no spring app was started.

Spring Boot projects are edrewe-serverapp and edrewe-transmission-server. A traditional spring app is in edrewe-webapp. But note, that edrewe-ui is not used in these servers so that project parsing is not really needed?!
Hope this helps so far.

sts4_debug.zip

[martinlippert]

That is awesome, thanks a lot for spending the time and running these tests, much much appreciated. Can you run one more check? I would love to get my hands on a head dump when the JVM is running out of memory. Can you add something like this to the VM args for the boot language server?

-XX:+HeapDumpOnOutOfMemoryError

You could also specify a path where to store the heap dump by adding -XX:HeapDumpPath=/opt/tmp/heapdump.bin

and send me that head dump somehow? That would be excellent... :-)

[martinlippert]

Oh, and I would be interested in the heap dump for your third config only (not all of them): boot_debug_2g

[hbrands]

@martinlippert So how can I share this 2 GB heap dump with you? E-Mail is no option. Any FTP address available?

[martinlippert]

Dropbox?

[hbrands]

I don't have a dropbox account. Can I upload the file to your dropbox nevertheless?
(https://www.dropbox.com/de/help/files-folders/received-file-request)

[martinlippert]

Yepp, can do that, just need your email... :-)

[martinlippert]

The heap dump is quite interesting, thanks a lot for that. I think for the next release (4.2.0), we have these two things implemented that should improve the situation for you a lot:

• please use the option to "ignore method bodies" for now - this is only useful when you define function-based web routes for Spring Webflux. If you are not using that yet, you can easily add that option to your args without loosing anything, but allowing the language server process to stay within the 1G heap boundaries.

• the symbol caching will avoid the re-creation of all the symbols on every startup in case your projects definitions and source files do not change. Feel free to give it a try using the latest CI builds (a committed a few fixes recently).

With this, I would close this issue and mark it as resolved for the 4.2.0 release. Nevertheless, I will not stop to work on the memory consumption and overall performance of this, but in separate, more specific issues. Also feel free to create new issues for whatever you find, including performance and memory issues. Since you have a fairly large workspace, those experiences are super valuable for us.

[martinlippert]

According to SpringProjectUtil.java a project is considered, if spring-core* and/or spring-boot* is on it's classpath?
Hmm, I'll check my workspace how many projects satisfy this condition tomorrow. (That could be a lot I assume)

Any news on this? Would be interesting to see why edrewe-ui (the project that you mentioned that isn't really a Spring app) is considered an interesting project by the language server.

[hbrands]

Did you see my comment
#212 (comment) ?

[hbrands]

@martinlippert What do you think is the limiting factor for spikes in heap usage, the size of the whole workspace or the size of individual projects in the workspace? Do you think splitting up big projects into smaller ones will help here?
Did you find out potential for optimization from the provided heapdump or is this issue located in Eclipse (not STS) parsing code?
Perhaps, the strategy to parse all projects which have spring jars in their classpath is suboptimal. Only part of the projects is really used in the spring server projects. (But then there is an additional spring client project, which consumes the ui projects, sigh!). Perhaps it makes sense to let the user select the root spring projects which should be analysed and then in turn only look at their dependencies (of which there might be projects open in the workspace). Just some thoughts...

[hbrands]

Thanks for looking into the issue and providing help and customization options!

[martinlippert]

Did you see my comment
#212 (comment) ?

Oh, yes, right, so that answers the question... :-)

[martinlippert]

@martinlippert What do you think is the limiting factor for spikes in heap usage, the size of the whole workspace or the size of individual projects in the workspace? Do you think splitting up big projects into smaller ones will help here?

Initially the language server parses all the files in a project as one big batch. This has the advantage that a lot of information (especially the lookup and type resolution information) is cached and shared when parsing each individual file. Re-creating that state for every single file would be super expensive. On the other side, the more files you have in your project, the more is parsed in this batch mode and the more memory this batch parsing allocates. This is the reason for the spike in the heap dump. So splitting this up into smaller projects would help indeed, since you would end up running multiple batch parses (one for each smaller project), resulting in less heap usage per batch (probably, somewhat hard to predict). GC would be able to free up the space from the previous batch parsing before the next one runs.

But I don't think the tooling should force you to a specific project structure. It should really be able to deal with larger projects and larger workspaces. Therefore I think making progress in this "ignore method bodies" area (as described above) makes a lot of sense anyway and should reduce the spike in heap usage.

Did you find out potential for optimization from the provided heapdump or is this issue located in Eclipse (not STS) parsing code?

Most of the heap is indeed used by the JDT parser internally, so nothing that I can change right away, but I will contact the JDT folks about it. Maybe we can find additional ways to reduce the memory consumption or to use the parser differently to reduce the memory consumption. Maybe running the parsing in more fine-grained batch sizes would be an improvement (would trade less memory for slightly less performance though).

Perhaps, the strategy to parse all projects which have spring jars in their classpath is suboptimal. Only part of the projects is really used in the spring server projects. (But then there is an additional spring client project, which consumes the ui projects, sigh!). Perhaps it makes sense to let the user select the root spring projects which should be analysed and then in turn only look at their dependencies (of which there might be projects open in the workspace). Just some thoughts...

Yeah, I agree, if we can find a way to reduce the scope of the parsing to fewer projects, that would help, too, of course. Not sure yet what the best way is here. Maybe we should open a separate issue for that and share ideas/thoughts over there. WDYT?

[hbrands]

👍

With your applied changes so far we should be able rollout STS 4.2.0.
Separate issues for further enhancements make sense.

[martinlippert]

Sounds good. I also noticed that you use the STS3 Add-On pack. What do you need the STS3 Add-On pack for? I am looking for feedback about what is missing in STS4. So what are you missing in STS4 that makes you install the STS3 Add-On ?

[hbrands]

@martinlippert From
https://github.com/spring-projects/sts4/wiki/STS3-Features#sts3-features-that-are-not-present-in-spring-tools-4
I assumed that XML config support in STS 4 is not (yet) on par with STS 3?
We still use XML configuration (mainly for the client application).

[martinlippert]

The question for me is: which parts of the XML tooling of STS3 do you use and would like to continue to have? My goal is to allow you to do everything you need in STS4 only some day in the future and we will not be able to port everything from STS3 over to STS4. Therefore learning about your requirements/priorities would help to prioritize the work... :-)

[hbrands]

It's basically code completion support in XML files and refactoring support for XML files. I'm not sure what other parts you are refering to? Anyway, we're in the process of migrating to Java config, so in a few months XML support isn't that important for us anymore.

[martinlippert]

@hbrands I implemented an additional optimization for the symbol indexing infrastructure, so that you don't need to configure your language server with -Dboot.ls.symbols.ignoreMethodBodies=true anymore. The language server does the symbol parsing in two phases now: the first one without the method bodies, and a second one including the method bodies, but only for those files where we actually need the method bodies. This should reduce the heap consumption of the language server in your case automatically down to the same result than with that setting added.

If you want to give it a try, feel free to download a nightly build from here: https://dist.springsource.com/snapshot/STS4/nightly-distributions.html

In case you give it a try, it would be super interesting to see the log output (includes a bit of timing information for the parsing) and a screenshot of the memory consumption over time (in case you have time for this). Thanks!!!