m2e Incremental Builder slowness

[stas-bob]

I have a huge workspace with 150 maven projects.
On every incremental build m2e builder resolves dependencies. I looked into it with jvisualvm.
The progress view prints "Invoking Maven Builder on Project xy".
Maven is constantly building and building and building. It comes to an end finally but starts all over again on another workspace modification like saving a space into a java file.
I came across mavenProjectCache in ProjectRegistryManager. I noticed the cache size was 5 and on every incremental build the contents where diffrent. Cache items are replaced by others during one incremental build.
I then created a plugin which simply gets reference to the MavenPlugin and then goes through the fields with reflection and eventually replaced the google cache with another one with 500 elements.
From that point all my builder problems disappeard. The workspace is very fast now
I dont even notice any maven builder in progress view now.

Please make a public API/preference setting to modify cache size.

Or will i have side effects now? I worked with it for a while and didnt notice wrong behaviour.

[mickaelistria]

150 modules is not huge, it's normal ;) FWIW m2e can scale with thousands of modules.
Can you please share your project, or just the pom files so this can be investigated? There may be some cases which are not handled optimally.
We cannot just increase the cache size too high: 1 Maven project in the cache costs ~1MB of RAM, we need to be careful of not growing it too big. Going to 500 is way too big, we could investigate something like 20 instead.
However, again the issue may just be a bug and just 1 necessary fix, so before blaming the cache, it should be investigated -thankd to a reproducer- what are the alternative solutions.

[stas-bob]

I can try to reconstruct the module structure without proprietary contents. Thats a lot of work but i want to try in a next response.

Maybe you have an idea what i could already investigate myself? What could be the pitfalls?
I dont really know how the maven builder works but it iterates through numerous projects which are sometimes not even related to each other. Why is it necessary for the builder to invoke readProjectWithDependencies on every incremental build?
I attached a screenshot of the call tree during the "invoke maven builder" progress.

I did several snapshots, they all looked like this one.

[laeubi]

@mickaelistria my Dev-Machine has 64 GB if I can spend a few hundreds of them to speed up m2e why not :-)

I think it might not be the number of modules but if they are to related/chained here, maybe an intermediate solution would be to expose the cache size in the preferences and set it to 50 on default?

[mickaelistria]

@stas-bob one possible pitfall is the usage of bom and import scope. Is this something you're using? Another one can simply be build cycles. Do you have some build cycles (ignoring versions as some other discussion suggests there can be a bug in how versions are resolved)?
@laeubi OK for a preference; and a higher default (I'd prefer 20 for the moment).

[mickaelistria]

See #161 that shows how big can a Maven project be, and why we should just not blindly increase cache size if we can find better optimizations.

[stas-bob]

Sorry i can't provide an example. The dependency tree is too big. I cant reconstruct it without proprietary stuff. Too much work.
Yes we use a lot boms and import scopes.
I definitely cant go back to normal cache. Cache size does matter.
I understand your point of view since i cant proof my case with an example. Never mind.

[vtintillier]

If you want I think the fake project attached to https://bugs.eclipse.org/bugs/show_bug.cgi?id=563742 can be used to reproduce slow project import or slow build after e.g. change to a parent pom.

[famod]

I had to use a self-patched version of m2e in larger projects (cache set to 100) as well.
Optimizing away the very need for a bigger cache is of course the best but also the more elaborate approach and will take a lot of time.
Until then, please give users an "experimental" preference. Thanks!

[HannesWell]

FYI: #850
But please keep in mind that this is internal and the System property may be removed at any time if it is not necessary any more.