ISPN-8478 Reduce build memory usage

[danberindei]

https://issues.jboss.org/browse/ISPN-8478

Since I first opened the PR, the ANSI escape codes in test suite output which were causing the OSGi tests OOME have been reverted.

So I've changed the target of the PR to get the build running on the new agents that only have 4GB of RAM.

[slaskawi]

hmmm it looks a bit weird - we set Xmx in integrationtests/osgi/pom.xml and here we use 1GB. Could you please tell me why? And shouldn't this be parametrized (e.g. taken from pom)?

[danberindei]

There are 3 JVMs running:

1. The Maven JVM - uses heap size from $MAVEN_OPTS or 1/4 or RAM.
2. The Surefire fork - uses heap size from forkJvmArgs (2G in parent/pom.xml), $MAVEN_FORK_OPTS, or 1/4 of RAM if $MAVEN_FORK_OPTS is non-empty and doesn't have -Xmx.
3. The Karaf process - 1/4 of RAM.

In this PR, I'm experimenting with the heap size of all 3 JVMs. I could use the same heap size for VMs 2 and 3, but I don't think it's ok to give that much memory to the Surefire fork when all it does is start the Karaf process.

[gustavocoding]

@danberindei In my env it appears the the Maven Launcher process itself crashed; the surefire fork exited normally, but the build exited with
[ERROR] Java heap space -> [Help 1].

I got no dump since only the fork has the dump flags, I am re-running adding them to MAVEN_OPTS.

I got 32GB of ram, in theory the lancher should have about 8Gb for itself...

[gustavocoding]

@danberindei In my env the maven launcher always crashes in the JCache TCK runner module, it allocated > 6Gb ram

[gustavocoding]

After looking at the dump (of the maven launcher), there's a Thread object with lots of crap in with:

Here's where it explodes:

main <--- OutOfMemoryError happened in this thread State: RUNNABLE tid: 1
  java.lang.OutOfMemoryError.<init>() OutOfMemoryError.java:48
  java.util.Arrays.copyOf(char[], int) Arrays.java:3332
  java.lang.AbstractStringBuilder.ensureCapacityInternal(int) AbstractStringBuilder.java:124
  java.lang.AbstractStringBuilder.append(String) AbstractStringBuilder.java:448
  java.lang.StringBuilder.append(String) StringBuilder.java:136
   org.apache.maven.plugin.surefire.booterclient.output.MultipleFailureException.getLocalizedMessage() MultipleFailureException.java:52
  org.apache.maven.plugin.surefire.booterclient.ForkStarter$CloseableCloser.run() ForkStarter.java:200
  org.apache.maven.surefire.shade.org.apache.maven.shared.utils.cli.CommandLineUtils$1.call() CommandLineUtils.java:282
  org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(Object, KeyValueSource, ForkClient, SurefireProperties, int, AbstractForkInputStream, boolean) ForkStarter.java:626
  org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(Object, KeyValueSource, ForkClient, SurefireProperties, AbstractForkInputStream, boolean) ForkStarter.java:533
  org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(SurefireProperties, Map) ForkStarter.java:279

@danberindei You mentioned we are messing up with surefire fork I/O, should we stop doing that maybe? :)

[danberindei]

@gustavonalle please revert Tristan's colourizing patch, #5520, and try again.

[gustavocoding]

Reverting #5520 fixes the leak

[gustavocoding]

@danberindei Could you rebase?

[danberindei]

Updated!

[gustavocoding]

Was hoping for an improvement in the build stability, but the build crashed the same :)

[gustavocoding]

Re-triggering it

[danberindei]

@gustavonalle The timeout in as-lucene-directory is unrelated, it looks like an issue with the H2 driver and/or WildFly.

Also the last build appears successful, only the commit status update failed for some reason.

[gustavocoding]

Merged, thanks @danberindei !

[danberindei]

@gustavonalle Looks like 700MB of heap is not always enough for the infinispan-wildfly-modules build after all:

http://ci.infinispan.org/job/Infinispan/job/PR-5580/1/console

I pushed a commit to make the heap 800MB directly to master.