spring-cloud-gcp memory issue

[suztomo]

Thanks to ConstantUtf8 cache (my local modification with MAX_CACHE_ENTRIES=2000000 and bcel.maxcached.size=200), "Spring Cloud GCP Autoconfigure Module" in spring-cloud-gcp does not fail with "OutOfMemoryError: GC overhead limit exceeded".

However, even with the ConstantUtf8 cache , the enforcer rule applied on spring-cloud-gcp project fails at spring-cloud-gcp-kotlin-app-sample (the last module in spring-cloud-gcp):

JProfiler shows 606 MB is used for LinkedHashMap$Entry instances. The biggest entries are held by SymbolReferenceMaps' ImmutableSetMultimap.Builder.

Memo

Maven option to let a debugger attach the process:

export MAVEN_OPTS='-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

Maven option to heapdump upon OutOfMemoryError:

 export MAVEN_OPTS='-XX:+HeapDumpOnOutOfMemoryError'

[elharo]

That's strange. I'd expect builders to be short-lived objects. Maybe we have a memory leak somewhere?

[suztomo]

Yes something is memory-leaking because an enforcer rule single invocation at the kotlin-sample-app directory passes.

I think the Builder is behaving as expected though. It’s scanning many class files in many jar files until JVM encounters OutOfMemoryError.

[suztomo]

Linkage Checker performs two steps:

• Step 1: Build SymbolReferenceMaps
• Step 2: Verify the references in the SymbolReferenceMaps.

Failed case (OutOfMemoryException at the end of mvn verify in the root directory, at the end of Step 1 of spring-cloud-gcp-kotlin-app-sample module):

• java.util.LinkedHashMap$Entry deep size 606 MB
  • SymbolReferenceMaps$Builder deep size 270 MB
• ClassPathRepository deep size 426 MB
  • _loadedClasses: 42195
• ConstantUtf8$CACHE_HOLDER$1 deep size 157 MB

Successful case (mvn verify in spring-cloud-gcp-kotlin-sample/spring-cloud-gcp-kotlin-app-sample, paused at the end of Step 1):

• java.util.LinkedHashMap$Entry deep size 460 MB
  • SymbolReferenceMaps$Builder deep size 298 MB
• ClassPathRepository deep size 471 MB
  • _loadedClasses: 46256
• ConstantUtf8$CACHE_HOLDER$1 deep size 114 MB

[suztomo]

Using a unit test to analyze spring-cloud-gcp jar files:

  @Test
  public void testMemoryUsage() throws Exception {
    List<Path> paths = new ArrayList<>();
    for (String jar : jars) { // ~200 JAR files spring-cloud-gcp-autoconfigure
      paths.add(Paths.get(new URI("file://" + jar)).toAbsolutePath());
    }

    LinkageChecker linkageChecker = LinkageChecker.create(paths, paths);
    linkageChecker.findSymbolProblems();
  }

The following table compares three implementations with 1G/2G heap size:

Repository Implementation	Xmx1G	Xmx2G
ClassPathRepository	OutOfMemoryError: gc overhead limit exceeded	10.9 s
MemorySensitiveClassPathRepository	OutOfMemoryError: gc overhead limit exceeded	10.8 s
NoCachingClassPathRepository	134.823 s	135.7 s
FixedSizeClassPathRepository(1000)	16 s	16 s

NoCachingClassPathRepository is a custom ClassPathRepository that does not cache class at all.

[suztomo]

Performance evaluation of new FixedSizeClassPathRepository with different maximum cache size:

Maximum Cache Size	Time (smaller better)
10	40
100	18
500	16
1000	16
2000	17
5000	18
10000	22

Maximum cache size 500-1000 gives the best performance (16 seconds). It's an increase from 10.8 seconds by ClassPathRepository, but does not suffer OutOfMemoryError in 1GB heap size.