Full GC on Jruby 9k happens less frequently than on Jruby 1.7

[AlexEzzeddine]

Environment Information

2 environments running same app:
ubuntu 14.04, java 7, jruby 1.7
ubuntu 18.04, java 8, jruby 9.2.8.0

JVM flags:

-XX:+HeapDumpOnOutOfMemoryError
-XX:InitialHeapSize=8053063680
-XX:+ManagementServer
-XX:MaxHeapSize=8053063680
-XX:+PrintGC
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:ThreadStackSize=2048
-XX:+UseCompressedClassPointers
-XX:+UseCompressedOops
-XX:+UseParallelGC

Other relevant info you may wish to add:
Rails 3.2, trinidad 1.4.4 (120 threads)
Issue only happens on production environment under heavy request load

Expected Behavior

App should not be slow on 9k

Actual Behavior

App is slow on 9k 2 hours after startup (12 pm on the screenshot is when it was started and it become slow at around 2:15pm).
Looks like the slowness is happening because app doesn't have enough memory for new objects

9k memory usage and gc.log for this period:

9k_gc.log

1.7 memory usage:

17_gc.log

[AlexEzzeddine]

Also I don't understand why I didn't get OOM at 14:15 when the spike on the graph happened. Instead all requests just started being slow (under a second -> more than 30 seconds)

[AlexEzzeddine]

@headius turns out the cause of the issue was not related to GC.
We were not specifying code cache settings and jvm was using default code cache size (50 mb on java 7 and 250mb on java8)
When I increased code cache size to 1gb (-J-XX:ReservedCodeCacheSize=1024m), performance issues went away, but now there is another issue.
During the warmup phase, code cache usage rapidly grows to around 600+ MBs and then instead of stabilizing it slowly grows over time. Please check attached screenshot.

Also when I check compilations logs during the second phase, most of the compilations are LambdaForm methods.
Is this normal? Any way to limit code cache usage growth?

[AlexEzzeddine]

Also I just found this guide about code cache in your wiki: https://github.com/jruby/jruby/wiki/PerformanceTuning#check-the-openjdksunjdkoraclejdk-code-cache-size
I don't know why, but I opened Performance Guide page multiple times and I never noticed the Code Cache part :D

[headius]

@AlexEzzeddine That's a very interesting discoery!

The LambdaForm compiles are part of the invokedynamic subsystem, and I'm not sure how much we can do to fix that. I would expect it to plateau, however, and ideally once things have jit compiled with optimized calls some of this cache should go away. We need to do some investigation into why it seems like the code cache grows forever.