Areas to continue improving memory calculation

[youngm]

With all the improvements of JBP 4 I'm still running into applications that grow outside their container.

I've been studying more about java native memory allocation and this issue contains some further areas that the JBP might consider accounting memory for.

I've been doing some tests with some of our applications with -XX:NativeMemoryTracking=summary turned on. (http://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html)

Here is the output of one of my applications:

Total: reserved=3302318KB, committed=1757634KB
-                 Java Heap (reserved=2603008KB, committed=1272832KB)
                            (mmap: reserved=2603008KB, committed=1272832KB)

-                     Class (reserved=167117KB, committed=155509KB)
                            (classes #24663)
                            (malloc=5749KB #56366)
                            (mmap: reserved=161368KB, committed=149760KB)

-                    Thread (reserved=42783KB, committed=42783KB)
                            (thread #85)
                            (stack: reserved=41808KB, committed=41808KB)
                            (malloc=268KB #432)
                            (arena=707KB #169)

-                      Code (reserved=270096KB, committed=121636KB)
                            (malloc=20496KB #25161)
                            (mmap: reserved=249600KB, committed=101140KB)

-                        GC (reserved=155708KB, committed=106352KB)
                            (malloc=26344KB #62265)
                            (mmap: reserved=129364KB, committed=80008KB)

-                  Compiler (reserved=249KB, committed=249KB)
                            (malloc=118KB #655)
                            (arena=131KB #3)

-                  Internal (reserved=19658KB, committed=19658KB)
                            (malloc=19626KB #61247)
                            (mmap: reserved=32KB, committed=32KB)

-                    Symbol (reserved=30106KB, committed=30106KB)
                            (malloc=25410KB #283936)
                            (arena=4696KB #1)

-    Native Memory Tracking (reserved=8308KB, committed=8308KB)
                            (malloc=518KB #8148)
                            (tracking overhead=7790KB)

-               Arena Chunk (reserved=201KB, committed=201KB)
                            (malloc=201KB)

-                   Unknown (reserved=5084KB, committed=0KB)
                            (mmap: reserved=5084KB, committed=0KB)

Reserving Thread space memory off of Xss doesn't account for system thread and their different stack size

Problem

This particular container is running 84 system and application threads. I'm using -Xss256K yet Java is using 41808KB of memory for stack allocation. 30 of my 84 threads are System threads which -Xss doesn't apply to. In addition many of those threads are GC threads which will vary based on the GC type used.

Solution?

Allow for a configurable System Thread static base value that can be used to provide a best guess buffer considering GC type used and stack size of native threads. Then add onto that the configured thread value multiplied by Xss value. In the case of G1GC This value would probably be about 30MB

GC Size Not accounted for at all

Problem

The application above is using the G1 GC which is using a wopping 106352KB of memory. To my knowledge this memory area isn't accounted for in JBP4 at all. Running some tests it appears that the GC area grows and shrinks with the Heap size specified.

Solution

The JBP should account for GC memory size by using a factor of total heap size allocated. This factor should be configurable because it will change based on the GC used. SerialGC uses much less memory. Some testing needs to take place to determine a proper factor for the default GC.

In this sample the Heap is set to -Xmx2601674K. So this application could benefit from a GC Memory factor of ~5%.

Total doesn't match linux process memory

Problem

Java Native memory calculator says java is using 1757634KB But linux says the process is using 2.036g.

Solution?

Worth additional study to understand how to account for this.
Turns out this is probably a native memory leak in the application.

Misc items to investigate further

This report shows several other items of native memory not accounted for that might be worth investigating. Most notably Internal and Symbol. More testing and analysis could be done here to help determine if the JBP can do anything smart with these.

Summary

I think there is still some work that can be done to better account for Java memory in the JBP. The Native Memory Tracking tools is a great help in discovering this data.

[youngm]

So, after doing more research it appears that "Total doesn't match linux process memory" is most likely a native memory leak for this application. That's good because that bunch of unaccounted for memory was scary.

[nebhale]

@glyn Want to take a look at this and come up with a concrete proposal for changes?

[youngm]

@glyn @nebhale Here is an interesting idea. In the issue above I listed several native memory areas discovered using Native Memory Tracking that we currently aren't accounting for. Instead of attempting to devise methods to guess what we think these values should be in most situations, we should simply start a JVM with the same opts as the application and determine overhead based on the output of PrintNMTStatistics. I've done a few tests and it appears most of this jvm base native memory is committed at startup time and only appears to grow a little from that point. If we had the Memory calculator run something like:

java -XX:NativeMemoryTracking=summary -XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics -Xms512m -Xmx512m -XX:+UseSerialGC -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

Native Memory Tracking:

Total: reserved=1845998KB, committed=546646KB
-                 Java Heap (reserved=524288KB, committed=524288KB)
                            (mmap: reserved=524288KB, committed=524288KB)

-                     Class (reserved=1056856KB, committed=4568KB)
                            (classes #338)
                            (malloc=88KB #100)
                            (mmap: reserved=1056768KB, committed=4480KB)

-                    Thread (reserved=11357KB, committed=11357KB)
                            (thread #11)
                            (stack: reserved=11308KB, committed=11308KB)
                            (malloc=36KB #57)
                            (arena=13KB #22)

-                      Code (reserved=249629KB, committed=2565KB)
                            (malloc=29KB #279)
                            (mmap: reserved=249600KB, committed=2536KB)

-                        GC (reserved=1719KB, committed=1719KB)
                            (malloc=7KB #79)
                            (mmap: reserved=1712KB, committed=1712KB)

-                  Compiler (reserved=132KB, committed=132KB)
                            (malloc=1KB #20)
                            (arena=131KB #3)

-                  Internal (reserved=195KB, committed=195KB)
                            (malloc=163KB #1141)
                            (mmap: reserved=32KB, committed=32KB)

-                    Symbol (reserved=1292KB, committed=1292KB)
                            (malloc=868KB #57)
                            (arena=424KB #1)

-    Native Memory Tracking (reserved=31KB, committed=31KB)
                            (malloc=2KB #28)
                            (tracking overhead=28KB)

-               Arena Chunk (reserved=499KB, committed=499KB)
                            (malloc=499KB)

Then add up the committed memory of all the areas we want to get a value for. From my testing it appears this provides a great initial value for System Threads size and GC size at the very least. Symbol and Internal not so much.

I would propose that we use the entire application's opts untouched, with the exception of Xms which should be the same as Xmx, allowing any javaagents to also initialize and capture data from them as well. Granted, we may not have a final -Xmx value at this point but a decent candidate for this value should work just fine.

Thoughts?

[youngm]

A possible downside of this approach is it wouldn't work with java 7.

[youngm]

Closing this issue to break it up into the 2 main memory findings I think this issue exposed:

System thread size:
#511

GC memory usage:
#512

I may eventually create issues for Symbol and Internal memory as I continue to study the native memory usage of my org's applications.