Skip to content
🥚 Example code for a presentation on investigating Java off-heap memory
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.idea Added QuestionableJniLib Aug 18, 2018
QuestionableJniLib Added QuestionableJniLib Aug 18, 2018
gradle/wrapper Initial commit Aug 11, 2018
src/main Consolidate memory logging and NMT baseline taking Sep 8, 2018
.gitignore Minor gitignore tweak Aug 18, 2018
LICENSE.txt Initial commit Aug 11, 2018
Readme.md Added requirements to readme Sep 21, 2018
build.gradle Added scripts for runtime analysis and updated readme with running steps Aug 18, 2018
gradlew Initial commit Aug 11, 2018
gradlew.bat
settings.gradle Added QuestionableJniLib Aug 18, 2018

Readme.md

Java Off-Heap Memory Leak Example

This is a little example project that consumes memory in Java in several ways to help demonstrate investigating memory leaks in Java for a presentation I gave on that topic.

Definitely do not look at this code for good practices. There are intentional leaks throughout.

Presentation

https://drive.google.com/open?id=1TsjfLCuIKoE_Q3kDFtwoCkuLZ3mr2KpyPO-t-qeYDyU

Requirements

  • Linux (definitely won't work on Windows, may work one day on OSX)
  • GCC, for building the native library
  • Java, ideally Oracle JDK 1.8
  • Gnuplot, script currently hard coded for gnuplot-wx but could use any Gnuplot front end
  • Jemalloc, built with profiling enabled
    • Not sure if your jemalloc has profiling enabled? Run jemalloc-config --config and look for --enable-prof in the output. If it's not there see the Jemalloc Runtime Error Messages section of this readme

Building

Build using gradle assemble which should generate an archive containing the distributable code in build/distributions

Note: I've got something wrong with my gradle sub-project build ordering setup that I haven't figured out how to fix. But if you want to build this project yourself, run assemble twice otherwise it will be missing the native library in the distribution.

Changing QuestionableJniLib

  • Change to the JNI projects Java directory
    • cd ./QuestionableJniLib/src/main/java
  • Generate header file
    • javah -jni -d ./../c uk.co.palmr.offheapleakexample.jni.QuestionableJniLib

Running

  • Make sure you're running on Oracle JDK 1.8
    • export JAVA_HOME=/path/to/oracle/jdk/on/your/machine
  • Unzip distribution and go into it's folder
    • unzip OffHeapLeakExample-1.0-SNAPSHOT.zip
    • cd OffHeapLeakExample-1.0-SNAPSHOT
  • Run the application from the root folder
    • ./bin/OffHeapLeakExample
  • While it's paused for 20 seconds before using memory
    • Create a Native Memory Tracking baseline
    • Start logging memory to a file for plotting
    • ./bin/baseline_NMT_and_log_memory.sh
  • Wait some time, got to let it gobble up memory...
  • Look at memory use
    • ./bin/plot_memory.gnuplot
    • Looking higher than expected?
  • See what the JVM has to say about the heap
    • jmap -heap <pid>
  • Investigate the heap and maybe buffers with visualvm
    • jvisualvm
  • Check out diff between the Native Memory Tracking baseline taken and now:
    • Summary diff (usually good enough): jcmd <pid> VM.native_memory summary.diff
    • Detail diff (usually too much detail): jcmd <pid> VM.native_memory detail.diff
  • Nothing particularly obvious there to explain endless memory use? Can kill the application now
  • Generate out the jemalloc profile diagrams
    • ./bin/jeprof_diagrams.sh
  • Hopefully by now you've spotted where the memory is going!

What do the scripts do?

./bin/OffHeapLeakExample

This script is generated by Gradle and in general just runs java with a classpath including the project jar and specifying the main class to run.

The gradle script does add some extra bits to this command which you might want to take note of:

./bin/baseline_NMT_and_log_memory.sh

This script find the pid for a process with the main class in its command string.

With that it takes a Native Memory Tracking baseline using the following command: jcmd ${PID} VM.native_memory baseline

It then starts to loop every half-second reading the Resident Set Size value from /proc/${PID}/status and writing it to a file, ./memory.log

./bin/plot_memory.gnuplot

Not a shell script but an executable file that runs gnuplot (requires gnuplot-wx installed) to plot the resident set use over time.

./bin/jeprof_diagrams.sh

Calls the jeprof command with input of all the jeprof.*.heap files and just the latest one to generate graphviz files.

The graphviz files are then converted to image (PNG) files for easy viewing.

Jemalloc Runtime Error Messages

If you see lines in the stdout from the application like:

<jemalloc>: Invalid conf pair: prof:true

Then the jemalloc on your machine was not compiled with profiling enabled.

The following is a bash script to do this if on a distro using dnf (i.e. Fedora, CentOS)

#!/bin/bash

# download source rpm to this folder
sudo dnf download --source jemalloc

# install source folder, will go to user home/rpmbuilds
rpm -ivh jemalloc*.src.rpm

# go to specs
cd ~/rpmbuild/SPECS

# get deps
sudo dnf builddep jemalloc.spec

# add --enable-prof to configure argument
sed -i '/^%configure/ s/$/ --enable-prof/' ~/rpmbuild/SPECS/jemalloc.spec

# build package
rpmbuild -bb jemalloc.spec

# install resultant rpm
sudo rpm -ivh `ls -1B ~/rpmbuild/RPMS/x86_64/jemalloc-*.rpm | grep -v 'dev\|debug'`
You can’t perform that action at this time.