Memory leak while scanning annotations

[ljorquera]

While scanning classes for annotations, jars are opened as zip files but they are not closed properly which caused the memory leak.

memory_leak_example.zip
We were investigating a Jetty application that used much more memory than it was supposed to do (near a Gb more than the max heap size plus the max metaspace size). We found out that the extra memory was Native memory, being allocated using malloc via some library (we don't have native code in our app). Then we used jemalloc ( http://www.canonware.com/jemalloc/) to find out which class was doing this allocation and we found out that it was java.util.zip.Inflater. Via jstack we were able to find calls to the library and the main caller was Jetty, while in the process of looking for annotations. We disable annotations for the application and the memory usage went back to normal.

We looked at the Jetty code for the version we are using (9.2) and definitly the bug is there ( scanClass(handlers, jar, clazz.getInputStream()) in line 956 of AnnotationParser.java, that input stream is never closed). What is very strange is that while code in version 9.3 seems to be fixed (it uses try-with-resources to close the input streams) the leak seems to still be there: we did the same tests than in 9.2, jemalloc reports the missing memory and the application with annotations uses a lot more memory than without.

We attach a project that shows the same behaviour than our application. If started with annotations it mallocs much more memory and it can be traced to annotations search.

[joakime]

This would be a duplicate of Issue #231

[ljorquera]

We don't need to undeploy the app, just deploying it causes the issue.

[joakime]

The problem exists regardless of undeploy/redeploy.
And the Inflater memory leak exists with straight Inflater usage as well (doesn't have to be part of the Jar file process)

The number of bugs we have filed against the JVM Inflater continues to grow (See #293, #295, and #300)

[janbartel]

I'm still talking to Oracle about this jvm bug. Interestingly, it doesn't seem to be reproduceable with jdk9. Are you able to test with jdk9 and report results back here?

[ljorquera]

I just tried with JDK9 and the bug is still there.
This is the exact version I used:
lorenzo@lorenzo-VirtualBox:/tmp/a/create-distrib-5.0.0.276-SNAPSHOT$ java -version
java version "9-ea"
Java(TM) SE Runtime Environment (build 9-ea+119)
Java HotSpot(TM) 64-Bit Server VM (build 9-ea+119, mixed mode)

[janbartel]

Here's a link to the java bugs database issue: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8156014

I'd like to be able to comment on it, but I can't seem to find a link to allow me to do that.

The comments I'd like to add are:

• the problem is still reproduceable as of jdk8u112
• the problem seems to be fixed in jdk9: I tested jdk9ea+149 and couldn't reproduce

I've tried some workarounds for jdk8: it seems the ServiceLoader impl uses the jarurlconnection caching, so it may be of some benefit to try to disable url caching (although not sure of the effects on performance).

Try creating an xml file (eg called "caches.xml") with the following contents:

<Configure id="Server" class="org.eclipse.jetty.server.Server">

 <Set class="org.eclipse.jetty.util.resource.Resource" name="defaultUseCaches">false</Set> 
 <New class="java.io.File">
   <Arg>
    <SystemProperty name="java.io.tmpdir"/>
   </Arg>
   <Call name="toURI">
     <Call id="url" name="toURL">
       <Call name="openConnection">
         <Set name="defaultUseCaches">false</Set>
       </Call>
     </Call>
   </Call>
 </New>
</Configure>

Then put that on the command line eg java -jar ../start.jar caches.xml

[yangl326-Dylan]

I got the same problem. by gperftools , I found this kind of memory leak.
Total: 55838.9 MB
41320.8 74.0% 74.0% 41320.8 74.0% updatewindow
9018.8 16.2% 90.2% 9018.8 16.2% inflateInit2_
1559.4 2.8% 92.9% 1559.4 2.8% os::malloc@907260
1556.4 2.8% 95.7% 1556.4 2.8% init
587.5 1.1% 96.8% 587.5 1.1% 00007fb480508a66
551.4 1.0% 97.8% 551.4 1.0% 00007f2b1d6f4a66
451.5 0.8% 98.6% 451.5 0.8% 00007fb48177e8e9
183.1 0.3% 98.9% 183.1 0.3% 00007fb46c1b6c91
128.2 0.2% 99.1% 128.2 0.2% 00007fb480508245
120.4 0.2% 99.4% 120.4 0.2% 00007f2b1d6f4245
111.0 0.2% 99.6% 9129.7 16.4% Java_java_util_zip_Inflater_init
102.8 0.2% 99.7% 102.8 0.2% 00007f2b1e96a8e9
74.2 0.1% 99.9% 74.2 0.1% readCEN
updatewindow,Java_java_util_zip_Inflater_init ... Inflater leak.
another temporary solution is that using lower version of JDK , such as jdk7u76 this version

[janbartel]

As Oracle closed this bug, they will not be backporting any fixes for it to jdk8 for public release.

It appears that further updates to jdk8 will be for Oracle customers only, therefore Oracle customers need to ask Oracle to take this any further. See:
https://www.oracle.com/technetwork/java/javase/eol-135779.html

I'm going to close this issue, as there's nothing that I can see that Jetty can do about it at this time, other than recommend updating to a newer jdk version.