Iterating over JarFile.entries() is very slow.

[larsgrefer]

Iterating over org.springframework.boot.loader.jar.JarFile#entries() is very slow. The successive calls to JarFileEntries#getEntry by EntryIterator#next are causing the underlying RandomAccessFile to jump back and forth in order to re-read the file headers from the central directory of the jar file. This could be much faster if JarFileEntries#visitFileHeader would store the complete FileHeader instead of just it's offset and the hash of its name.

Some Background

While working on joinfaces/joinfaces#565 i noticed that ClassGraph is about 500ms slower when scanning a repackaged Spring Boot application than scanning the same application in its unpacked form. So I had a look at the code of ClassGraph and saw that it extracted all the nested jars in order to scan them. Hoping to improve the performance of scanning nested jars, I prepared the following patch to use the JarFile implementation of spring-boot-loader in order to avoid the extra cost of extracting all the nested jars:
https://github.com/larsgrefer/classgraph/compare/ba4c69347eaf915571e9f5142e09f7a481471570...cfb317aff4d6949afbddcc1fd0ad78b118ef52ec?expand=1

Surprisingly this approach is even a bit slower, so I dug deeper into the code and traced the performance difference down to the Iteration done here: https://github.com/classgraph/classgraph/blob/b170d2bebb871824f7d53d54aa7a9b6939f25cf0/src/main/java/io/github/classgraph/utils/JarfileMetadataReader.java#L148

In my tests, iterating over a org.springframework.boot.loader.jar.JarFile is about 5 to 10 times slower than iterating over a java.util.zip.ZipFile. Ths performance impact is so severe that its even faster to extract the nested jar first, to be able to use java.util.zip.ZipFile

Conclusion

After I read the commit message of e2368b9 it seems to me, that memory efficiency is more important for you than iteration performance. So my question is if you would accept a pull request which changes the current behavior or allows ClassGraph to change the behavior of the JarFileEntries implementation and how this PR should look like.

[wilkinsona]

Interesting findings, thank you for sharing them.

Our overriding goal with any changes that we've made to Spring Boot's fat jar support is to optimise for startup performance. While the changes made in e2368b9 reduced memory usage, the end result was faster startup due to reduced GC pressure. This includes scanning of the jar's contents by Spring Framework's class path scanning machinery.

We haven't, to my knowledge, done any analysis of startup performance when ClassGraph is involved. On first impression, it seems to take a different approach to scanning than is taken by Spring Framework. It may be that the change you're proposing here could be generally useful or it could only be of use when ClassGraph-style scanning is being performed. Another possibility is that we can recommend some changes to ClassGraph that improve its performance without requiring any changes in Boot.

Could you please provide some example code that reproduces the behaviour that you've described above? I'd like to take a look at both the real-world scenario with Joinfaces and the more focussed case where you've looked at org.springframework.boot.loader.jar.JarFile vs java.util.zip.ZipFile.

[larsgrefer]

I'm not sure if the proposed change would have a big impact on the GC pressure. The necessary objects are created anyway and passed to JarFileEntries#visitFileHeader. They just aren't stored, so getEntry needs to create them again while iterating.

Instead of changing the current implementation it would also be possible to add a second implementation of JarFileEntries.

I'll publish my example code later and then link it here so you can have a look at it.

[larsgrefer]

Here is my example project: https://github.com/larsgrefer/zip-performance-test

Just execute the unit tests and have a look at their output (stdout)

On my machine these are the results:

3.738206 ms/iteration for java.util.zip.ZipFile
51.757802 ms/iteration for org.springframework.boot.loader.jar.JarFile

2,796645 ms/iteration for java.util.zip.ZipFile
53,356237 ms/iteration for org.springframework.boot.loader.jar.JarFile

[larsgrefer]

On first impression, it seems to take a different approach to scanning than is taken by Spring Framework.

Spring Framework iterates over JarFile#entries() exactly here. (The Spring Boot JarFile is obtained here)

[wilkinsona]

Thanks for the sample. Unfortunately, testing ZipFile and JarFile in isolation can be misleading in a situation like this. Memory usage or garbage creation in one component can have a significant effect on overall performance or the performance of another component. By looking at ZipFile and JarFile in isolation, many of the others factors that contribute to GC-related performance have been eliminated. This is why I asked for examples of both the real-world scenario and the more focussed case. You've provided the latter but not the former. Without the former, we could make changes to JarFile that appear to make it faster, but actually slow things down in the real-world scenario.