Migration from v3.8.3 to v3.9.3 (3.9.0) gives OutOfMemory Error

[AndrewG10i]

Describe the bug
First of all - thank you so much for your work and such a great image lib!

We are facing with the following issue and would appreciate if you can help to figure out whether it is related to TwelveMonkeys ImageIO lib, or to something else.

So far it is confirmed that after replacing only one single dependency (TwelveMonkeys ImageIO) from v3.8.3 to v3.9.X (tested 3.9.0 and 3.9.3) we start getting Java Heap Memory Errors while running load-tests which pass absolutely fine using v3.8.3. (Environment, JVM settings, etc - all remains the same in tests, only lib version changes).

Memory dump is available as well for analysis.

Version information

1. TwelveMonkeys ImageIO library: 3.9.3 (3.9.0 also gives the same issue)

2. Java:
   openjdk version "11.0.14.1" 2022-02-08 LTS
   OpenJDK Runtime Environment Zulu11.54+25-CA (build 11.0.14.1+1-LTS)
   OpenJDK 64-Bit Server VM Zulu11.54+25-CA (build 11.0.14.1+1-LTS, mixed mode)

3. Extra information about OS version, server version, standalone program or web application packaging, executable wrapper, etc.
   Linux CentOS 8 Stream

To Reproduce
Currently not available but willing to prepare.

Expected behavior
No Java OutOfMemory error to happen the same as with v3.8.3.

Example code
Currently not available but willing to prepare.

Sample file(s)
N/A

Stak trace

__ejb-thread-pool2
  at com.twelvemonkeys.imageio.stream.MemoryCache.fetchBlock()[B (MemoryCache.java:50)
  at com.twelvemonkeys.imageio.stream.MemoryCache.read(Ljava/nio/ByteBuffer;)I (MemoryCache.java:86)
  at com.twelvemonkeys.imageio.stream.BufferedChannelImageInputStream.fillBuffer()Z (BufferedChannelImageInputStream.java:156)
  at com.twelvemonkeys.imageio.stream.BufferedChannelImageInputStream.read([BII)I (BufferedChannelImageInputStream.java:197)
  at org.apache.pdfbox.jbig2.io.SubInputStream.fillBuffer()Z (SubInputStream.java:141)
  at org.apache.pdfbox.jbig2.io.SubInputStream.read()I (SubInputStream.java:88)
  at javax.imageio.stream.ImageInputStreamImpl.readBits(I)J (Unknown Source)
  at org.apache.pdfbox.jbig2.JBIG2Document.reachedEndOfStream(J)Z (JBIG2Document.java:334)
  at org.apache.pdfbox.jbig2.JBIG2Document.mapStream()V (JBIG2Document.java:197)
  at org.apache.pdfbox.jbig2.JBIG2Document.<init>(Ljavax/imageio/stream/ImageInputStream;Lorg/apache/pdfbox/jbig2/JBIG2Globals;)V (JBIG2Document.java:113)
  at org.apache.pdfbox.jbig2.JBIG2ImageReader.getDocument()Lorg/apache/pdfbox/jbig2/JBIG2Document; (JBIG2ImageReader.java:315)
  at org.apache.pdfbox.jbig2.JBIG2ImageReader.getPage(I)Lorg/apache/pdfbox/jbig2/JBIG2Page; (JBIG2ImageReader.java:322)
  at org.apache.pdfbox.jbig2.JBIG2ImageReader.read(ILjavax/imageio/ImageReadParam;)Ljava/awt/image/BufferedImage; (JBIG2ImageReader.java:213)
  at org.apache.pdfbox.filter.JBIG2Filter.decode(Ljava/io/InputStream;Ljava/io/OutputStream;Lorg/apache/pdfbox/cos/COSDictionary;ILorg/apache/pdfbox/filter/DecodeOptions;)Lorg/apache/pdfbox/filter/DecodeResult; (JBIG2Filter.java:106)
  at org.apache.pdfbox.cos.COSInputStream.create(Ljava/util/List;Lorg/apache/pdfbox/cos/COSDictionary;Ljava/io/InputStream;Lorg/apache/pdfbox/io/ScratchFile;Lorg/apache/pdfbox/filter/DecodeOptions;)Lorg/apache/pdfbox/cos/COSInputStream; (COSInputStream.java:87)
  at org.apache.pdfbox.cos.COSStream.createInputStream(Lorg/apache/pdfbox/filter/DecodeOptions;)Lorg/apache/pdfbox/cos/COSInputStream; (COSStream.java:175)
  at org.apache.pdfbox.pdmodel.common.PDStream.createInputStream(Lorg/apache/pdfbox/filter/DecodeOptions;)Lorg/apache/pdfbox/cos/COSInputStream; (PDStream.java:243)
  at org.apache.pdfbox.pdmodel.graphics.image.PDImageXObject.createInputStream(Lorg/apache/pdfbox/filter/DecodeOptions;)Ljava/io/InputStream; (PDImageXObject.java:882)
  at org.apache.pdfbox.pdmodel.graphics.image.SampledImageReader.from1Bit(Lorg/apache/pdfbox/pdmodel/graphics/image/PDImage;Ljava/awt/Rectangle;III)Ljava/awt/image/BufferedImage; (SampledImageReader.java:396)
  at org.apache.pdfbox.pdmodel.graphics.image.SampledImageReader.getRGBImage(Lorg/apache/pdfbox/pdmodel/graphics/image/PDImage;Ljava/awt/Rectangle;ILorg/apache/pdfbox/cos/COSArray;)Ljava/awt/image/BufferedImage; (SampledImageReader.java:211)
  at org.apache.pdfbox.pdmodel.graphics.image.SampledImageReader.getRGBImage(Lorg/apache/pdfbox/pdmodel/graphics/image/PDImage;Lorg/apache/pdfbox/cos/COSArray;)Ljava/awt/image/BufferedImage; (SampledImageReader.java:154)
  at org.apache.pdfbox.pdmodel.graphics.image.PDImageXObject.getOpaqueImage()Ljava/awt/image/BufferedImage; (PDImageXObject.java:588)
  at org.apache.pdfbox.pdmodel.graphics.image.PDImageXObject.getImage(Ljava/awt/Rectangle;I)Ljava/awt/image/BufferedImage; (PDImageXObject.java:509)
  at org.apache.pdfbox.pdmodel.graphics.image.PDImageXObject.getImage()Ljava/awt/image/BufferedImage; (PDImageXObject.java:477)
  at org.apache.pdfbox.rendering.PageDrawer.drawImage(Lorg/apache/pdfbox/pdmodel/graphics/image/PDImage;)V (PageDrawer.java:1111)
  at org.apache.pdfbox.contentstream.operator.graphics.DrawObject.process(Lorg/apache/pdfbox/contentstream/operator/Operator;Ljava/util/List;)V (DrawObject.java:67)
  at org.apache.pdfbox.contentstream.PDFStreamEngine.processOperator(Lorg/apache/pdfbox/contentstream/operator/Operator;Ljava/util/List;)V (PDFStreamEngine.java:966)
  at org.apache.pdfbox.contentstream.PDFStreamEngine.processStreamOperators(Lorg/apache/pdfbox/contentstream/PDContentStream;)V (PDFStreamEngine.java:541)
  at org.apache.pdfbox.contentstream.PDFStreamEngine.processStream(Lorg/apache/pdfbox/contentstream/PDContentStream;)V (PDFStreamEngine.java:516)
  at org.apache.pdfbox.contentstream.PDFStreamEngine.processPage(Lorg/apache/pdfbox/pdmodel/PDPage;)V (PDFStreamEngine.java:155)
  at org.apache.pdfbox.rendering.PageDrawer.drawPage(Ljava/awt/Graphics;Lorg/apache/pdfbox/pdmodel/common/PDRectangle;)V (PageDrawer.java:284)
  at org.apache.pdfbox.rendering.PDFRenderer.renderImage(IFLorg/apache/pdfbox/rendering/ImageType;Lorg/apache/pdfbox/rendering/RenderDestination;)Ljava/awt/image/BufferedImage; (PDFRenderer.java:355)
  at org.apache.pdfbox.rendering.PDFRenderer.renderImage(IFLorg/apache/pdfbox/rendering/ImageType;)Ljava/awt/image/BufferedImage; (PDFRenderer.java:272)
  at org.apache.pdfbox.rendering.PDFRenderer.renderImageWithDPI(IFLorg/apache/pdfbox/rendering/ImageType;)Ljava/awt/image/BufferedImage; (PDFRenderer.java:258)
  ...
  at java.util.concurrent.FutureTask.run()V (Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (Unknown Source)
  at java.lang.Thread.run()V (Unknown Source)

Screenshots

Additional context
This is a Java EE application running in Payara Micro.
I do understand that so far there is not much information provided but OutOfMemory error comes after 90-120min of running load/performance tests and so far not investigated much as can be fixed simply by downgrading to 3.8.3. Will highly appreciate any clues how to better approach this issue and open to cooperate on resolving & testing any potential fixes. Thanks!

[haraldk]

Hi Andrew,

Thanks for reporting and providing insight!

Unfortunately I can't work much these days (I have covid... 🙁), but will look into this as soon as possible!

In the mean time, it would be great if you can provide code to reproduce!

I introduced new buffered stream classes in 3.9 which seems to cause the problems you see. For my own plugins, testing has shown that they provide better performance than the original JDK streams. But your stack trace indicates using PDFBox and their JBIG2 plugin, which I haven't tested. I guess further digging has to be done to figure out why/what happens...

The MemoryCache class needs to keep as much of the stream as needed by the plugin in memory to allow backwards seeking, but this memory should be all released when the stream is closed. I would expect the 1.5GB and 874MB to reflect the file sizes you are reading at the time.

However, the BufferedChannelImageInputStream holding 4 GB is very strange... Do you have a graph of the allocations here, to see where this data is held?

Again, thanks, I will look into it!

[AndrewG10i]

Hi Harald,

thank you for your reply and please get well soon! Yep, it is clear that reproducer is needed and I have also walked through changes in releases after 3.8.3 and saw those updates with introduced new streams.

I started working on the reproducer yesterday and good thing that I was able to identify 2 samples which cause OOM. Moreover preliminary seems it is related only to the samples which contain JBIG images (as samples with other image types pass fine).

So the next step is actual reproducer with minimum code. This can take a while as I bit busy with other projects, but will do my best to prepare it asap.

Thank you!

[haraldk]

Hi @AndrewG10i,

Have you been able to look more into this? I understand creating code sample to reproduce might take some time, no worries.

I really want to fix this problem, however. In the mean time, perhaps you could upload the memory dump, to see if I can get some more details out of it? Uploading the JBIG sample images that cause OOM would also be helpful, if that's possible.

[AndrewG10i]

Hi @haraldk,

Almost done with the reproducer. Code itself is ready, surprisingly it is just few lines. Now trying to find a sample that can be shared, as the one I have - could not be shared.

Hopefully will update you shortly today on this!
Thank you!

[haraldk]

I did find a possible way to trigger the OOME today, added a test case and a fix for it.

Could happen if someone was searching (far) beyond the end of the stream.

Unfortunately, there's an issue with the deploy step in the GHA workflow, so snapshot isn't deployed. But if you can test from sources, this fix could potentially fix your issue. Seems my signing key was expired. If only the error message would says something that could be interpreted like that... 😛

[AndrewG10i]

Great, thank you for the update! ...on our end we are still requiring some time to reproduce exactly the same exception using sources, as currently getting a bit different exception when using TM source code. Anyway will aim to test it today!

[haraldk]

New release (3.9.4) is out now, if you want to try that.

And yes, I was actually a bit surprised that you got the OutOfMemoryError as I catch that and throw an IOException instead... Could be some aggressive optimizations happening, I don't know. I did get IOException from my test case, before fixing.

[AndrewG10i]

Great! Yes, just managed to try with master and also 3.9.4 - reproducer works fine (meaning - no issue reproduced anymore). Will run full test and let you know in case of any issues (hopefully not :-)).