digiKam: JPEG RGB image of 85 Mpx can be saved as JPEG2000, but cannot be re-loaded

[cgilles]

Hi,

Please read the story from the DOWNSTREAM bug : https://bugs.kde.org/show_bug.cgi?id=447240.

Image can be converted from 85Mpx JPEG RGB to JPEG2000 with jasper but image sounds like impossible to open later in digiKam with alos libjasper.

Image can be loaded without problem with Krita 5.0, Gimp 2.10, or Darktable, as all used OpenJPEG framework instead LibJasper.

The error message coming from libjasper :

maximum number of samples exceeded (72480768 > 67108864)
error: cannot decode code stream

Note : image with reduced physical size can be converted and re-open in digiKam. Probably a limit have been introduced with recent version of libjasper which cause this dysfunction.

The problem can be reproduced in Linux and Windows.

• Windows : libjasper 2.0.33
• Linux : libjasper 2.0.27

Note 2 : I tried the same workflow with a simple Sony A77 of 24Mpx JPEG, and the dysfunction is the same.

Can you reproduce this problem ?

Thanks and happy new year.

Gilles Caulier

[mdadams]

Gilles, I couldn't seem to find the specific image file that could not be decoded at the link that you provided. But given the situation that you have described, I don't think that I need it. It seems that this is a simple case of your application trying to decode an image with more samples than what the application using JasPer allows. This is not the fault of JasPer, as the application must set this value appropriately, since what value is reasonable depends very much on the environment in which JasPer is being used and there is no way that JasPer can know this. Based on what your message said, the application you are using has either implicitly (via a default) or explicitly(via the JasPer API) set the maximum number of samples to 67108864 and the image to be decoded has 72480768 samples, which is too large. Each application program that uses JasPer needs to set the maximum number of samples that are allowed for an image to be decoded. This is an important security measure against DoS attacks and such. It sounds like the application that you are using does not set a high enough limit. In any case, this situation is likely to improve in JasPer 3.x (i.e., the next release of JasPer), as it is introducing a new mechanism to protect against exhaust-all-available-memory attacks, which is based on limiting total memory usage by the library, instead of restricting the number of samples in individual images to be decoded. I think that application developer are more likely to set a bound on memory usage to something reasonable than a bound on the number of samples for individual images. I hope that helps. In short, JasPer can certainly decode very large images, if you allow it do to so (via the appropriate settings).

If increasing the maximum samples values in the application program does not eliminate the problem, please let me know and I can reopen this issue. It seems very likely, however, that the problem is simply that the application did not set this value appropriately.

[cgilles]

Hi Michael, and thanks for your feedback.

We have take a look in Jasper API. Please take a look to the feedback after investigations :

https://bugs.kde.org/show_bug.cgi?id=447240#c11

Note : the source code of digiKam Jasper loader is located here if you want take a look :

https://invent.kde.org/graphics/digikam/-/blob/master/core/dplugins/dimg/jpeg2000/dimgjpeg2000loader_load.cpp

If you can give your expertise, it will be appreciate.

Thanks in advance

Gilles Caulier

[mdadams]

Gilles,
I had a quick look at the code that you referenced. In your code, you have the following line:

jp2_image = jas_image_decode(jp2_stream, fmt, nullptr);

For the last argument in the function call (which specifies the decoder options), you have provided a null pointer. This results in the decoder default options being used. If you want to override the default value of the maximum number of samples for decoding, you can pass a pointer to a null-terminated character string like:

"max_samples=100000000" (i.e., 100 megapixels assuming a single component image or 100/3 megapixels for a 3 component image)

Incidentally, I'm sorry that some of the decoder options are not very well documented. You can probably locate most of them by searching for "OPT_" in the source code. As part of the 3.x release, I am trying to improve some of the documentation for JasPer. At present, codec options are not mentioned in any detail the JasPer manual. Some of the options can be found in the file called jasper.pdf, but this file is very old and not up-to-date. I don't think that the max_samples option is described in this document.

By the way, JasPer takes security quite seriously. I haven't looked at the code for OpenJPEG in any detail, but some people have told me that JasPer has much better properties with respect to security. At least, a quick scan of the issues for OpenJPEG today shows numerous open CVEs, some of which have been open and unfixed since 2016. Imposing limits on the size of images to be decoded is extremely important from a security standpoint. Unfortunately, since some aspects of JasPer are not so well documented, it is sometimes easy for application developers to miss things like useful codec options that control the maximum size of image that can be decoded.

In the upcoming 3.x release of JasPer, this problem that you are having may go away (since max_samples may effectively be defaulted to infinity with a JasPer library memory limit instead being used to protect against memory-exhaustion attacks). But I cannot say for certain as the code is still in development, and things are still changing significantly at this point. In any case, the above fix to your problem should still work in JasPer 3.x (assuming no unexpected code changes are made between now and the 3.x release), but the fix might not still be needed.

I hope that helps.

--Michael

[cgilles]

Thanks Michael, your feedback help a lots and it work like a charm with this fix:
https://invent.kde.org/graphics/digikam/commit/2379fef330c4449803325430b1bf1bf1778ceac7
Best regards

Gilles Caulier

[mdadams]

Gilles, thanks for confirming that this fixed your problem.

[cgilles]

Hi Michael,

Just a feedback from a long time use (more than 15 years now) : when i talk with other main open source project from graphics world few years ago, all tips me to switch from libjasper to openjpeg about security issues. After a quick look to openjpeg API, i was lost and not convinced by this main changes in digiKam code. Now with your comment about openjpeg, I'm sure, i will never left libjasper for openjpeg.

So when libjapser 3 will be available for testing, don't hesitate to ping me for technical feedback.

My best

Gilles Caulier

[mdadams]

Hi Gilles,

To be fair, JasPer did go through a period where it accumulated quite a few open bug reports. Fortunately, this situation has been greatly improved in more recent times. I had some help from a few people along the way (which helped greatly) and I have also devoted a lot more of my own time to fixing bugs and working on other improvements. The main new features in 3.x are custom memory allocators, memory allocation tracking/limiting, custom logging, and multithreading support. A preliminary version of the code can be found on the master branch. Feedback is always welcome. I still have a few more things to change/fix before the code is ready for release. If you just want to pick up new bug fixes and don't mind warning messages about using deprecated functionality, you are certainly welcome to try the current code on the master branch now. If you do so and you have any problems, please let me know. Since so many significant things have changed since the last release, there is always the danger that some library users might be adversely affected (e.g., build problems or other issues). If you would be interested in doing some testing of a preview (i.e., alpha/beta) release, I can certainly let you know when I have this ready. It would probably be good if I can find a few testers to ensure that real-world applications still build okay with the new release before I officially release it. I am not sure exactly when 3.x will be ready, but I am hoping for sometime this month (at least for an alpha/beta release). Is the new multithreading support in JasPer likely to be useful to you? Or are your applications mostly single threaded?

Incidentally, I have added some material to the manual on the encoder/decoder options for the various codecs in JasPer. This information can be found on the master branch, as well as in a draft version of the 3.x manual at:

• https://jasper-software.github.io/jasper-manual/releases/manual-version-mdadams-20220109-2039/html/codecs.html

--Michael

[mdadams]

Gilles,
If you would like to try out a prerelease alpha version of JasPer 3.0.0, I have just tagged a release version-3.0.0-unstable3:

https://github.com/jasper-software/jasper/releases/tag/version-3.0.0-unstable3

A pre-built version of the manual can be found at:

https://jasper-software.github.io/jasper-manual/releases/version-3.0.0-unstable3/html/index.html

If try it and have any problems with it, please let me know.

--Michael

[cgilles]

Thanks for the info Michael. It's on my todo list. I give you a feedback later after fist compilation and run-time tests

Best

Gilles