Support for JXL (JPEG XL)

[schrmh]

I saw #546 in which JPEG XL was mentioned in a few comments and decided to create this separate issue so that discussion can be done here instead of in an issue that is about a different format.

I want to use a software using this library to search text in screenshots stored in JXL. But this library doesn't support JXL yet.

I have hundreds of thousands in screenshots in JXL by now since those files take on average, after lossless convertion only about 47% of the space compared to when they were PNG (cjxl -q 100 -m 1 -e 9 --brotli_effort 11 -E 3 -I 100 -g 3 -j 1 in.png out.jxl).
No other common file format can compete there.

It could be even better now since 0.8.1 is out but I used 0.7 to test this (1777M PNG → 838M JXL after converting 1681 screenshots; ca. 52.841868 % reduction); in 0.6.1 I achieved about 53% new size (after converting 163552 screenshots by using cjxl -q 100 -m -s 9 -E 3 -I 1: 112216M → 59561M; ca. 46.9228986953732 % reduction. I know those are way more images but I tried 0.7 with images I converted before and did not want to spend months to get a preciser value after the comma. Also, no: I did not switch reduction and total size on accident, even tho it is kinda funny how those reduction values changed).

In a recent release -e 10 was introduced but it is experimental and it takes way longer to complete and way more RAM (20 GiB+ and almost 2 hours for a image of about one Megs; -e 9 takes usally less than 1 GiB and maybe one minute on my machine with a Ryzen 7 3700X) and thus I have only done tests with very few files with -e 10 so far and got varying results from additional 0.5 percent points to 6 percent points saved.

[DanBloomberg]

Lossless compression to 47% of PNG is very impressive. How does lossless webp compare? And what level of zlib compression in the png was used in your study?

As you saw in #546, I have no interest in adding yet another image compression format to support in leptonica. It's a huge amount of work with each format, because (1) these formats all have many features and for any feature there exist people who want it supported, (2) the reading and writing shims have to be meticulously tested with fuzzing to prevent attacks that can cause any type of failure, including crashes, memory overruns and memory smashes. For jpeg xl, we have the additional problem that Google has removed (or is in the process of removing) support in Chrome; e.g.,

     https://www.cnet.com/tech/computing/chrome-banishes-jpeg-xl-photo-format-that-could-save-phone-space/

Leptonica was an early adopter with webp, because I saw the level of internal support at Google, and I understood the advantages in perceptual quality and rate/distortion that webp has using wevelet encoding, compared to jpeg's DCT. I'm fine with waiting a few years to see how things progress with support for jxl.

[schrmh]

I did not do excessive tests with WebP since I saw this table that somebody created a while ago: https://docs.google.com/spreadsheets/d/1ju4q1WkaXT7WoxZINmQpf4ElgMD2VMlqeDN2DuZ6yJ8/
It doesn't include regular desktop screenshots but even in the area it performed best, WebP v2 was at least 5.2% (for v1 10.5%) bigger than the output of cjxl -e 9 -E 3 -I 1 (last parameter would now be -I 100) and it seems to be way bigger on black & white content (could be that JXL is just really good on that content in particular; not sure at this time). Maybe the difference is smaller when some better options are used for webp but I have yet to come across a webp that would be smaller than a JXL created with the options I mention in the first post.
However, if I knew which options are considered to be worth trying to get the best results for webp (v2) then I could test it with a few thousand files (if it isn't much worse than cjxl with -e9, etc. when it comes to resource usage.)
However, twitter.com/jonsneyers should also know some things here and will likely answer if asked something.

Regarding zlib compression level:
Can't tell right now. It is likely not 9 since the PNG files can still be made smaller by using optipng... I created the majority of the PNG files by using xfce4-screenshooter and did not find the compression level in source code or documentation — but I think I read about it years ago...
I created an issue on the project and will comment here regarding that if they answer.
(From the source code I gathered that it seems to use ZlibCompressor from glib-2.0 / gio:
It's using basically this header file: https://gitlab.gnome.org/GNOME/glib/-/blob/main/gio/gzlibcompressor.h but on my system there is no level parameter for g_zlib_decompressor_new in that header file...
I also looked at the documentation: https://docs.gtk.org/gio/class.ZlibCompressor.html and https://docs.gtk.org/gio/enum.ZlibCompressorFormat.html#zlib but there is no level mentioned...)

Regard adding it JXL to leptonica:
Yes, I understand it is a huge amount of work, especially for an open source project that might be a side project / not well funded (not really sure if this is the case for your project; I only skimmed a bit through the website and there is also no Wikipedia page about this library).
The thing here is this library is used by tesseract and OpenCV which are both pretty big projects, so it is kinda blocking those from being able to process JXL.
From tesseract-ocr/tesseract#4048 when I asked about JXL support: "tesseract uses leptonica for image-opening images, so we can do nothing here."
For OpenCV there is an issue open since almost two years which got no answer from the project: opencv/opencv#20178
(however, for OpenCV this might be usable somehow: https://github.com/google/brunsli/blob/master/contrib/py/jxl_library_patches/README.txt).

Maybe some kind of development fund could be used here in case funding is an issue... I know that Germany has quite a few.

And well... Google's Chromium is a web browser. Sure, the web is important nowadays but should not be the main factor when it comes to determining if it is worth supporting a file format or not (especially not if the decision to not include it in the browser was made against a lot of known — and some very huge — names in the IT space — Facebook, Adobe, Intel, VESA, Krita, The Guardian, libvips, Cloudinary, Shopify. CTRL+F here: https://bugs.chromium.org/p/chromium/issues/detail?id=1178058 — and just to push a overall worse format). Most formats are not used on the web anyways and there are actually JXL viewer extensions for web browsers.
And both Qt and GTK, the dominating toolkits on GNU/Linux, support JXL and thus every image displaying application using one of those should be able to display JXL (at least my experience when I tried a bunch of image viewers after adopting JXL early about two years ago).

Well, maybe somebody with some free time and enough experience with image formats sees this issue and decides to contribute here.

[DanBloomberg]

I appreciate your interest and concern with efficient codecs. I have been responsible for the generation and storage of about 100 pB of images, so I share your interest and understand the importance of using good compression. And when I first heard about jxl it seemed like a codec worth supporting, especially if the high-level interface to the library is essentially the same as jpeg. As mentioned above, I haven't given up on it.

Decisions on which codecs to support are complicated. Leptonica was an early adopter of the openjpeg implementation of jpeg2000 (because it uses wavelet compression which avoids jpeg DCT artifacts at edges), and it was a mess, with interfaces changing over the first 5 years, and very poor encoding performance. I also reluctantly wrote shims for gif, which for 20 years had patent issues with their compression, has relatively poor universal compression, max of 256 colors, and a big interface change about 5 or 6 years ago with gif5. But it has widespread use, so supporting it in leptonica was the right thing to do.

Direct monetary support for leptonica development has never been an issue, or a consideration.

I understand that interest in jxl will continue to grow, especially if the codex library can be made fast (especially decoding) and effective for both lossy and lossless compression compared to other wavelet-based codecs. We'll keep the issue open.

[schrmh]

Ah, I see. Thanks for keeping it open.

I got a reply from a xfce4-screenshooter dev and they told me it is likely zlib compression level 5:
https://gitlab.gnome.org/GNOME/gdk-pixbuf/-/blob/ad6f0d44fbbae586e454d5f380e27ca80dc3b00e/gdk-pixbuf/io-gdip-png.c#L36

Can't say much about changing interfaces for JXL. But "the file format and core coding system were formally standardized on 13 October 2021 and 30 March 2022 respectively" (https://en.wikipedia.org/wiki/JPEG_XL) and all JXL files created with that standard since then will be decodable by future versions.
However, I can't judge code changes so far since I haven't taken a look there but there were some — from the perspective of a user — unexpected changes of options from cjxl 0.6.1 to 0.7.0 (-s deprecation in favor of -e; -I range change to 0-100 from 0-1; -m needing 0 or 1 after it now).

Also yep, it seems like there is a lot of potential to make handling JXL faster (and less memory using):

And there is room for further compression improvements:

https://www.reddit.com/r/jpegxl/comments/11pf9kw/comment/jbxjkfw/

[schrmh]

Small update: Looks like the iOS 17 beta introduces OS-level JXL support. They also have JPEG XL on Safari release notes: https://developer.apple.com/documentation/safari-release-notes/safari-17-release-notes
For Safari and Photos app I saw the support on pictures (https://twitter.com/luciascarlet/status/1665864299686907906/photo/1 & https://media.discordapp.net/attachments/803574970180829194/1115418398030053446/IMG_0961.png) but one person said that most 3rd party apps should be able to display it as well (can't look into that myself since I'm lacking a capable Apple device).