opj_compress v2.1.2 can create images opj_decompress cannot read

[jechols]

We're having a problem where opj_compress can occasionally produce a JP2 which opj_decompress cannot read. Sometimes changing the encoding rate fixes the problem, but sometimes we iterate hundreds of times with different rates and still have no luck. The original TIFF files don't seem to have any problems, and opj_encode doesn't seem to have any unusual complaints creating a JP2.

We've created a github repository with some examples, mostly for our own documentation, but the information provided may be of some assistance.

We get broken JP2 files when converting from raw TIFFs, when converting from TIFF to PNG and then to JP2, and sometimes even when going from TIFF to BMP to JP2. The latter gives us the most success, but the resulting JP2 usually has 3 channels despite the TIFF having only 1, so we can't really use that process.

Here's an example of output when decoding a JP2 we encode with opj_compress (for convenience, we've put the JP2 in question into the github repository as well):

$ /usr/local/openjpeg-2.1.2/build/bin/opj_decompress -i batch_dallaspubliclibrary_20170104BronzeManananggalPlantingSunflowers-sn93
051615-1869-05-22-0001.jp2 -o /tmp/out.png -r 4

[INFO] Header of tile 39 / 79 has been read.
skip: segment too long (3078) with max (177) for codeblock 0 (p=0, b=1, r=3, c=0)
[ERROR] Failed to decode.
[ERROR] Failed to decode tile 40/80
[ERROR] Failed to decode the codestream in the JP2 file
ERROR -> opj_decompress: failed to decode image!

[boxerab]

It would be most helpful if you could please

a) upload the original TIFF file
b) post the encoder settings
c) post the decoder settings
d) upload the encoded file

for a TIFF file that produces a broken JP2 file.

[malaterre]

one of the most interesting bug report in while. Scary that lossless encoding is not lossless...

[boxerab]

Hello Mathieu, actually this isn't bad - lossless encoding with compression ratio greater than 1 results in lossy compression, it just uses the lossless 5/3 wavelet transform, with no quantization. So, this is expected.
Problem is messed up MQ coder segments. That is bad.

[jechols]

I can't attach the TIFFs as they're too big, but follow this link to one I just double-checked. Github also won't let me attach JP2 files, so I've included what I just now generated from the commands below:
jp2s.zip

My settings:

• Created a lossless version: opj_compress -i batch_dallaspubliclibrary_20170104BronzeManananggalPlantingSunflowers-sn93051615-1869-05-22-0001.tif -o out.jp2 -t 1024,1024 -r "20"
• Created a lossy version: opj_compress -i batch_dallaspubliclibrary_20170104BronzeManananggalPlantingSunflowers-sn93051615-1869-05-22-0001.tif -o out-lossy.jp2 -t 1024,1024 -r "20" -I
• Attempted to decode the lossless version: opj_decompress -i ./out.jp2 -r 4 -o .out.png (this fails on tile 40)
• Attempted to decode the lossy version: opj_decompress -i ./out.jp2 -r 4 -o .out.png (this succeeds)

I tried with and without the -I flag in the past, and results were still unpredictable, but I can't say for sure which rates and which TIFF files will give any particular error. We built a script to iterate over a few hundred rates on the problem files and found that most worked if we tweaked the rate, but they all had problems at some rate values, and some didn't work for any of the rates we tried.

[boxerab]

Thanks. So, even in pure lossless mode, I am seeing this error. i.e. this really is quite bad: lossless encoding is actually lossy.

[boxerab]

By the way, not to be pedantic, but both files that you created would be considered lossy. The first uses lossless wavelet transform and no quantization, while second uses lossy wavelet transform and quantization.

[jechols]

Good to know. I haven't delved very deep into creating JP2 files, so I've mostly just played with settings until things seem to work reasonably well. We were originally just using GraphicsMagick, telling it to tile with a given quality value, and calling it good. When it died on us (different problem), we had to kind of rush jp2_compress into our process.

[boxerab]

As a work around, you could try setting tile size to 512x512 ? This seems to work for me.

[jechols]

I think changing settings in general seems to give working results. The problem is that the results are inconsistent. For instance, 512x512 works well on the TIFF I've provided, but doesn't work for another TIFF in our repository

Our workaround has been to just twiddle settings (mostly the rate) until something works, but we haven't found a single setting that we can apply to all TIFFs we have in a uniform way.

[boxerab]

Yeah. I think it is related to the dimensions of the image and the dimensions of the tiles. So, this isn't a very good workaround, I guess :( This kind of bug will take a long time to fix, as it is happening in the guts of the codec.

[boxerab]

@jechols If you like, you can use Grok codec to encode these images
https://github.com/GrokImageCompression/grok

I tested with your image and encoding parameters, and everything works without error. Also, if no compression ratio is set, then compression is truly lossless : mathematically equal to original. This indicates that all is well with both encoder and decoder.

[rouault]

Fixed per #949

[jechols]

I hadn't been keeping up on this for a while - thanks for the fix!