Support Sony ARW lossless compression decoding

[kmilos]

See SubIFD0 of the Sony A1 sample here.

This is very similar to the already supported DNG lossless JPEG (same compression "7"), with the small difference in tile arrangement. The ARW SubIFD0 is currently successfully decompressed (by lj92 codec I presume), but the image output is slightly jumbled because of difference in channel interleaving.

In DNG, a 256x256 CFA tile is compressed as 128x256x2ch JPEG SOF3 tile, and I think line de-interleaved like this:

1111
2222
1111
2222

In ARW, this is 512x512 CFA tile to 256x256x4ch JPEG SOF3 tile which I guess should be de-interleaved like

1212
3434
1212
3434

(or potentially channels 2 and 3 swapped)

[cgohlke]

I think this is out of the scope of tifffile. With the exception of YCbCr compressed JPEG streams, which are converted to RGB by the libjpeg library, tifffile does not handle color-spaces at all. Tifffile attempts to read/decompress/unpredict/unpack/assemble the data in the file to numpy arrays if possible but does not apply any further transformations based on other metadata like upsampling, applying palettes, or demosaicing.

[kmilos]

Tifffile attempts to read/decompress/unpredict/unpack/assemble the data in the file to numpy arrays if possible but does not apply any further transformations based on other metadata like upsampling, applying palettes, or demosaicing.

Perhaps I did not explain clearly enough and created an unnecessary misunderstanding - there is no colorspace conversion taking place here nor any significant new functionality/transformation/interpolation required on the part of the tifffile library.

DNG tile reshaping after jpeg_decode() is already (automagically?) supported (JPEG SOF3 128x256x2 -> unpacked 256x256x1 CFA), I'm just asking for a small variant to support the "new" 256x256x4 -> 512x512x1 reshape. The data after that is still "raw" CFA, no color space conversion or demosaicing is asked for, just unpacking in the correct CFA order.

[cgohlke]

It's not a reshape, but something like

t = numpy.empty((512, 512), 'uint16')
t[::2, ::2] = data[..., 0]
t[1::2, ::2] = data[..., 1]
t[::2, 1::2] = data[..., 2]
t[1::2, 1::2] = data[..., 3]
data = t

Where is this specified? I couldn't find it in the TIFF, EP, or DNG specifications.

[kmilos]

That's right, some form of interlace rather than simple reshape.

And I don't think there is a spec - seems to be Sony's concoction, the ARW lossless compression mode has just been rolled out with the A1 at the start of this year.

Speaking of specs, I don't see the 256x256x1 CFA <-> 128x256x2 packing & unpacking for the DNG lossless JPEG specified anywhere neither (I also concluded that from the JPEG SOF3 signature), but it just works automagically in tifffile without e.g.

t = numpy.empty((256, 256), 'uint16')
t[::2, ...] = data[..., 0]
t[1::2, ...] = data[..., 1]
data = t

I wasn't able to dig in deep enough to understand how just yet, maybe the reshape() with the 3rd axis of 2 just works out as explained here.

[kmilos]

Actually the DNG scheme (line interlacing) works out with a reshape. I needed a transpose in there somewhere I don't see in the current code though, but I'm assuming (128,256,2) decoded tile shape which is maybe incorrect. Here's an example w/ (4,8,2):

data2 = np.ones((4,8,2), dtype=np.uint16)
data2[..., 1] = data2[..., 1]*2
data2.transpose((0,2,1)).reshape((8,8))
Out[53]: 
array([[1, 1, 1, 1, 1, 1, 1, 1],
       [2, 2, 2, 2, 2, 2, 2, 2],
       [1, 1, 1, 1, 1, 1, 1, 1],
       [2, 2, 2, 2, 2, 2, 2, 2],
       [1, 1, 1, 1, 1, 1, 1, 1],
       [2, 2, 2, 2, 2, 2, 2, 2],
       [1, 1, 1, 1, 1, 1, 1, 1],
       [2, 2, 2, 2, 2, 2, 2, 2]], dtype=uint16)

It turns out the ARW 4-way interlacing can be done with 2 successive reshapes (w/ maybe some more care on the transpose if the channels 2 and 3 need to be swapped), so this can potentially be done in a general way using a while loop where the 3rd axis is halved each time (4 -> 2 ->1). Example w/ (4,4,4) tile shape to (8,8)

data4 = np.ones((4,4,4), dtype=np.uint16)
data4[..., 1] = data4[..., 1]*2
data4[..., 2] = data4[..., 2]*3
data4[..., 3] = data4[..., 3]*4
data4.reshape((4,8,2)).transpose((0,2,1)).reshape((8,8))
Out[59]: 
array([[1, 3, 1, 3, 1, 3, 1, 3],
       [2, 4, 2, 4, 2, 4, 2, 4],
       [1, 3, 1, 3, 1, 3, 1, 3],
       [2, 4, 2, 4, 2, 4, 2, 4],
       [1, 3, 1, 3, 1, 3, 1, 3],
       [2, 4, 2, 4, 2, 4, 2, 4],
       [1, 3, 1, 3, 1, 3, 1, 3],
       [2, 4, 2, 4, 2, 4, 2, 4]], dtype=uint16)

[kmilos]

Btw, I just came across a DNG that also has a 4-component JPEG SOF3, converted by Adobe DNG converter from an older Fujifilm S5Pro camera... It comes from a more complicated non-Bayer CFA geometry:

BitsPerSample                   : 16 16
Compression                     : JPEG
PhotometricInterpretation       : Color Filter Array
SamplesPerPixel                 : 2
TileWidth                       : 192
TileLength                      : 160
CFARepeatPatternDim             : 2 4
CFAPattern2                     : 0 1 2 1 2 1 0 1

The JPEG SOF3 is 160x96x4: FF C3 00 14 10 00 A0 00 60 04 00 11 00 01 11 00 02 11 00 03 11 00

No idea how one would unpack that...

And then there's the more recent Fujifilm 6x6 X-Trans CFA that in DNG ends up as a singe component in JPEG SOF3: FF C3 00 0B 10 01 00 01 00 01 00 11 00, which fortunately requires no extra unpacking.

So even the DNG packing/unpacking scheme is not just a single (Bayer) one either currently assumed, and unfortunately not documented in the spec, one would probably have to go through the SDK code to cover them all...

[cgohlke]

As far as I understand the DNG spec, JPEG compressed segments in DNG do not require unpacking. This kind of unpacking is not part of the TIFF/DNG/EP specifications. I am surprised that ARWs differ.

DNG specification, May 2019, Page 18:
"For lossless JPEG, the internal width/length/components in the JPEG stream are not required to match the strip or tile’s width/length/components. Only the total sample counts need to match. It is common for CFA images to be encoded with a different width, length or component count to allow the JPEG compression predictors to work across like colors."

[kmilos]

And yet, they are stored as 2-component SOF3 as confirmed in hex editor, and also by Mr Baldwin's reverse engineering effort...

The quoted comment in the spec also corroborates this! "Only the total sample counts need to match."

[cgohlke]

I wasn't able to dig in deep enough to understand how just yet, maybe the reshape() with the 3rd axis of 2 just works out as explained here.

Thanks for the link.

I'm not sure how to proceed:

I think the current code is correct for DNG. If you have DNGs created by Adobe's converter that are decoded wrong by tifffile, please open a new issue. One possible problem is that imagecodecs has two codecs for reading lossless JPEG. It could be that for DNG only the codec based on lj92 should be used...

As for ARW, so far I consider their compression scheme requiring additional unpacking a bug. I'm not sure to super-special case ARW/LJPEG/4samples in the tifffile decoder.

[kmilos]

The current code is indeed correct for lossless DNGs (as produced by Adobe DNG converter), and I have verified for old Fujifilm EXR/SuperCCD, new Fujifilm X-Trans and regular Bayer CFAs. They all have different JPEG SOF3 packing, but it all works out (for some reason though my breakpoints don't hit so I wasn't to convince myself how just yet).

As far as Sony lossless ARW goes, I don't think Sony (or anyone else) claims those are DNG compliant files. ARW is a proprietary format that happens to be TIFF based. So they are free to choose the JPEG SOF3 packing scheme and I wouldn't call this a bug. More of an unnecessary reinvention of the wheel type of thing...

In any case, the added warning is a nice touch, and this can stay on the back burner.

Btw, can you please also add tags on GitHub for the latest tifffile and imagecodecs releases?

[cgohlke]

can you please also add tags on GitHub for the latest tifffile and imagecodecs releases?

That's strange. GitHub Desktop seems to have stopped pushing tags from my local repositories.

I don't think Sony (or anyone else) claims those are DNG compliant files

Sure. It' probably an oversight. I think Sony could have registered another compression tag value, SONY_ARW2, and include a note that it's LJPEG with unpacking, to avoid confusion.