Support for Sony A7R V lossless compressed RAW files

[anxix]

The current official release (5.9) doesn't support the lossless compressed RAW files generated by Sony A7R V cameras (SONY ILCE-7RM5). The camera supports these in three different resolutions:

Large: 61MP (9504×6336 pixels)
Medium: 26M (6240×4160 pixels)
Small: 15M (4752×3168 pixels)

RawTherapee cannot read these at all. https://raw.pixls.us/ contains sample images in these formats.

These files are significantly smaller than the uncompressed option, and allow selecting different resolutions as well, which is quite convenient. I have found no solutions capable of handling these files on Linux, though Lightroom for example does support them on Android.

[Benitoite]

Here I am testing https://raw.pixls.us/getfile.php/6235/nice/Sony%20-%20ILCE-7RM5%20-%2014bit%20compressed%20(3:2).ARW with the metadata-exiv2 branch. Seems to decode and adjust raw parameters ok.

[anxix]

That one works in the official 5.9 release as well. I think that's probably lossy compressed rather than lossless compressed, which is a different storage option.
I uploaded yesterday a full new set to the website with various compression and aspect ratio options, but for some reason they don't show up (maybe there's some review process before they get listed?). Instead here are links to another online storage:

Large lossless compressed: https://drive.proton.me/urls/17CEH1EZ14#AyOMq8ZrQ1Q2
Medium lossless compressed: https://drive.proton.me/urls/NH1QW8TPXC#jHGdKnLlod8v
Small lossless compressed: https://drive.proton.me/urls/QN4EC4TJVR#JD0da8aoF2uV

These don't open in 5.9.

[anxix]

Ah, I see, the file names on raw.pixls.us are confusing - the lossless compressed files are not named compressed at all. For example https://raw.pixls.us/getfile.php/6238/nice/Sony%20-%20ILCE-7RM5%20-%204:3.ARW I think might be lossless compressed small and it doesn't open.

Another thing I noticed is that the aspect ratio setting for the supported files is ignored by RT, but I'm not sure it's generally intended behaviour. The full 3:2 information is of course in the RAW, but I'd maybe expect the crop configured in the camera to be preselected? But this is minor and maybe intentional in RT.

[Benitoite]

I cannot open those either.. Large (85MB), Small (42MB) etc.

[kmilos]

Another thing I noticed is that the aspect ratio setting for the supported files is ignored by RT, but I'm not sure it's generally intended behaviour. The full 3:2 information is of course in the RAW, but I'd maybe expect the crop configured in the camera to be preselected? But this is minor and maybe intentional in RT.

See also darktable-org/darktable#8523 (comment) and darktable-org/darktable#8523 (comment)

Also, the ARW lossless "small" (and "medium" if not from APS-C crop) are actually completely different beast from lossless FF or APS-C, as they are not raw (Bayer CFA) at all, but similar to demosaiced Canon sRaw it would seem...

[Lawrence37]

Related to #6226.

Another thing I noticed is that the aspect ratio setting for the supported files is ignored by RT, but I'm not sure it's generally intended behaviour.

Yes, it's intended. To automatically crop the image according to the settings, there would need to be a "Camera" option in the crop tool.

[tankist02]

I have Sony A7IV and RT 5.9 dev cannot open lossless compressed L files from it. The size difference between these and uncompressed is huge, e.g. 41 vs 68 MB. It would be really helpful to be able to use the lossless compression format. For comparison Darktable 4.7 dev has no problems with such images. I tried the metadata-exiv2 branch, but it also was not able to load the images. I'm familiar with Linux and C++ programming, can I help in any way?

[Lawrence37]

Hi @tankist02, thanks for your interest. The easiest way to add support would be to port the RawSpeed (I think that's what darktable uses for these images) implementation to RawTherapee's custom dcraw decoder. The first steps are of course to locate and understand the relevant RawSpeed code. If you are able to help with this, it will get the ball rolling.

FYI the current development version has everything the metadata-exiv2 one has.

[kmilos]

The codec is actually just good old ljpeg92 (i.e. same as DNG/Canon CR2 compression, so it should be present somewhere in dcraw already), just that channels/MCUs are arranged somewhat differently (for that you can refer to RawSpeed).

[tankist02]

Hi @Lawrence37, I took a look at DCraw stuff in RT and LibRaw with RawSpeed in DT. Wow, a lot of code which is totally unfamiliar to me. I saw that Canon CR3 support was "stolen" from LibRaw as separate functions in dcraw.h in RT. Do you have something similar in mind for Sony A7IV?

[Lawrence37]

We can borrow from RawSpeed. There are a couple of snippets from RawSpeed and LibRaw in dcraw.cc you can refer to. Whether a separate function(s) is needed depends on how much new code is required. Just follow reasonable coding practices.