Sony ILCE-1 camera support - JPEG compression?

[1900mcm]

Unable to open file in darktable ( maybe this needs to be in rawspeed )

[rawspeed] (ilce-1_DSC0735.ARW) virtual rawspeed::RawImage rawspeed::ArwDecoder::decodeRawInternal(), line 149: Unsupported compression

(darktable:3675): GLib-GObject-WARNING **: 17:08:02.938: invalid unclassed pointer in cast to 'GtkDarktableExpander'

** (darktable:3675): CRITICAL **: 17:08:02.938: dtgtk_expander_get_expanded: assertion 'DTGTK_IS_EXPANDER(expander)' failed

[1900mcm]

I think the ARW is uploaded to raw.pixls.us - though I can't find it after uploading.

[github-actions]

This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[kmilos]

Btw, I found it a bit strange that this new compression mode has

Exif.SubImage1.ImageWidth                     8704
Exif.SubImage1.ImageLength                    6144

as opposed to 8672x5784 of the uncompressed or lossy compressed...

Even though 512x512 tiles are used, the TIFF writing/reading libraries should handle desired size and pad/crop tiles along the right and bottom borders automatically, right? I don't think there is a requirement to set the size to be an integer number of tiles, so not sure what Sony is doing here...

As a result, the generated info on RPU is a bit whacky as well (should still be 3:2, 50.16 MP and say "compressed"):

[github-actions]

This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[kmilos]

Actually, this new ARW mode keeps getting weirder, I just noticed both strips and tiles related tags in SubImage1?! So one has to actually look at the JPEG bytestream to confirm what's the case here...

Edit:

Indeed, there are numerous tiles starting w/ FF D8 FF C3 00 14 0E 01 00 01 00 04: lossless JPEG SOF3 256x256 w/ 4 14-bit components to represent the 512x512 Bayer CFA tile.

I guess strip related tags should be ignored and are erroneous here (another WTH Sony?)

[feiwaltan]

@kmilos any updates available for this camera support issue? Are more raw lossless compressed samples needed?

[kmilos]

Nope.

[cytrinox]

Just found out how to decode this and implemented a PoC into dnglab (will publish a clean version next days).

1. (First) TileOffsets and StripOffset is pointing to same location, so it's really a tiled image.
2. The tile size of 512x512 is correct as reported by the file. It also fits the image width/height dimensions in this IFD (8704/512==17.0 etc.)
3. Each tile is compressed with lossless jpeg with ncomp=4 and 256x256 dimension.
4. The predictor mode is 1.
5. After decompression you get a 1024x256 result image.
6. Each cluster of 4 samples per line in result image produce a two-line CFA pattern which yields a 512x512 output tile.
7. Reassemble the tiles like in good old DNG.

For step 6 pseudo code would look like:

// decompressed_line: RGGBRGGBRGGBRGGB....
out[0] = decompressed_line[0]; // R
out[1] = decompressed_line[1]; // G
out[img_width+0] = decompressed_line[2]; // G
out[img_width+1] = decompressed_line[3]; // B

out[2] = decompressed_line[4]; // R
....

@kmilos Hope this helps you.

[lclevy]

if that could help : https://github.com/lclevy/sony_raw

[kmilos]

@LucasRueckert @lclevy @NickLplus

We don't really need other implementation details, thank you, here is already all the info we need - rawspeed already implements LJPEG92 w/ predictor 1 in https://github.com/darktable-org/rawspeed/blob/develop/src/librawspeed/decompressors/LJpegDecompressor.cpp

It "just" (famous last words) needs to be adapted to 4 (2x2 interleave, not 4x1 as currently assumed) components instead of DNG's 2 (2x1) in a way that keeps the rawspeed maintainer happy, and some glue code in https://github.com/darktable-org/rawspeed/blob/develop/src/librawspeed/decoders/ArwDecoder.cpp

(One can start maybe by looking at DngDecoder.)

[lclevy]

Canon implemented lossless jpeg in cr2 in ... 2004

[kmilos]

Canon implemented lossless jpeg in cr2 in ... 2004

So perhaps they can update the rawspeed code for us? 😉

[lclevy]

https://github.com/lclevy/libcraw2/blob/master/src/craw2.c

[kmilos]

I though I was clear: we have all the info we need, but thanks again.

What we don't have is a developer up for the job.

[artemist]

I hacked something together in https://github.com/artemist/rawspeed/tree/sony-ljpeg though I haven't tested it much since I don't have any Sony cameras yet. I don't think it's upstream ready but it does work on some samples from https://raw.pixls.us/

[octaflop]

@artemist have you considered pushing a PR up to the rawspeed mainline with your changes?

[AchimDittrich]

Can you meanwhile say by when SONY A7R V uncompressed and lossless compressed RAW files will be supported?

[kmilos]

Sony ILCE-7RM5 uncompressed and lossy is already supported in dt 4.2, as already explained in #13183

As for the lossless mode, I'm going to start repeating @MStraeten more and more: “Grass does not grow faster if you pull it”.

[AchimDittrich]

Hi Miloš,   apologies, you are right, uncompressed RAW files are also supported. I just verified it.     Best wishes,   Achim Dittrich   Homepage: achimdittrich.de / achimdittrich.com Instagram: achimdittrich_photography     Gesendet: Mittwoch, 11. Januar 2023 um 17:34 Uhr Von: "Miloš Komarčević" ***@***.***> An: "darktable-org/darktable" ***@***.***> Cc: "AchimDittrich" ***@***.***>, "Comment" ***@***.***> Betreff: Re: [darktable-org/darktable] Sony ILCE-1 camera support - JPEG compression? (#8523)   Sony ILCE-7RM5 uncompressed and lossy is already supported in dt 4.2, as explained in #13183 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[1900mcm]

Uncompressed is supported ( A1 and A7R5 )
Compressed is supported ( A1 and A7R5 )
Lossless Compressed ( S, M, L ) do not seem to be supported in darkroom edit tab ( imbedded jpeg can be viewed in lighttable tab ) ( A1 and A7R5 - probably the A74 as well )

now i have to find the mstraeten ref - ( sounds like "never ask when!" )
may thanks for the support to date for these sony cameras

[AchimDittrich]

Morning 1900mcm!   Regarding the support of uncompresed RAW files, I'm aware of it. But in the light of the fact that lossless compressed (lossless L) files are nearly only half as large as uncompressed files it would really be useful to have that support too. But I'll be patient and wait until that support arrives.     Best wishes,   Achim   Homepage: achimdittrich.de / achimdittrich.com Instagram: achimdittrich_photography     Gesendet: Mittwoch, 11. Januar 2023 um 23:46 Uhr Von: "1900mcm" ***@***.***> An: "darktable-org/darktable" ***@***.***> Cc: "AchimDittrich" ***@***.***>, "Comment" ***@***.***> Betreff: Re: [darktable-org/darktable] Sony ILCE-1 camera support - JPEG compression? (#8523)   Uncompressed is supported ( A1 and A7R5 ) Compressed is supported ( A1 and A7R5 ) Lossless Compressed ( S, M, L ) do not seem to be supported in darkroom edit tab ( imbedded jpeg can be viewed in lighttable tab ) ( A1 and A7R5 - probably the A74 as well ) now i have to find the mstraeten ref - ( sounds like "never ask when!" ) may thanks for the support to date for these sony cameras — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[LebedevRI]

I think this was resolved by 1b707f3.