Fujifilm compressed RAF support. #107

muessel · 2017-06-23T18:56:56Z

Hello,

this pull request implements support for Fujifilm compressed RAF Files for the X-Pro2, X-T2 and X-T20. The code is based on LibRaw / rawtherapee.

LebedevRI · 2017-06-23T19:33:16Z

Hi.
Nice to finally see that.
Are there any changes other than adapting to some of rs's infrastructure?

LebedevRI

Please make CI happy :)
The code is far from ideal, which is quite understandable, but it will be easier for me to fix it after merging as-is...

LebedevRI · 2017-06-23T19:43:06Z

src/librawspeed/decompressors/FujiDecompressor.cpp

+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2 of the License, or (at your option) any later version.


+ License as published by the Free Software Foundation; either + version 2 of the License, or (at your option) any later version.

Hmm, where did you get that?
LibRaw/LibRaw@aeab4d8#diff-03b018f1deeb8fc38d2c927f09465635R11 says:

+LibRaw is free software; you can redistribute it and/or modify +it under the terms of the one of three licenses as you choose: + +1. GNU LESSER GENERAL PUBLIC LICENSE version 2.1 + (See file LICENSE.LGPL provided in LibRaw distribution archive for details). + +2. COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0 + (See file LICENSE.CDDL provided in LibRaw distribution archive for details). +

(2 vs 2.1)

Is it enough to change the version from 2 to 2.1 or do I have to copy the whole LibRaw license header?

I think it is enough just to change the version.

LebedevRI · 2017-06-23T19:45:42Z

data/cameras.xml

+			<ColorRow y="4">GGRGGB</ColorRow>
+			<ColorRow y="5">RBGBRG</ColorRow>
+		</CFA2>
+		<Crop x="0" y="6" width="-16" height="0"/>


Ugh, so they are even writing different raw image WxH in compressed vs. uncompressed formats..

Yes, uncomressed and comressed files are totally different, eg.:

exiftool Fujifilm\ -\ X-T20\ -\ compressed\ (3-2).RAF
...
Raw Image Width : 6032
Raw Image Height : 4032
Raw Image Full Width : 6048
Raw Image Full Height : 4038

exiftool Fujifilm\ -\ X-T20\ -\ Uncompressed\ (3-2).RAF
...
Raw Image Width : 6032
Raw Image Height : 4032
Raw Image Full Width : 6160
Raw Image Full Height : 4032

Yes, that is what i was talking about.

@muessel
An entry for FUJIFILM X100F is missing.
Is there some problem with that camera, or it is a simple oversight?

LebedevRI · 2017-06-23T19:49:13Z

src/librawspeed/decompressors/FujiDecompressor.cpp

+  info->cur_bit = (8 - (bits_left_in_byte & 7)) & 7;
+}
+
+int FujiDecompressor::bitDiff(int value1, int value2) {


int __attribute__(const) FujiDecompressor::bitDiff(int value1, int value2) {

LebedevRI · 2017-06-23T19:50:04Z

src/librawspeed/decoders/RafDecoder.cpp

+    auto id = mRootIFD->getID();
+    const Camera* cam = meta->getCamera(id.make, id.model, mRaw->metadata.mode);
+    if (!cam)
+      ThrowRDE("Couldn't find camera %s %s", id.make, id.model);


ThrowRDE("Couldn't find camera %s %s", id.make.c_str(), id.model.c_str());

LebedevRI · 2017-06-23T19:52:09Z

src/librawspeed/decompressors/FujiDecompressor.h

+public:
+  FujiDecompressor(Buffer input, RawImage img, int offset);
+  ~FujiDecompressor(void);
+


i guess you want to add

using ushort = ushort16;

right here for now.

LebedevRI · 2017-06-23T21:14:26Z

Ok, i see.
Ignore those windows clang-4.0 warnings, i'll handle them manually.

muessel · 2017-06-23T21:25:09Z

Travis CI has problems with mac builds with std::abs. Do I have to change them to abs and then include "using std::abs;" ?

LebedevRI · 2017-06-23T21:26:05Z

Will fix locally, thank you!

muessel · 2017-06-24T16:14:01Z

Hello Roman, I just forgot the the X100F also supports the compressed format. I've seen you also implemented support for the GFX 50S. In the method FujiDecompressor::copy_line_to_bayer a static CFA is used. Maybe this should be converted into something configurable like with the X-Trans sensors. Regards Uwe 2017-06-24 13:22 GMT+02:00 Roman Lebedev <notifications@github.com>:

…

***@***.**** commented on this pull request. ------------------------------ In data/cameras.xml <#107 (comment)> : > @@ -8598,6 +8598,19 @@ <Crop x="0" y="0" width="-128" height="0"/> <Sensor black="1024" white="16383"/> </Camera> + <Camera make="FUJIFILM" model="X-Pro2" mode="compressed"> + <ID make="Fujifilm" model="X-Pro2">Fujifilm X-Pro2</ID> + <CFA2 width="6" height="6"> + <ColorRow y="0">GGRGGB</ColorRow> + <ColorRow y="1">GGBGGR</ColorRow> + <ColorRow y="2">BRGRBG</ColorRow> + <ColorRow y="3">GGBGGR</ColorRow> + <ColorRow y="4">GGRGGB</ColorRow> + <ColorRow y="5">RBGBRG</ColorRow> + </CFA2> + <Crop x="0" y="6" width="-16" height="0"/> @muessel <https://github.com/muessel> An entry for FUJIFILM X100F is missing. Is there some problem with that camera, or it is a simple oversight? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#107 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADnzUfFnwWiZHlr3NnBgj9G4aaRCtmwTks5sHPFqgaJpZM4OD7TO> .

LebedevRI · 2017-06-24T16:24:45Z

Hello Roman,
I just forgot the the X100F also supports the compressed format.

Yeah, i checked, and X100F compressed works too, fixed in branch.
Also, while there, FUJIFILM GFX 50S uncompressed+compressed support added - the only fuji camera with bayer sensor and with this compresson scheme, i believe.

What i'm currently missing is some compressed .RAF with 12-bits data, so this branch is taken:

rawspeed/src/librawspeed/decompressors/FujiDecompressor.cpp

Lines 106 to 110 in fae7f40

    
           } else if (info->q_point[4] == 0xFFF) { 
        
             info->total_values = 4096; 
        
             info->raw_bits = 12; 
        
             info->max_bits = 48; 
        
             info->maxDiff = 64;

I think, it is produced in high-speed continuous shooting mode, 14/12/8/whaever fps.

LebedevRI · 2017-06-24T16:32:46Z

I've seen you also implemented support for the GFX 50S. In the method
FujiDecompressor::copy_line_to_bayer
a static CFA is used. Maybe this should be converted into something
configurable like with the X-Trans sensors.

All that is very strange. If that is the actual code used in current libraw, then it may mean two things (both can be valid at the same time):

All cameras which support compressed raw, have the exact same CFA filter pattern
The CFA used in compression is static/the same for all cameras/does not necessarily match the actual camera's CFA pattern.

muessel · 2017-06-24T19:05:10Z

What i'm currently missing is some compressed .RAF with 12-bits data, so this branch is taken: https://github.com/darktable-org/rawspeed/blob/ fae7f40/src/librawspeed/decompressors/ FujiDecompressor.cpp#L106-L110 I think, it is produced in high-speed continuous shooting mode, 14/12/8/whaever fps.

I've tested it on my X-T2 with 8fps mechanical shutter and 11/14 fps electronical shutter. The .RAFs are all 14bit according to the code mentioned above.

LebedevRI · 2017-06-27T15:55:00Z

https://discuss.pixls.us/t/compressed-raf-help-needed/4527

Floessie · 2017-06-28T07:08:13Z

@muessel

Travis CI has problems with mac builds with std::abs. Do I have to change them to abs and then include "using std::abs;" ?

Uwe, just my 2¢ (I know Roman will fix or has already fixed it): Most often the abs() problem isn't about the namespace, but a change from the way this was defined for C++03 vs. C++11. There's an integer version in <cstdlib> and a float version in <cmath>. As the latter link notes:

Between C++11 and C++14, the standard erroneously required std::abs to have overloads for integer types returning double. This requirement was removed in C++17 by defect report 2735.

That causes trouble again and again, and that's most probably the case here also. The best solution usually is to use std::fabs() for floats and to make sure, you don't feed std::abs() with an unsigned.

HTH,
Flössie

LebedevRI · 2017-07-01T14:38:10Z

Update: still no sign of that elusive 12bit sample, and no reply from @LibRaw.
I'll wait a few more days and proceed without it...

Fujifilm compressed RAF support.

d37b332

LebedevRI requested changes Jun 23, 2017

View reviewed changes

LebedevRI changed the base branch from develop to raf-compressed June 23, 2017 20:01

muessel added 2 commits June 23, 2017 22:32

ushort fixes

8f95368

more suggested fixes

963f8e5

LebedevRI merged commit c3c4b7d into darktable-org:raf-compressed Jun 23, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fujifilm compressed RAF support. #107

Fujifilm compressed RAF support. #107

muessel commented Jun 23, 2017

LebedevRI commented Jun 23, 2017

LebedevRI left a comment

LebedevRI Jun 23, 2017

muessel Jun 23, 2017

LebedevRI Jun 23, 2017

LebedevRI Jun 23, 2017

muessel Jun 23, 2017

LebedevRI Jun 23, 2017

LebedevRI Jun 24, 2017

LebedevRI Jun 23, 2017

LebedevRI Jun 23, 2017

LebedevRI Jun 23, 2017

LebedevRI commented Jun 23, 2017

muessel commented Jun 23, 2017

LebedevRI commented Jun 23, 2017

muessel commented Jun 24, 2017 via email

LebedevRI commented Jun 24, 2017

LebedevRI commented Jun 24, 2017

muessel commented Jun 24, 2017 via email

LebedevRI commented Jun 27, 2017

Floessie commented Jun 28, 2017

LebedevRI commented Jul 1, 2017

Fujifilm compressed RAF support. #107

Fujifilm compressed RAF support. #107

Conversation

muessel commented Jun 23, 2017

LebedevRI commented Jun 23, 2017

LebedevRI left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LebedevRI commented Jun 23, 2017

muessel commented Jun 23, 2017

LebedevRI commented Jun 23, 2017

muessel commented Jun 24, 2017 via email

LebedevRI commented Jun 24, 2017

LebedevRI commented Jun 24, 2017

muessel commented Jun 24, 2017 via email

LebedevRI commented Jun 27, 2017

Floessie commented Jun 28, 2017

LebedevRI commented Jul 1, 2017