Segfault opening Fujifilm X-E2 firmware v4.0 RAF files

[andymcdonaldU2]

RawTherapee crashes when trying to open the edit window with a Fuji X-E2 RAF file. This problem is for files from a Fuji X-E2 with the v4.0 firmware applied. I have confirmed that RawTherapee does not crash when opening a Fuji X-E2 RAF file from and X-E2 with the previous (v3) firmware applied.

This was tested on Windows 10 64 bit on RawTherapee 4.2.699

[Beep6581]

Raw file needed http://filebin.net/

[andymcdonaldU2]

File is uploaded, let me know if you need any other information

[heckflosse]

The link to the file?

[andymcdonaldU2]

Sorry, my bad - it is here: http://filebin.net/5f0vykiy6q/DSCF9049.RAF

[Beep6581]

Have you confirmed this problem with all v4.0 RAF files, to make sure its not just one corrupt file?

[andymcdonaldU2]

Yes I have. Also. someone else on DPReview has seen it too.

[heckflosse]

Duplicate of #3110 Different cam, but exactly same crash.

[Beep6581]

@andymcdonaldU2 can you check in your camera whether the v4.0 firmware default to compressed raw files? Can you try a v4.0 RAF with compression disabled, if possible?

[andymcdonaldU2]

@Beep6581 have checked the camera menus and the new operators manual and neither mention being able to choose whether to have a compressed raw file or not, so I don't believe that this is something that I can change.

[sguyader]

I haven't seen any word about compressed raw files in the official announcement for new the v4.0 firmware.

[arangast]

I'm the other person who posted on the DPR forum. The X-E2 doesn't support compressed RAW files, in addition to this other non-dcraw based editors can still open them (including a 2 year old copy of Silkypix).

The crash is occurring in dcraw within crop_masked_pixels() due to a dodgy looking top_margin value which in turn is derived from a dodgy looking height member. From what little I could tell (not the easiest code to read from scratch) it appears the header format has changed and dcraw is parsing it incorrectly.

I compiled parse.h also on Dave Coffin's website, and the broken raw header has quite a few different sections and some tags (such as 0x1020 - Macro) have been removed. I've emailed him some details and am awaiting a response.

For your reference, here's another non-working RAF:
https://www.dropbox.com/s/kzi1o44gh0t5s1j/20160204-7859.raf?dl=0

...and a working RAF taken before the update:
https://www.dropbox.com/s/x3l9j0wnb0kqavl/20160202-7829.raf?dl=0

[heckflosse]

'not the easiest code to read from scratch': That's true!

[iliasg]

Ingo, I 've sent you an email about x-pro2, have you received any ?

t looks like the reason RT (and Dcraw) crash/freeze is wrong reading of exif (sensor size ?).
I see in exiftool that Fuji x-pro2 has changed the address for sensor width/height and so dcraw reads a crazy size and RT crashes (Dcraw freeze)
(the previous exiftool 10.05 reports wrong numbers also)

latest exiftool 10.10 reports at -RAF- group ..
X100T width at 0x0004 and height at 0x0008
X-PRO2 width at 0x0008 and height at 0x000c
The sizes are: full sensor 6160X4032 and the raw crop 6032X4032

I tried with camconst the following

{ // Quality B
    "make_model": "FUJIFILM X-PRO2",
    "dcraw_matrix": [ 11434,-4948,-1210,-3746,12042,1903,-666,1479,5235 ], // DNG_v9.4 D65
    "raw_crop": [ 0, 0, 6032, 4032 ], // full raw 6160,4032, Usable 6032,4032 - experimental crop
    "ranges": { "white": 16100 }
},

and with this RT stops crashing but still does not decode correctly (just a zone of garbage is displayed). I think that if we had a way to overwrite (a new tag in camconst.json ?) what Dcraw reads as size I would be able to "support" the uncompressed mode.

[heckflosse]

@iliasg I was spammed a bit last time, so maybe I missed your email. I'll take a look tomorrow.
Do you think, Dave will make a new dcraw version soon? In this case I would like to wait. But I don't want to wait too long, since we are already one patch behind latest dcraw version

[iliasg]

I cannot communicate with Dave :(. The usual Dcraw release at Dec-Jan is already very delayed .. who knows.

BTW we need to add raw crop for X-E2 also to stop the crash, although it still decodes faulty.
{ // Quality B
"make_model": [ "FUJIFILM X-T1", "FUJIFILM X-T10", "FUJIFILM X-E2" ],
"dcraw_matrix": [ 8458,-2451,-855,-4597,12447,2407,-1475,2482,6526 ], // DNG D65
// "dcraw_matrix": [ 9289,-3279,-632,-3539,11137,2758,-1049,1950,6544 ], // X-RITE D55
"raw_crop": [ 0, 0, 4952, 3288 ], // full raw 4992,3296, Official crop 4936,3296
"ranges": { "white": 16100 }
},

[heckflosse]

@iliasg I agree that avoiding the crash is first priority! DrSlony and Hombre seem to able to communicate with Dave. Last time I tried I failed :( Maybe we should try to communicate this stuff to Dave via DrSlony or Hombre?

Ingo

[arangast]

Hello again, I have received a patch from Dave. The RAF tag had indeed changed and the address that once contained the width and height now contained a series of characters.

The fix involves replacing the line:
if ((tag = get4()) > 10000) tag = get4();

...with:
while ((tag = get4()) > 10000);

In the RT copy of dcraw this is line 6661 of dcraw.c and line 6685 of dcraw.cc.

I've built RT on windows and it works. I also successfully opened a sample X-Pro2 RAF.

Can someone else try it out?

Cheers!

P.S, here's the full email from Dave:

I haven't figured out Fuji's 0xc000 tag.  In the older file
it's (using my parse.c program):

Fuji tag=0xc000, len=21984, data = 00 00 4f 02 48 13 00 00 e0 0c 00 00 ...

     0x00001348 is the width (4936 decimal) and 0x00000ce0 is
the height (3296), in little-endian byte order.  With the new
firmware it looks like:

Fuji tag=0xc000, len=21984, data = 00 00 56 03 57 52 54 53 48 13 00 00 e0 0c 00 00

     The extra four bytes "57 52 54 53" spell out the letters
"WRTS".  Not sure what that stands for.  I also see WRNEDCTS,
DSCF, DCNETSTS, TSNERDTS, RDNESQTS, SQNERSTS, RSNECPTS, and
CPNEVFTSVFNE, none of which appear with the old firmware.

     Anyway, this patch finds the width and height until the
next firmware update:

--- dcraw.c     2015/05/25 02:29:14     1.476
+++ dcraw.c     2016/02/10 21:28:34
@@ -6721,7 +6721,7 @@
     } else if (tag == 0xc000) {
       c = order;
       order = 0x4949;
-      if ((tag = get4()) > 10000) tag = get4();
+      while ((tag = get4()) > 10000);
       width = tag;
       height = get4();
       order = c;

[heckflosse]

I'll try it