Add camconst support for foveon files

[Beep6581]

Originally reported on Google Code with ID 2729

This patch adds camconst support for foveon files.
Ilias, could you add the camconst entries for the foveon cams please?

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-30 17:13:41

---

- _Attachment: [camconst_foveon.patch](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2729/comment-0/camconst_foveon.patch)_

[Beep6581]

Last patch had a bug in thumbnail generation when a raw crop was defined in camconst.json
for a foveon cam. Fixed.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-30 19:04:38

---

- _Attachment: [issue2729_01.patch](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2729/comment-1/issue2729_01.patch)_

[Beep6581]

Ingo can you confirm if dcraw.cc reads/uses the color matrix from the exif data when
we do not define one in camconst.json ?.

I see abnormal temp/tint and this is usually an indication of a wrong (or not existing)
color matrix.

Reported by iliasgiarimis on 2015-03-30 20:57:04

[Beep6581]

Ilias, afaik it uses this one: 

dcraw.cc line 7724

  /* index 0 -- all Foveon cameras */
  { 1.4032,-0.2231,-0.1016,-0.5263,1.4816,0.017,-0.0112,0.0183,0.9113 },

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-30 21:02:52

[Beep6581]

Ilias, confirmed #3 by looking at verbose output:

rgb_cam:[ [ 1.403200 -0.526300 -0.011200], [-0.223100 1.481600 0.018300], [-0.101600
0.017000 0.911300] ] (not bayer)

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-30 21:38:55

[Beep6581]

Ilias, I had a look at the dcraw foveon code. It seems we have to call the foveon_interpolate
function too. But there the problems begin. It doesn't find "WhiteBalanceIlluminants"
for SD1 Merrill and then stops the function, leaving the image unchanged. I found this:
http://www.dpreview.com/forums/post/41138764
Perhaps I take some time to dig into that stuff...

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-31 17:59:54

[Beep6581]

Hi Ingo, if we want RT to become a reliable foveon converter there is much to do ..


May be digging in related code http://www.proxel.se/x3f.html will help ..  it's open
source but under "BSD lookalike license"

For a properly decoded foveon data some cryptic exif info must be parsed and decoded.
Ronald Karlsson (the author of x3f tools) suggested to read the side optically black
data to calculate the BlackLevel http://www.dpreview.com/forums/post/55485828
although I cannot find the proper area where we can read the correct value .. 

For the moment I stole the correct BL=17 and built a better icm profile adapted to
this BL .. 
https://drive.google.com/file/d/0B0NqktTgc54sa0JzWlVYVjN0QTA/view?usp=sharing

Some black frames from foveon users will help ..

Is the "save reference image for profiling" aborting the color matrix at #3 ??. Or
is it used and then my profile is a correction profile upon the matrix ?.

I also saw Dcraw aborting a part of the code .. as I understand by the comments it
is about color interpolation .. so I just found a workaround by using RT's .. color
denoise. It works acceptably :) 

Reported by iliasgiarimis on 2015-03-31 20:08:34

[Beep6581]

Hi Ilias, I think we already made a good first step now. As you say there is much to
do, but it hasn't to be done immediately. With your new icm profile it already looks
'good' for SD1 Merrill. Perhaps users in forum can supply black frames too.

About 'reference image for profiling' I have to look at the code. Hopefully I can answer
your question tomorrow.

Shall I commit the foveon camconst patch or do you want to add some black levels and
crops to camconst.json?

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-31 20:37:49

[Beep6581]

Ingo,

Wait some hours for camconst.json .. tonight I will add black levels and raw crops
for as many foveon models I can find samples of.

Now the only serious defect is the WB which falls out of range .. maybe we should try
with inverting the matrix #3 ?? ..
I had the hope that copying the matrix from the icm profiles in camconst.json (see
last comments by Anders .. https://code.google.com/p/rawtherapee/source/detail?r=0bdc98a3d57ee55dc0ea044ab73a2fecd1257dd4
)
would do the job but failed .. maybe an matrix inversion is needed ?. 

Reported by iliasgiarimis on 2015-03-31 20:52:00

[Beep6581]

Ilias, no hurry to commit. Take your time to make the changes.
I'll have a look at the matrix now.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-31 21:09:06

[Beep6581]

Ilias, I tried the inverted matrix in dcraw code. The auto wb value changed from about
49000 to 24000, but when using auto wb the output was identical, independent on matrix!
When using fixed wb, the output was different of course.
So I guess, if we can build a correct matrix, the auto wb value will also be in normal
range.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-31 22:09:41

[Beep6581]

OK I lost my time with this alien WB :( .. But I came to a dirty solution .. :-)

The problem is that the WB algo is not tuned for layered sensors where the possibility
of a photon to get registered at the deepest level (red color) decreases exponentially.
So for a neutral grey surface under daylight we take R=0.5, G=1.0, B=2.0 instead of
a usual bayer which gives R=2.5 G=1.0 B=1.5. This is a extreme 5X difference for R
.. These abnormal multipliers are the reason that the WB algo get crazy ..

The solution is to insert a positive bias for R and G to adapt the values to the WB
algorithm's operative range. This is exactly what we can do for R with the "red/blue
equalizer" slider when we push to the right. But still the bias we need for G is missing.

The dirty solution I came with is to multiply by 1.5 the G and by 1.5*1.5=2.25 the
R when calculating a matrix from the icm profile at #6
( mutiply the icm's matrix with [ 2.25, 1.5, 1.0 ] then bradford adapt D50 to D65,
then invert ..). Use the result in camconst.json. Of course this matrix is only usable
for WB calculations while the color rendering it gives is terrible. But we have the
icm profile for correct colors :)

Please try ..

    { // Quality C, frame corrections, color matrix only usable for WB calculations
..
        "make_model": "Sigma SD1 Merrill",
        "dcraw_matrix": [ 15440,-7911,-3450,3593,3228,-512,-3726,9191,5084 ], // experimental,
WB-preconditioned, inverted icm's matrix
      //  "dcraw_matrix": [ 14032,-2231,-1016,-5263,14816,170,-112,183,9113 ], // 
hardcoded internal
        "ranges": { "black": 17, "white": 4070 },
        "raw_crop": [ 12, 52, -110, -8 ]
    },  

Reported by iliasgiarimis on 2015-04-01 06:25:50