Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Cannot use camera white balance" error in console when using Sony files and RT compiled with gcc 4.8.x #2105

Closed
Beep6581 opened this issue Aug 11, 2015 · 42 comments
Closed

"Cannot use camera white balance" error in console when using Sony files and RT compiled with gcc 4.8.x #2105

Beep6581 opened this issue Aug 11, 2015 · 42 comments
Assignees
@Beep6581

Comments

@Beep6581
Copy link
Owner

Originally reported on Google Code with ID 2121

Some users have reported that some images display in incorrect colors, probably incorrect
white balance.
When asked to run RT from a console (Linux/OS X) or to set "verbose=true" in the "options"
file (Windows), they report seeing "Cannot use camera white balance" when opening the
raw file.
For a step-by-step of what's been tried so far, see https://code.google.com/p/rawtherapee/issues/detail?id=1636#c13
I can't reproduce, two users so far can.
I use exiv2 0.23 and so does one of the users who can reproduce (vamp898).

Vamp reported that this was caused by compiling RT using -march=bdver1 and -march=native
on his AMD FX 8150, see https://code.google.com/p/rawtherapee/issues/detail?id=1980

To summarize:
- cache has been emptied
- Neutral applied
- Input profile set to (Camera)
- RT build using buildRT and "native" proctarget

Reproduced by those affected with Sony SLT-A99V and A77.
Get raw files from:
1- http://www.ignaz.org/DSC05481.ARW (mirror: http://filebin.net/6bywnx676p )
2- http://www.rawtherapee.com/shared/play/playraw38/playraw38_tim.ARW

Reported by entertheyoni on 2013-12-12 22:38:14
@Beep6581
Copy link
Owner Author

Reported by entertheyoni on 2013-12-12 22:44:43

@Beep6581
Copy link
Owner Author 

Other users who can reproduce are:
- gio http://rawtherapee.com/forum/memberlist.php?mode=viewprofile&u=3844
- Simon Cropper http://rawtherapee.com/forum/memberlist.php?mode=viewprofile&u=4614
Forum bug report: http://rawtherapee.com/forum/viewtopic.php?f=3&t=5205

Reported by entertheyoni on 2013-12-12 22:53:37

@Beep6581 Beep6581 self-assigned this Aug 11, 2015
@Beep6581
Copy link
Owner Author 

I'm on Win7/64, but I also use a Bulldozer (In my case AMD FX 8350), compile with -march=native
and run RT from console (for this test even in verbose-mode), but opening DSC05481.ARW
shows no "Cannot use camera white balance error"

It shows:

Loading Sony SLT-A99V image from T:\Amazetestnew\DSC05481.ARW...
constants exists for "Sony SLT-A99V" in camconst.json
black levels: R:512 G1:512 B:512 G2:512 (provided by camconst.json)
white levels: R:16350 G1:16350 B:16350 G2:16350 (provided by camconst.json)
color matrix provided by camconst.json
cam_mul:[2432.000000 1024.000000 1464.000000 1024.000000], AsShotNeutral:[0.421053
1.000000 0.699454 1.000000]
pre_mul:[1.000000 0.421053 0.601974 0.421053], scale_mul:[4.137833 1.742246 2.490867
1.742246], cblack:[512.000000 512.000000 512.000000 512.000000]
rgb_cam:[ [ 1.807031 1.646390 1.432531], [1.807031 1.646390 1.432531], [1.807031 1.646390
1.432531] ]
cam_mul:[2432.000000 1024.000000 1464.000000 1024.000000], AsShotNeutral:[0.421053
1.000000 0.699454 1.000000]
pre_mul:[1.000000 0.449328 0.703304 0.447880], scale_mul:[4.137833 1.859243 2.910154
1.853253], cblack:[512.000000 512.000000 512.000000 512.000000]
rgb_cam:[ [ 1.807031 1.646390 1.432531], [1.807031 1.646390 1.432531], [1.807031 1.646390
1.432531] ]
Raw As Shot White balance: temp 5124.561310, tint 1.074380
Raw Reference (auto) white balance: temp 4585.407257, tint 1.063273, multipliers [3.006932
2.167818 1.511414 | 2.554482 1.677678 2.065688]
Load T:\Amazetestnew\DSC05481.ARW: 574039 µsec

Ingo

Reported by heckflosse@i-weyrich.de on 2013-12-12 23:05:01

@Beep6581
Copy link
Owner Author 

The details of my build is as follows...

Branch: default
Version: 4.0.11.181
Changeset: dc9b6fded698
Compiler: cc 4.8.1
Processor: undefined
System: Linux
Bit depth: 64 bits
Gtkmm: V2.24.4
Build type: Release
Build flags:  -Wno-unused-result -march=native -fopenmp -O3 -DNDEBUG
Link flags:   -march=native
OpenMP support: ON
MMAP support: ON

Simon Cropper

Reported by simonc462 on 2013-12-12 23:21:27

@Beep6581
Copy link
Owner Author 

Hi Simon,

could you post the console-output when opening DSC05481.ARW in verbose-mode as I did
above? I would like to see whether there are differences.

Thanks, Ingo

here's my build info:

Branch: default
Version: 4.0.11.184
Changeset: d1ce9789f20d
Compiler: gcc 4.7.1
Processor: undefined
System: Windows
Bit depth: 64 bits
Gtkmm: V2.22.0
Build type: Release
Build flags:  -march=native -fopenmp -O3 -DNDEBUG
Link flags:   -march=native
OpenMP support: ON
MMAP support: ON

Reported by heckflosse@i-weyrich.de on 2013-12-12 23:33:37

@Beep6581
Copy link
Owner Author 

Hi heckflo...@i-weyrich.de

How do I get verbose mode on Linux? Once you respond I will run the test.

The output from the console is...

rawtherapee DSC05481.ARW
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
(rawtherapee:7618): Gtk-WARNING **: Unable to locate theme engine in module_path: "clearlooks",
Cannot use camera white balance.

The short version however is that that file actually throws the same error.

Reported by simonc462 on 2013-12-13 00:09:21

@Beep6581
Copy link
Owner Author 

Open your options-file in Editor. There's a Section at the top named 'General' with
an entry 'Verbose=False'. Close RT, Open the options-File, change the setting to 'Verbose=true',
save the options-file and start RT. Don't know exactly where it is on your machine.
Maybe you have more than one. Executing 'cd /home' followed by 'find -name options'
should list them, if there are more than one.

Ingo

Reported by heckflosse@i-weyrich.de on 2013-12-13 00:24:42

@Beep6581
Copy link
Owner Author 

Hi,

Reading the discussion about this error message it appears that people are unclear
where the error message originates.

The error appears to come from the DCRAW library (line 3879).

This lead me to David Coffins site and the list of supported cameras and only the Sony
SLT-A99V is currently supported. The Sony A77 is not on the list. When upgrading my
camera recently it took several months for updated library to propagate through the
various servers and distros and builds out there.

Maybe this is what is happening.

The dcraw in my build is Revision: 1.456 Dated: 2013/06/16 18:01:08 (the latest version).
By-and-by, maybe this should also be included in the AboutThisBuild.txt. Maybe different
builds use different dcraw versions -- some that don't support the Sony cameras. Although
this theory does not take into account that I can not get the files to work properly.

The simple fact though that this error is being reported in other open source forums
and that I get funny results using other RAW-aware software (digiKam/showPhoto/DarkTable/Rawstudio/UFRaw)
or the files just plain don't open (Darkroom). This points to an underlying library
not working properly rather than something RawTherapee specific.

Reported by simonc462 on 2013-12-13 00:46:22

@Beep6581
Copy link
Owner Author 

Output from both files being investigated attached.

Reported by simonc462 on 2013-12-13 00:55:36

- _Attachment: [DSC05481.txt](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2121/comment-9/DSC05481.txt)_ - _Attachment: [playraw38_tim.txt](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2121/comment-9/playraw38_tim.txt)_

@Beep6581
Copy link
Owner Author 

I cannot reproduce, here's my verbose output:
(...)
Loading Sony SLT-A99V image from /tmp/DSC05481.ARW...
constants exists for "Sony SLT-A99V" in camconst.json
black levels: R:512 G1:512 B:512 G2:512 (provided by camconst.json)
white levels: R:16350 G1:16350 B:16350 G2:16350 (provided by camconst.json)
color matrix provided by camconst.json
cam_mul:[2432.000000 1024.000000 1464.000000 1024.000000], AsShotNeutral:[0.421053
1.000000 0.699454 1.000000]
pre_mul:[1.000000 0.421053 0.601974 0.421053], scale_mul:[4.137833 1.742246 2.490867
1.742246], cblack:[512.000000 512.000000 512.000000 512.000000]
rgb_cam:[ [ 1.807031 1.646390 1.432531], [1.807031 1.646390 1.432531], [1.807031 1.646390
1.432531] ]
cam_mul:[2432.000000 1024.000000 1464.000000 1024.000000], AsShotNeutral:[0.421053
1.000000 0.699454 1.000000]
pre_mul:[1.000000 0.449328 0.703304 0.447880], scale_mul:[4.137833 1.859243 2.910154
1.853253], cblack:[512.000000 512.000000 512.000000 512.000000]
rgb_cam:[ [ 1.807031 1.646390 1.432531], [1.807031 1.646390 1.432531], [1.807031 1.646390
1.432531] ]
Raw As Shot White balance: temp 5124.561310, tint 1.074380
Raw Reference (auto) white balance: temp 4585.407257, tint 1.063273, multipliers [3.006932
2.167818 1.511414 | 2.554482 1.677678 2.065688]
Load /tmp/DSC05481.ARW: 716421 µsec
(...)

x86_64 Intel(R) Core(TM) i7 CPU Q 820 @ 1.73GHz GenuineIntel GNU/Linux
Branch: default
Version: 4.0.11.185
Changeset: 2971fb10c155
Compiler: cc 4.7.3
Processor: undefined
System: Linux
Bit depth: 64 bits
Gtkmm: V2.24.4
Build type: Release
Build flags:  -Wno-unused-result -Wno-aggressive-loop-optimizations -march=native -fopenmp
-O3 -DNDEBUG
Link flags:   -march=native
OpenMP support: ON
MMAP support: ON

Reported by entertheyoni on 2013-12-13 09:24:31

@Beep6581
Copy link
Owner Author 

That's a very strange error! I don't have any problem neither. I'm using the latest
changeset of RT, and therefore dcraw 9.19 (Revision: 1.456 Dated: 2013/06/16 18:01:08)
like simonc462. What we can see in the files from comment #9 is that the cam_mul[]
values are wrong.

I don't know how 2 identical RT version can lead to different cam_mul values when reading
the same file!? Or maybe endianess problem?It looks like i86 proc can read the files
correctly. What is the processor of those who have problem?

Anyway, The error message come from RT, based on values set in dcraw, so dcraw should
be where to look for the bug, but it's too cryptic for me!

Reported by natureh.510 on 2013-12-13 11:48:58

@Beep6581
Copy link
Owner Author 

How can enable console messages with 4.0.11.141

http://rawtherapee.com/forum/viewtopic.php?f=1&t=4377&start=165#p34275

Reported by iliasgiarimis on 2013-12-13 12:16:51

@Beep6581
Copy link
Owner Author 

re #12: You can start rt from console and let it write the output to a file.

rawtherapee.exe > verbose.log

Perhaps you have to start cmd.exe as Administrator to allow the verbose.log to be written.

Ingo

Reported by heckflosse@i-weyrich.de on 2013-12-13 12:34:57

@Beep6581
Copy link
Owner Author 

Re #12: see #7 :)

Reported by natureh.510 on 2013-12-13 13:03:05

@Beep6581
Copy link
Owner Author 

Thanks Ingo .. now I can :) :)

Reported by iliasgiarimis on 2013-12-13 14:14:37

@Beep6581
Copy link
Owner Author 

This is some toolchain-related issue - on debian stable it works correct - 
https://www.dropbox.com/sh/ompeik0ws8i2muo/6SGR1VHvO_

Reported by leitner.f.johannes on 2013-12-13 19:34:21

@Beep6581
Copy link
Owner Author 

https://www.dropbox.com/sh/ompeik0ws8i2muo/bixmaabSv0/rt.txt
"white levels: R:17204 G1:17204 B:17204 G2:17204 (provided by dcraw)"

This cannot be correct ..

Reported by iliasgiarimis on 2013-12-14 04:28:34

@Beep6581
Copy link
Owner Author 

I guess the way we solved Issue 2056 was not correct.

Simon, can you try to compile with -fno-aggressive-loop-optimizations ?

Ingo

Reported by heckflosse@i-weyrich.de on 2013-12-14 10:46:30

@Beep6581
Copy link
Owner Author 

Ingo, about the wrong WL at #18 (they WL cannot be > 2^14-1)it happens with 4.0.11.141
which does not include the Issue2056 fix.
It's not related to this (2121) Issue also, so I suppose I'll have to open a new one
..

Reported by iliasgiarimis on 2013-12-14 16:06:59

@Beep6581
Copy link
Owner Author 

ilias: issue 2056 has no fix. It was said that the warnings were safe to ignore, so
I just made buildRT ignore them by default. -W changes which warnings are reported,
as far as I know it doesn't affect the code. -f does.

"Before reporting that GCC compiles your code incorrectly, compile it with gcc -Wall
-Wextra and see whether this shows anything wrong with your code. Similarly, if compiling
with -fno-strict-aliasing -fwrapv -fno-aggressive-loop-optimizations makes a difference,
your code probably is not correct."

Reported by entertheyoni on 2013-12-14 16:47:51

@Beep6581
Copy link
Owner Author 

DrSlony, I couldn't have said it better. The users which have this Issue all compiled
with gcc 4.8.x IIRC and  Issue 2056 is also about gcc 4.8.x and dcraw.cc. So there
may be a relation.

Ilias, I guess your RT version isn't compiled using gcc 4.8.x, so you don't get the
'Cannot use camera white balance' in verbose mode and you should get good results at
least with http://www.ignaz.org/DSC05481.ARW. And yes, the WB problem you mention in
#18 is a different Issue (please open it).

Ingo

Reported by heckflosse@i-weyrich.de on 2013-12-14 17:29:11

@Beep6581
Copy link
Owner Author 

I have no problem with current issue, I use gaaned92's builds (made with gcc 4.6.1)

Already opened the new Issue 2109 ..

Reported by iliasgiarimis on 2013-12-14 17:37:43

@Beep6581
Copy link
Owner Author 

Issue seems to have gone stale. Please verify whether this is still a problem with today's
code, otherwise I'll remove it from the 4.1 blocker list. Do we blame a specific GCC
version, specific optimization flags, or dcraw?

Reported by entertheyoni on 2013-12-30 10:48:59

@Beep6581
Copy link
Owner Author 

It's still in today's code.
Origin is in dcraw-code. But it only occurs with gcc 4.8.x where the aggressive-loop-optimizations
seems to be on by default. It has to be disabled with -fno-aggressive-loop-optimizations
for gcc 4.8.x at the moment.
Modifying dcraw.cc to fix this error is also possible.

Ingo

Reported by heckflosse@i-weyrich.de on 2013-12-30 11:05:33

@Beep6581
Copy link
Owner Author 

The same issue is happening for me with my ARW files (A77). In debug build of changeset
9f7627adfddb, on Ubuntu 13.10 x64 (gcc 4.8.1).

Reported by Eugene.Andreeshchev on 2013-12-31 03:42:10

@Beep6581
Copy link
Owner Author 

Shit. I tried 4.8.2 with -fno-aggressive-loop-optimizations in -DCMAKE_C_FLAGS and -DCMAKE_CXX_FLAGS,
AboutThisBuild shows "Build flags:  -fno-aggressive-loop-optimizations -Wno-unused-result
-Wno-aggressive-loop-optimizations -march=native -fopenmp -O3 -DNDEBUG" but the build
still prints "Cannot use camera white balance." and indeed does not show the correct
wb.
Gentoo Linux 64-bit, using buildRT.

Reported by entertheyoni on 2013-12-31 15:22:15

@Beep6581
Copy link
Owner Author 

Found the culprit. I'll make the changes to dcraw.patch when this one is tested,

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-01 16:32:59

  • Status changed: PatchSubmitted
- _Attachment: [issue2121_00.patch](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2121/comment-30/issue2121_00.patch)_

@Beep6581
Copy link
Owner Author 

Fix confirmed :)

Reported by entertheyoni on 2014-01-01 18:36:52

@Beep6581
Copy link
Owner Author 

*Fix confirmed using GCC-4.8.2 64-bit without -fno-aggressive-loop-optimizations

Reported by entertheyoni on 2014-01-01 18:37:43

@Beep6581
Copy link
Owner Author 

Fine :-) Here's the patch with the changes to the dcraw.patch file and with some LF-cleanup
to dcraw.cc.

Reported by heckflosse@i-weyrich.de on 2014-01-01 18:44:58

- _Attachment: [issue2121_01.patch](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2121/comment-33/issue2121_01.patch)_

@Beep6581
Copy link
Owner Author 

Addition: I made and tested the patch using gcc 4.8.1, so I can confirm that it's also
working with this version

Reported by heckflosse@i-weyrich.de on 2014-01-01 18:54:08

@Beep6581
Copy link
Owner Author

Reported by heckflosse@i-weyrich.de on 2014-01-01 19:28:16

@Beep6581
Copy link
Owner Author 

I can test the new patch in a few moments, just running RT through valgrind at the moment.
Thank you for the summary correction, this makes things much easier e.g. when writing
RELEASE_NOTES months from now.

Reported by entertheyoni on 2014-01-01 19:38:14

@Beep6581
Copy link
Owner Author 

issue2121_01.patch patch doesn't change functionality compared to issue2121_00.patch.
It only includes the additional changes to dcraw.patch which are required when a new
dcraw version gets released and some corrections to Line-endings. Would be glad to
get a review from Hombre or Michael whether the changes to dcraw.patch are correct.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-01 19:45:30

@Beep6581
Copy link
Owner Author 

I patched vanilla dcraw.c without problems. Still works fine after compilation.

Reported by entertheyoni on 2014-01-01 20:13:50

@Beep6581
Copy link
Owner Author 

DrSlony, thanks for testing. Will commit ASAP as it's save I think.

Reported by heckflosse@i-weyrich.de on 2014-01-01 20:23:52

@Beep6581
Copy link
Owner Author 

Yes, it should be safe.

Reported by natureh.510 on 2014-01-01 22:06:06

@Beep6581
Copy link
Owner Author 

Hombre, thank you! Committed to default.

Reported by heckflosse@i-weyrich.de on 2014-01-01 22:08:49

  • Status changed: FixedPendingConfirmation

@Beep6581
Copy link
Owner Author 

Hi Ingo, thanks for the patch. Sorry I only now got to my workstation.
Happy new year:)

Reported by michaelezra000 on 2014-01-02 16:37:22

@Beep6581
Copy link
Owner Author 

Hi Michael, no problem. And of course Happy new year :-)

Reported by heckflosse@i-weyrich.de on 2014-01-02 17:26:30

@Beep6581
Copy link
Owner Author

Reported by heckflosse@i-weyrich.de on 2014-01-04 15:01:43

  • Status changed: Fixed

@Beep6581
Copy link
Owner Author 

Works for me with the latest RawTherapee HG, also new Noise Reduction is awesome =)
:D

Reported by vamp898@web.de on 2014-01-17 21:08:48

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Beep6581 Beep6581

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Beep6581