Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hasselblad X2D 100C support #463

Merged
merged 1 commit into from May 3, 2023
Merged

Conversation

kmilos
Copy link
Contributor

@kmilos kmilos commented Apr 28, 2023

Addresses darktable-org/darktable#14345

Stub only (using ADC 15.3), levels and crop need to be verified.

Depends on #466 for OOC 3FR uncompressed support.

@kmilos kmilos requested a review from LebedevRI as a code owner April 28, 2023 07:42
@kmilos
Copy link
Contributor Author

kmilos commented Apr 28, 2023

Note also that the 3FR has these tags (does unique ID matter?):

UniqueCameraModel               : Hasselblad X2D 100C
ColorMatrix1                    : 0.5950573751 -0.146674752 -0.03412952103 -0.5159296309 1.26306183 0.2826110949 -0.103825823 0.1675577691 0.6152086

but no idea what that matrix is calibrated for... I presume older Hasselblad models also use the Adobe one instead of the 3FR embedded one? (They also seem to use the Adobe unique ID instead of the 3FR one... And this doesn't seem to matter as canonical_id is not used by dt currently...)

@codecov
Copy link

codecov bot commented May 1, 2023

Codecov Report

Patch coverage: 17.39% and project coverage change: -0.08 ⚠️

Comparison is base (915f2c0) 59.26% compared to head (6eda23b) 59.19%.

❗ Current head 6eda23b differs from pull request most recent head a51941d. Consider uploading reports for the commit a51941d to get more accurate results

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #463      +/-   ##
===========================================
- Coverage    59.26%   59.19%   -0.08%     
===========================================
  Files          231      231              
  Lines        13766    13786      +20     
  Branches      1917     1921       +4     
===========================================
+ Hits          8159     8160       +1     
- Misses        5473     5491      +18     
- Partials       134      135       +1     
Flag Coverage Δ
benchmarks 8.30% <0.00%> (-0.02%) ⬇️
integration 47.47% <18.18%> (-0.07%) ⬇️
linux 56.98% <18.18%> (-0.08%) ⬇️
macOS 18.96% <0.00%> (-0.04%) ⬇️
rpu_u 47.47% <18.18%> (-0.07%) ⬇️
unittests 18.33% <0.00%> (-0.03%) ⬇️
windows ∅ <ø> (∅)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/librawspeed/decoders/ThreefrDecoder.h 100.00% <ø> (ø)
src/librawspeed/decoders/ThreefrDecoder.cpp 50.00% <9.52%> (-26.48%) ⬇️
...brawspeed/decompressors/HasselbladDecompressor.cpp 74.35% <100.00%> (ø)
...brawspeed/decompressors/HasselbladLJpegDecoder.cpp 75.00% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@kkempter-getswift
Copy link

I pulled an image from the X2d into lightroom and then exported it t a DNG file.

Here is the BlackLevel info from the resulting DNG file:

$ exiv2 -pt B0000050.dng| grep BlackLevel
Exif.SubImage1.BlackLevel Rational 1 4267/1
Exif.SubImage3.BlackLevel Rational 3 51/1 51/1 51/1
Exif.SubImage4.BlackLevel Rational 3 51/1 51/1 51/1
Exif.SubImage5.BlackLevel Rational 3 51/1 51/1 51/1

I'm not sure what to do with this, how/where to update the xml file so I just added it here in case it helps

@kkempter-getswift
Copy link

and here are the color matrix info:

$ exiv2 -pt B0000050.dng| grep CalibrationIlluminant2
Exif.Image.CalibrationIlluminant2 Short 1 D65

$ exiv2 -pt B0000050.dng| grep ColorMatrix2
Exif.Image.ColorMatrix2 SRational 9 6468/10000 -1899/10000 -545/10000 -4526/10000 12267/10000 2542/10000 -388/10000 1276/10000 6096/10000

from the same DNG file (see my comment above)

@LebedevRI
Copy link
Member

$ exiv2 -pt B0000050.dng| grep CalibrationIlluminant2 Exif.Image.CalibrationIlluminant2 Short 1 D65

$ exiv2 -pt B0000050.dng| grep ColorMatrix2 Exif.Image.ColorMatrix2 SRational 9 6468/10000 -1899/10000 -545/10000 -4526/10000 12267/10000 2542/10000 -388/10000 1276/10000 6096/10000

So we can infer/guess that at least for that camera the camera-provided color matrix is D65, i guess.

@kmilos
Copy link
Contributor Author

kmilos commented May 1, 2023

@LebedevRI Looks like we'll also need to bump up the accepted size a bit: https://discuss.pixls.us/t/looking-for-workflow-advice-with-3fr-files-from-a-hasselblad-x2d/36897/27

H6D we already had on RPU is 12000 x 8816, new X2D is 11904 x 8842

@kofa73
Copy link

kofa73 commented May 1, 2023

With darktable master + your PR:

kofa@eagle:~/darktable/src/external/rawspeed$ git status
On branch hassy_x2d
nothing to commit, working tree clean
kofa@eagle:~/darktable/src/external/rawspeed$ git rev-parse HEAD
7fd3ca0e0d7d6fa1e2a504d9b94a014861a7fd9f

kofa@eagle:~/darktable$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   src/external/rawspeed (new commits)

no changes added to commit (use "git add" and/or "git commit -a")
kofa@eagle:~/darktable$ git rev-parse HEAD
aaf481c2aee61b4c2b9eca90ea95a15787e31d3e

Opening the raw from https://nextcloud.phxicg.com/s/9eGHMdPc8Q6jTEa, I get:

92.6376 [rawspeed] (B0000050.3FR) rawspeed::JpegMarker rawspeed::AbstractLJpegDecoder::getNextMarker(bool), line 299: (Noskip) Expected marker not found. Probably corrupt file.
92.6376 [libraw_open] extensions whitelist: `cr3'
92.6377 [dt_dev_load_raw] loading the image. took 0.404 secs (0.000 CPU)
92.9398 [temperature] failed to read camera white balance information from `B0000050.3FR'!
92.9482 Trouble: [lens correction] camera/lens not found (B0000050.3FR 24873)
92.9542 [colorin] could not find requested profile `standard color matrix'!
93.0787 [histogram] took 0.000 secs (0.000 CPU) scope draw

@kmilos
Copy link
Contributor Author

kmilos commented May 1, 2023

That is pretty weird as the RPU sample is uncompressed.

@kkempter-getswift Does the camera support both uncompressed and compressed raw output? Please upload all possible outputs to RPU, as requested there.

@LebedevRI
Copy link
Member

92.6376 [rawspeed] (B0000050.3FR) rawspeed::JpegMarker rawspeed::AbstractLJpegDecoder::getNextMarker(bool), line 299: (Noskip) Expected marker not found. Probably corrupt file.

I haven't checked, but that sounds like #144.

@kkempter-getswift
Copy link

Hi everyone

I could use some help. I am a DBA and less of a developer, I use git & friends but minimally and even then I am quite rusty.

I was the one who uploaded the .3FR files

I downloaded 4.2.1 source code from https://www.darktable.org/install/ and compiled on my local Fedora laptop.

Now I want to grab the source code that would include this PR and re-test. Can someone help me with how to grab the right code?

Thanks in advance

@kmilos
Copy link
Contributor Author

kmilos commented May 1, 2023

I haven't checked, but that sounds like #144.

But it should be uncompressed! And the size seems to corroborate that - 2*100MB...

@LebedevRI
Copy link
Member

I haven't checked, but that sounds like #144.

But it should be uncompressed! And the size seems to corroborate that - 2*100MB...

I'm not sure what you mean?
For 3FR, we don't even have non-LJpeg options:

RawImage ThreefrDecoder::decodeRawInternal() {
const auto* raw = mRootIFD->getIFDWithTag(TiffTag::STRIPOFFSETS, 1);
uint32_t width = raw->getEntry(TiffTag::IMAGEWIDTH)->getU32();
uint32_t height = raw->getEntry(TiffTag::IMAGELENGTH)->getU32();
uint32_t off = raw->getEntry(TiffTag::STRIPOFFSETS)->getU32();
// STRIPBYTECOUNTS is strange/invalid for the existing 3FR samples...
const ByteStream bs(DataBuffer(mFile.getSubView(off), Endianness::little));
mRaw->dim = iPoint2D(width, height);
HasselbladLJpegDecoder l(bs, mRaw);
mRaw->createData();
l.decode();
return mRaw;
}

@kkempter-getswift
Copy link

per this request:

That is pretty weird as the RPU sample is uncompressed.

@kkempter-getswift Does the camera support both uncompressed and compressed raw output? Please upload all possible outputs to RPU, as requested there.

The camera allows me to select either RAW or "RAW + JPG"

and then I can choose 14bit or 16bit

there are no other options. The files I uploaded are 16bit

I just uploaded a 14bit example file (B0000077.3FR)

@kmilos
Copy link
Contributor Author

kmilos commented May 1, 2023

I'm not sure what you mean?
For 3FR, we don't even have non-LJpeg options

Ok, that would explain it then 😉

At least this 16b X2D sample is uncompressed so needs to be hooked up differently.

@kmilos
Copy link
Contributor Author

kmilos commented May 2, 2023

I just uploaded a 14bit example file (B0000077.3FR)

There is no difference whatsoever between the 14-bit and 16-bit modes it would seem - still uncompressed, and data is 16-bit, with no gaps in the values.

@kmilos
Copy link
Contributor Author

kmilos commented May 2, 2023

Also, the X1D II 50C samples on RPU are uncompressed, and I think supported? @LebedevRI

Edit: Looks like there is also a compressed X1DM2-50c (.fff) sample, and the X1D II 50C (.3FR) uncompressed one. Only compressed mode was checked perhaps? So this report then makes more sense?

Yep, out of e.g. these
image
only the compressed X1DM2-50c works, while the top three seem to be the identical "X1D II 50C" body. (No idea where the "cfv 50C II" duplicate came from, it seems to have nothing like that in the metadata...)

@kmilos
Copy link
Contributor Author

kmilos commented May 2, 2023

Also added (copied w/ crop tweak for Mk II as well) support for the X1D while we're at it since the sample is there.

@LebedevRI
Copy link
Member

Please split this into 3+ sequential PR's - uncompressed, and then per camera.

@kmilos
Copy link
Contributor Author

kmilos commented May 2, 2023

Please split this into 3+ sequential PR's - uncompressed, and then per camera.

Done.

This now depends on #466

<Color x="0" y="1">GREEN</Color>
<Color x="1" y="1">BLUE</Color>
</CFA>
<Crop x="126" y="94" width="-118" height="0"/>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, crop still leaves a few garbage pixels in.
How do you come up with it? Have you tried the methodology i outlined in some previous PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I indeed had no time to go through the zoom & scroll procedure, this was from raw data PPM dump...

@LebedevRI LebedevRI merged commit fc8fc38 into darktable-org:develop May 3, 2023
43 checks passed
@LebedevRI
Copy link
Member

Thank you!

@kkempter-getswift
Copy link

I suspect you all have fixed this already, and I suspect I'm doing something wrong to get the latest code with the x2d fixes, however just in case it's helpful:

I checked out the master branch and compiled

I also tried via checking out the master branch then:
~/darktable$ cd src/external/rawspeed/
~/darktable/src/external/rawspeed$ gh pr checkout 463
and compiled

In both cased above I get this when I open darktable, import my 3FR images and then double click on one of the images:

 **2.5480 [rawspeed] (B0000060.3FR) rawspeed::HasselbladLJpegDecoder::HasselbladLJpegDecoder(rawspeed::ByteStream, const rawspeed::RawImage&), line 43: Unexpected image dimensions found: (11904; 8842)
 2.5480 [libraw_open] extensions whitelist: `cr3'
 2.8348 [temperature] failed to read camera white balance information from `B0000060.3FR'!**

@kmilos kmilos deleted the hassy_x2d branch May 3, 2023 08:39
@kmilos
Copy link
Contributor Author

kmilos commented May 3, 2023

@kkempter-getswift All the necessary changes have been merged now, so you can just pull the develop branch again, no need for individual PR checkout...

Also, as mentioned, if possible, please upload the equivalent .FFF file to RPU please.

@kkempter-getswift
Copy link

Thanks,

I dont see any .FFF files, when I connect the camera as a storage device I have a DCIM directory and in that directory a sub directory (this one is named 114HASBL) and within the sub directory I have only the .3FR files, here is a directory listing:

file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000004.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000005.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000006.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000007.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000008.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000009.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000010.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000011.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000012.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000013.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000014.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000015.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000016.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000017.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000018.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000019.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000020.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000021.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000022.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000023.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000024.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000025.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000026.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000027.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000028.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000029.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000030.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000031.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000032.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000033.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000034.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000035.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000036.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000037.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000038.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000039.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000040.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000041.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000042.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000043.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000044.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000045.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000046.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000047.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000048.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000049.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000050.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000051.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000052.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000053.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000054.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000055.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000056.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000057.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000058.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000059.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000060.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000061.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000062.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000063.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000064.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000065.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000066.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000067.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000068.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000069.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000070.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000071.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000072.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000073.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000074.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000075.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000076.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000077.3FR
file:///run/media/kevink/X2D-100C-CF/DCIM/114HASBL/B0000078.3FR

If there is something else I can provide let me know, or if I am doing it wrong

Thanks in advance

@kmilos
Copy link
Contributor Author

kmilos commented May 3, 2023

If there is something else I can provide let me know, or if I am doing it wrong

I think you get .FFF only if you transfer from camera using Phocus.

Please test transferring both a 16-bit and a 14-bit file - i.e. for the exact two 3FR files you uploaded to RPU (B0000001 and B0000077), create the matching FFF ones by transferring them in Phocus please.

@kkempter-getswift
Copy link

I just uploaded to RPU a file named Job_0002.fff it was created by the Hasselblad Phocus software when I imported B0000050.3FR I think I already uploaded B0000050.3FR but I will do so again to be sure

@LebedevRI
Copy link
Member

No B0000050.3FR on RPU.

#469 (comment)

@kkempter-getswift could you please follow the guideline outlined on the RPU, i.e. please provide all the nessesary samples (i.e. all 4?, including re-shooing existing samples) from said camera of the same scene, ideally something like daylight landscape?

@kkempter-getswift
Copy link

I have uploaded the below set of files, each of these are more in line with the guidelines:

B0000009.3FR and Job_0009.fff

@kkempter-getswift
Copy link

I have also uploaded the below set of files, each of these are more in line with the guidelines:

B0000010.3FR and Job_0010.fff

@LebedevRI
Copy link
Member

Just so we are at the ideal conditions, could you please upload 3 samples //with same content//:

  • 16bit uncompressed
  • 14bit (uncompressed)
  • the *.fff produced from the 16bit uncompressed sample
    ?

@kkempter-getswift
Copy link

I have uploaded the following 4 files:

B0000079.3FR -> Job_0079.fff (16bit)

B0000080.3FR -> Job_0080.fff (14bit)

@LebedevRI
Copy link
Member

@kkempter-getswift thank you, verified the uploads!

@kkempter-getswift
Copy link

All

I compiled the latest development branch, everything is working, Thank You!!!

I am however seeing these output errors on the command line:

80.4232 Trouble: [color calibration] white balance error (B0000020.3FR 17)
81.4348 Trouble: [color calibration] white balance error (B0000036.3FR 33)
81.4432 Trouble: [color calibration] white balance error (B0000036.3FR 33)
81.4571 Trouble: [white balance] double application of white balance (B0000036.3FR 33)
81.4821 Trouble: [color calibration] white balance error (B0000036.3FR 33)
84.2032 Trouble: [color calibration] white balance error (B0000036.3FR 33)
84.4754 Trouble: [color calibration] white balance error (B0000036.3FR 33)
84.4755 Trouble: [white balance] double application of white balance (B0000036.3FR 33)
84.4884 Trouble: [white balance] double application of white balance (B0000036.3FR 33)
84.5225 Trouble: [color calibration] white balance error (B0000036.3FR 33)
84.7614 Trouble: [color calibration] white balance error (B0000036.3FR 33)

@LebedevRI
Copy link
Member

Nice!
That question is more suited for appropriate darktable communication channels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Resolved
Development

Successfully merging this pull request may close these issues.

None yet

4 participants