Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Canon lens handling regression in 0.28.0 #2746

Closed
douzzer opened this issue Aug 26, 2023 · 51 comments · Fixed by #2938
Closed

Canon lens handling regression in 0.28.0 #2746

douzzer opened this issue Aug 26, 2023 · 51 comments · Fixed by #2938
Labels
bug i18n Internationalisation lens Issue related to lens detection
Milestone

Comments

@douzzer
Copy link

douzzer commented Aug 26, 2023

Describe the bug

Up to and including 0.27.7-release, I'm able to get correct lens detection for this lens setup in Darktable, by putting this in ~/.exiv2:

[canon]
748=Canon EF 100-400mm f/4.5-5.6L IS II USM + 1.4x extender

But with 0.28.0-release, that doesn't work anymore, and the lens is misdetected as Tamron. Nothing else I've tried has worked. In particular, when I tried patching out the conflicting Tamron lenses, Darktable reported "Lens 748 not found" or words to that effect, even though obviously the Canon 748 entry is still there. This is the patch I tried:

--- exiv2-0.28.0/src/canonmn_int.cpp.dist       2023-05-08 11:01:13.000000000 -0500
+++ exiv2-0.28.0/src/canonmn_int.cpp    2023-08-26 12:42:58.326572444 -0500
@@ -1865,8 +1865,6 @@ constexpr TagDetails canonCsLensType[] =
                                           {747, "Canon EF 100-400mm f/4.5-5.6L IS II USM"},
                                           {747, "Tamron SP 150-600mm f/5-6.3 Di VC USD G2"},  // 1
                                           {748, "Canon EF 100-400mm f/4.5-5.6L IS II USM + 1.4x"},
-                                          {748, "Tamron 100-400mm f/4.5-6.3 Di VC USD + 1.4x"},  // 1
-                                          {748, "Tamron 70-210mm f/4 Di VC USD + 2x"},           // 2
                                           {749, "Tamron 100-400mm f/4.5-6.3 Di VC USD + 2x"},
                                           {749, "Canon EF 100-400mm f/4.5-5.6L IS II USM + 2x"},  // 1
                                           {750, "Canon EF 35mm f/1.4L II USM"},

The most worrisome syndrome though is that the override entry in ~/.exiv2 is being ignored.

To Reproduce

I've posted a CR2 with this lens here

(github has a 25MB attachment limit, and doesn't allow CR2 files besides.)

I'm running Darktable 4.4.2 but it wasn't the variable, and it's detecting the lens correctly when built and linked against exiv2-0.27.7.

To see the syndrome, set up the ~/.exiv2 above, import the picture into DT, and admire the detected lens ID on the left.

Note that you'll see the no-lens-info error in the lens correction tool for both the Canon and Tamron versions of ID 748. I run a Lensfun with the correction data for the Canon setup patched in.

Expected behavior

In the left pane with the image attributes, the lens entry should read

Canon EF 100-400mm f/4.5-5.6L IS II USM + 1.4x extender

(Which is what it says when built+linked with v0.27.7, and it's what I have in my ~/.exiv2.)

Desktop (please complete the following information):

  • Gentoo
  • exiv2 0.27.7 vs 0.28.0
  • gcc-11.4.1
  • built out of Gentoo portage. with these use flags:
 U I
 - - abi_x86_32 : 32-bit (x86) libraries
 + + bmff       : Add support for the ISOBMFF format
 - - doc        : Add extra documentation (API, Javadoc, etc). It is recommended to enable per package instead of globally
 - - examples   : Install examples, usually source code
 + + nls        : Add Native Language Support (using gettext - GNU locale utilities)
 + + png        : Add support for libpng (PNG images)
 - - test       : Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)
 - - webready   : Support reading metadata over the internet
 + + xmp        : Enable support for Extensible Metadata Platform (Adobe XMP)

Additional context

I believe this DT issue relates: darktable-org/darktable#14882

@MStraeten
Copy link

seems to be dependent on camera or lens:

Canon 7D:

exiv2 -pt --grep lens/i /c/Users/.../20130625-IMG_5459.CR2
Exif.CanonCs.LensType                        Short       1  Canon EF 24-105mm f/4L IS USM
Exif.CanonCs.Lens                            Short       3  24.0 - 105.0 mm
Exif.Canon.LensModel                         Ascii      74  EF24-105mm f/4L IS USM
Exif.CanonLe.LensSerialNumber                SLong       1  1980760064
Exif.Photo.LensSpecification                 Rational    4  24-105mm
Exif.Photo.LensModel                         Ascii      23  EF24-105mm f/4L IS USM

Canon M6:

exiv2 -pt --grep lens/i /c/Users/.../20170604-IMG_0790.CR2
Exif.CanonCs.LensType                        Short       1  Unknown Lens (254)
Exif.CanonCs.Lens                            Short       3  100.0 mm
Exif.Photo.LensSpecification                 Rational    4  100mm
Exif.Photo.LensModel                         Ascii      28  EF100mm f/2.8L Macro IS USM
Exif.Photo.LensSerialNumber                  Ascii      11  0000008672

Canon 5DmIV:

exiv2 -pt --grep lens/i /c/Users/.../20210227-IMG_0511.CR2
Exif.CanonCs.LensType                        Short       1  Unknown Lens (747)
Exif.CanonCs.Lens                            Short       3  100.0 - 400.0 mm
Exif.Canon.LensModel                         Ascii      74  EF100-400mm f/4.5-5.6L IS II USM
Exif.CanonLe.LensSerialNumber                SLong       1  -1879011276
Exif.CanonAfC.USMLensElectronicMF            SLong       1  Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible      SLong       1  Continue Focus Search
Exif.Photo.LensSpecification                 Rational    4  100-400mm
Exif.Photo.LensModel                         Ascii      33  EF100-400mm f/4.5-5.6L IS II USM
Exif.Photo.LensSerialNumber                  Ascii      11  3490009075

Canon R6 - EF Lens:
exiv2 -pt --grep lens/i /c/Users/.../IMG_3123.CR3

Exif.Photo.LensSpecification                 Rational    4  100mm
Exif.Photo.LensModel                         Ascii      28  EF100mm f/2.8L Macro IS USM
Exif.Photo.LensSerialNumber                  Ascii      11  0000008672
Exif.CanonCs.LensType                        Short       1  Unknown Lens (254)
Exif.CanonCs.Lens                            Short       3  100.0 mm
Exif.CanonFi.RFLensType                      SShort      1  n/v
Exif.Canon.LensModel                         Ascii     138  EF100mm f/2.8L Macro IS USM
Exif.CanonLe.LensSerialNumber                SLong       1  -2046820352
Exif.CanonAfC.USMLensElectronicMF            SLong       1  Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible      SLong       1  Continue Focus Search

Canon R6 - RF Lens:

exiv2 -pt --grep lens/i /c/Users/.../20231008-IMG_0599.CR3
Exif.Photo.LensSpecification                 Rational    4  24-105mm
Exif.Photo.LensModel                         Ascii      23  RF24-105mm F4 L IS USM
Exif.Photo.LensSerialNumber                  Ascii      11  7413007751
Exif.CanonCs.LensType                        Short       1  Canon RF 24-105mm F4-7.1 IS STM
Exif.CanonCs.Lens                            Short       3  24.0 - 105.0 mm
Exif.CanonFi.RFLensType                      SShort      1  Canon RF 24-105mm F4L IS USM
Exif.Canon.LensModel                         Ascii     138  RF24-105mm F4 L IS USM
Exif.CanonLe.LensSerialNumber                SLong       1  1996493684
Exif.CanonAfC.USMLensElectronicMF            SLong       1  Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible      SLong       1  Continue Focus Search

@jim-easterbrook
Copy link
Contributor

I've noticed something interesting about the Exif.CanonLe.LensSerialNumber values shown above. -1879011276 in hex is 90009034, which is a rearrangement of the first eight characters of the Exif.Photo.LensSerialNumber value 3490009075. Similarly -2046820352 in hex is 86000000 for a lens with serial number 0000008672, and 1996493684 in hex is 77001374, serial number 7413007751.

My Canon EOS 100D with 18-55mm kit lens reports:

Exif.CanonLe.LensSerialNumber                SLong       1  422576128
Exif.CanonLe.0x0001                          SLong       1  129
Exif.Photo.LensSerialNumber                  Ascii      11  0000301981

422576128 in hex is 19300000 and 129 in hex is 81. Join those bytes together in the right order and you get the complete serial number 0000301981.

I spotted this while investigating #2138 - is it possible that Exif.CanonLe.LensSerialNumber should actually be a 6-byte hex coded value instead of a 4 byte integer?

@douzzer
Copy link
Author

douzzer commented Dec 27, 2023

A brief follow-up:

Yesterday, motivated by CVE-2023-44398, I finally put together a workaround that restored functionality (and then some) for Canon lenses with extenders. To get it to work right, two patches were needed:

  • Patch out the conflicting 3rd party lenses in src/canonmn_int.cpp, so that the mapping from lens ID number to model name is unique and well-behaved.
  • Patch lensfun to parse the vital statistics from lens model names with extenders/reducers/etc.

My strategy isn't generally applicable because it removes all support for 3rd party EF mount lenses with conflicting ID numbers, but it probably points in the direction of how to resolve the problem.

And it is well to note, the problem seems to be a system-level interaction between exiv2 and lensfun.

Lensfun patch attached for reference.
parse_lens_name_extenders.patch.txt

@AxelG-DE
Copy link

AxelG-DE commented Jan 7, 2024

Hello all....
forgive me, I did'nt read to all this thoroughly, as I am not feeling well.

I have similar case (gentoo, exiv2-0.28.1) but with Nikon camera and Tamron lense (70/200 2.8 Di G2) it was recognized well in the past, but not anymore. It get listed as ID=226, even well implemented in ~/.exiv2

@AxelG-DE
Copy link

AxelG-DE commented Jan 7, 2024

BTW gentoo also has a lensfun trouble. Only Version 0.3.4_rc1 is available, which is afaik not the official stable version.

With that 0.3.4_rc1, the lensfun-update-data does not work (bug is opened since long time). Donno how much this can be related

@AxelG-DE
Copy link

AxelG-DE commented Jan 8, 2024

either @douzzer or maybe @kmilos would you mind changing the title, as I have strong indication it is not just canon lenses but also the Tamron 70/200 2.8 G2 connected to my Nikon D750?

Do you need me to provide some prove? If yes, what do you need me to do?

@kmilos
Copy link
Collaborator

kmilos commented Jan 8, 2024

Do you need me to provide some prove?

Yes, please. For example exiv2 ex extracted data and exiv2 -Pkvt output for the current 0.28.1 failing case and the previously working older one.

@AxelG-DE
Copy link

AxelG-DE commented Jan 9, 2024

@kmilos per your request :)

the current situation:
grafik

$ exiv2 ex 2023-12-29_151301_AG7_9385.NEF
Warning: Exif tag Exif.NikonPreview.JPEGInterchangeFormatLength not encoded
Warning: Exif IFD NikonPreview not encoded
Warning: Exif tag Exif.Photo.MakerNote not encoded
Warning: Exif tag Exif.NikonSi02xx.0x027a not encoded
Warning: Exif tag Exif.Nikon3.0x00bc not encoded

and here the output of -Pkvt
exiv2_current_Pkvt_20240109.txt

and here comes the former scenario (hopefully because I cannot downgrade to exiv2-0.27......

first a screenshot:
grafik

$ exiv2 ex 2018-01-08_180136_AG7_1503.NEF
Warning: Exif tag Exif.NikonPreview.JPEGInterchangeFormatLength not encoded
Warning: Exif IFD NikonPreview not encoded
Warning: Exif tag Exif.Photo.MakerNote not encoded
Warning: Exif tag Exif.NikonSi02xx.0x027a not encoded
Warning: Exif tag Exif.Nikon3.0x00bc not encoded

and here the output of -Pkvt

exiv2_former-scenario_Pkvt_20240109.txt

I used diff and there is no difference :-(

my .exiv2

$ cat .exiv2

# how to locate, where this file should be:
# exiv2 --verbose --version --grep config_path
# first approach should be: 
# exiftool -G -n -Lens* PathToFileandFilename
# then use something like:
# [MakerNotes]    Lens ID Number                  : 226
# or if there is no Lens ID Number
# exiv2 -pv --grep lenstype/i PathToFileandFilename
# then use something like:
# 0x0201 OlympusEq    LensType                    Byte        6  2 0 5 16 0 17

[nikon]
#
#226=Tamron/ Tamron SP 70-200mm F/2.8 Di VC USD G2
#226=Tamron SP 70-200mm F/2.8 Di VC USD G2
226=Tamron SP 70-200mm F/2.8 Di VC USD G2

[olympus]
2 0 5 16 0 17=Lumix G 20mm F1.7 II Asph.
#0 0 35 0 1 0=Funny Wee Lens

[canon]
137=Tamron SP 24-70mm F/2.8 Di VC USD
502=TAMRON 17-35mm F/2.8-4 Di LD Aspherical (IF)

Tonight I have to focus on other priorities, so please understand, I will be a bit silent, but that topic really bothers me...

@kmilos
Copy link
Collaborator

kmilos commented Jan 9, 2024

Thanks for the data.

I used diff and there is no difference :-(

As the output between 0.27.x and 0.28.x is the same, and this lens was never in the internal database, can't really say this is the same bug.

Now added in #2889

@AxelG-DE
Copy link

AxelG-DE commented Jan 9, 2024

@kmilos my data "former scenario" was also created with exiv2-0.28.1, just old files, where I did refresh the metadata via darktable at that time.

I have dependency blockage and hence cannot downgrade exiv2. That blockage is tricky for some reason, but this would digress here too much. Maybe in the next few days that resolves (as kde currently not fully upgraded due to missing other deps)

My major trouble is, that the ~/.exiv2 is not obayed

@kmilos
Copy link
Collaborator

kmilos commented Jan 9, 2024

my data "former scenario" was also created with exiv2-0.28.1, just old files, where I did refresh the metadata via darktable at that time

Well, that's not very useful here.

that topic really bothers me...

My major trouble is, that the ~/.exiv2 is not obayed

Back to square one then. We'll just have to wait until someone w/ "skin in the game" (or otherwise interested) digs into the code and fires up a debugger.

@AxelG-DE
Copy link

AxelG-DE commented Jan 9, 2024

my data "former scenario" was also created with exiv2-0.28.1, just old files, where I did refresh the metadata via darktable at that time

Well, that's not very useful here.

I am really sorry for this. Currently I do not see a way to downgrade exiv2 in gentoo.

  • usually it would work and re-emerge several packages
  • however, on top it complains kfilemetadata-5.112.0:5/5.112::gentoo is not available (even not in the rebuild list)
  • that is correct, because kfilemetadata is available at version 5.113.0 but that version cannot be installed due to many many other unresolved dependencies, which typically just resolve over time
  • once that is done, I may have a chance to downgrade exiv2
  • I know there are emerge-switches to ignore deps, but I am not familiar with it and it is kind of asking for trouble when it is above my head
  • (plus I got a flu, so my concentration is low)

I will amend as soon as any possible

@AxelG-DE
Copy link

@kmilos

I made it to downgrade exiv-0.27.7 (finally it was my boon mistake why I stuck, I am embarrassed :-( )

The output of "exiv2 ex [File]" looks same but actually the files are not same, when I diff them

exiv2-0.28.1

$ exiv2 ex 2023-12-29_151301_AG7_9385.NEF
Warning: Exif tag Exif.NikonPreview.JPEGInterchangeFormatLength not encoded
Warning: Exif IFD NikonPreview not encoded
Warning: Exif tag Exif.Photo.MakerNote not encoded
Warning: Exif tag Exif.NikonSi02xx.0x027a not encoded
Warning: Exif tag Exif.Nikon3.0x00bc not encoded

exiv-028_2023-12-29_151301_AG7_9385.exv.zip

exiv2-0.28_current_Pkvt_20240109.txt

exiv2-0.27.7

$ exiv2 ex 2023-12-29_151301_AG7_9385.NEF
Warning: Exif tag Exif.NikonPreview.JPEGInterchangeFormatLength not encoded
Warning: Exif IFD NikonPreview not encoded
Warning: Exif tag Exif.Photo.MakerNote not encoded
Warning: Exif tag Exif.NikonSi02xx.0x027a not encoded
Warning: Exif tag Exif.Nikon3.0x00bc not encoded

exiv-027_2023-12-29_151301_AG7_9385.exv.zip

exiv2-0.27_current_Pkvt_20240112.txt

So if I do a diff between the two Pkvt outputs, I get:
(left is old 0.27.7 and right is new 0.28.1 with the problem not to obay ~/.exiv2)

$ diff ~/analysis/exiv2-0.27_current_Pkvt_20240112.txt ~/analysis/exiv2-0.28_current_Pkvt_20240109.txt 
108c108
< Exif.NikonPc.QuickAdjust                      255  n/a
---
> Exif.NikonPc.QuickAdjust                      255  n/v
153c153
< Exif.NikonLd3.LensIDNumber                    226  Tamron SP 70-200mm F/2.8 Di VC USD G2
---
> Exif.NikonLd3.LensIDNumber                    226  226
164c164,189
< Exif.Nikon3.FlashInfo                         48 49 48 55 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  48 49 48 55 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
---
> Exif.NikonFl7.Version                         48 49 48 55  1.07
> Exif.NikonFl7.FlashSource                     0  Keine
> Exif.NikonFl7.ExternalFlashFirmware           0  n/v
> Exif.NikonFl7.ExternalFlashData1              0  External flash off
> Exif.NikonFl7.ExternalFlashData2              0  Aus
> Exif.NikonFl7.FlashCompensationMaster         0  n/v
> Exif.NikonFl7.FlashFocalLength                0  n/v
> Exif.NikonFl7.RepeatingFlashRate              0  n/v
> Exif.NikonFl7.RepeatingFlashCount             0  n/v
> Exif.NikonFl7.FlashGNDistance                 0  n/v
> Exif.NikonFl7.FlashColorFilter                0  Keine
> Exif.NikonFl7.FlashGroupAControlData          0  Aus
> Exif.NikonFl7.FlashGroupBCControlData         0  Aus, Aus
> Exif.NikonFl7.FlashCompensationGroupA         0  n/v
> Exif.NikonFl7.FlashCompensationGroupB         0  n/v
> Exif.NikonFl7.FlashCompensationGroupC         0  n/v
> Exif.NikonFl7.ExternalFlashData3              0  Exposure Comp.: Entire frame
> Exif.NikonFl7.CameraFlashOutput               0  0.0 EV
> Exif.NikonFl7.CameraExposureCompensation      0  0.0 EV
> Exif.NikonFl7.CameraFlashCompensation         0  0.0 EV
> Exif.NikonFl7.ExternalFlashData4              0  Illumination Pat.: Standard
> Exif.NikonFl7.FlashZoomHeadPosition           0  n/v
> Exif.NikonFl7.FlashMasterOutput               0  n/v
> Exif.NikonFl7.FlashGroupAOutput               0  n/v
> Exif.NikonFl7.FlashGroupBOutput               0  n/v
> Exif.NikonFl7.FlashGroupCOutput               0  n/v
183c208
< Exif.NikonAf2.ContrastDetectAFInFocus         0  0
---
> Exif.NikonAf2.ContrastDetectAFInFocus         0  Nein

@kmilos
Copy link
Collaborator

kmilos commented Jan 12, 2024

Thanks @AxelG-DE for your effort, that's more useful. 👍

However, I can't reproduce locally (w/ 0.28.x branch), using either of your .exv files and this exiv2.ini (I'm on Windows):

[nikon]
226=Foo Bar 3

I get

Exif.NikonLd3.LensIDNumber                    226  Foo Bar 3

@kmilos
Copy link
Collaborator

kmilos commented Jan 12, 2024

Just a wild guess: perhaps Gentoo maintainers (and other distros having similar issues) forgot to add and enable the inih dependency needed for parsing these config files since 0.28.0?

Edit: Yep. Didn't forget, but not enabled on Gentoo - @douzzer as well, so please file bug there so it's tracked somewhere. @MStraeten your (Canon only?) case might still be different/unrelated I think.

@AxelG-DE
Copy link

@kmilos seems you have the right smell

from the ebuild (that controls how a package will be built and installed):

multilib_src_configure() {
        local mycmakeargs=(
                -DEXIV2_BUILD_SAMPLES=NO
                -DEXIV2_ENABLE_BROTLI=OFF
                -DEXIV2_ENABLE_NLS=$(usex nls)
                -DEXIV2_ENABLE_PNG=$(usex png)
                -DEXIV2_ENABLE_CURL=$(usex webready)
                -DEXIV2_ENABLE_INIH=OFF # multilib/libdir logic is broken
                -DEXIV2_ENABLE_WEBREADY=$(usex webready)
                -DEXIV2_ENABLE_XMP=$(usex xmp)
                -DEXIV2_ENABLE_BMFF=$(usex bmff)

@kmilos
Copy link
Collaborator

kmilos commented Jan 12, 2024

Yep. As requested, let's please not discuss the Gentoo problem here any further but move it to their tracker (as Team Exiv2 has nothing to do with how distros package), and leave this to analysing the remaining issue @MStraeten is seeing.

@AxelG-DE
Copy link

@kmilos I have filed an upstream bug https://bugs.gentoo.org/921937

gentoo-bot pushed a commit to gentoo/gentoo that referenced this issue Jan 12, 2024
We must have this enabled as it's an important feature. Also, I was wrong before,
we just needed to make inih multilib.

Bug: Exiv2/exiv2#2746
Closes: https://bugs.gentoo.org/921937
Signed-off-by: Sam James <sam@gentoo.org>
@ktatar156
Copy link

I can confirm that this bug also affect myself. Is there change that you'll fix this?

@joaobrisson
Copy link

joaobrisson commented Mar 28, 2024

files are shared under CC0 and I am willing to share the original CR2 files under the same license for testing

@joaobrisson
Copy link

joaobrisson commented Mar 28, 2024

to answer your early question and make things weirder

exiv2 0.28.2 on MacOS

however repeat of your test held

% exiv2 -pa -g LensType IMG_*.exv
\IMG_0001.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM
IMG_0002.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM
IMG_0003.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM
IMG_0004.exv Exif.CanonCs.LensType Short 1 Unknown Lens (193)
IMG_0005.exv Exif.CanonCs.LensType Short 1 Unknown Lens (193)
IMG_0006.exv Exif.CanonCs.LensType Short 1 Unknown Lens (193)
IMG_0007.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS
IMG_0008.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS
IMG_0009.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS
IMG_0010.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS
IMG_0011.exv Exif.CanonCs.LensType Short 1 Unknown Lens (51)
IMG_0012.exv Exif.CanonCs.LensType Short 1 Unknown Lens (51)
IMG_0013.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
IMG_0014.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
IMG_0015.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
IMG_0016.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
IMG_0017.exv Exif.CanonCs.LensType Short 1 Unknown Lens (238)
IMG_0018.exv Exif.CanonCs.LensType Short 1 Unknown Lens (238)
IMG_0019.exv Exif.CanonCs.LensType Short 1 Unknown Lens (238)

search for lens model did not flinch

% exiv2 -pa -g LensModel IMG_*.exv
IMG_0001.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0001.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0002.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0002.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0003.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0003.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0004.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0004.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0005.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0005.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0006.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0006.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM
IMG_0007.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0007.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0008.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0008.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0009.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0009.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0010.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0010.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0011.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0011.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0012.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0012.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS
IMG_0013.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0013.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0014.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0014.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0015.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0015.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0016.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0016.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0017.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0017.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0018.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0018.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0019.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
IMG_0019.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM

On MacOS Preview Inspector Exif tab lenses ids are shown twice, first as "EF...." and then as "Canon EF..."

Last, the Metadata Converter app on MacOS correctly identifies the lenses on all cases

May I hint that this is a problem with the exiv2 implementation for MacOS?

@kmilos
Copy link
Collaborator

kmilos commented Mar 28, 2024

May I hint that this is a problem with the exiv2 implementation for MacOS?

I already hinted above std::regex implementation is not to be trusted across platforms.

@kmilos
Copy link
Collaborator

kmilos commented Mar 28, 2024

Another idea: regex could also be affected by locale settings?

@hassec
Copy link
Member

hassec commented Mar 28, 2024

@joaobrisson My test was also on macOS.

Are you on an ARM based mac (like M1-M3 processor) or x86?
How was exiv2 installed, or did you compile it from source? If so, which flags?
What is the output of the following commands:

  • exiv2 --version --verbose (you can skip the lines starting with xmlns=....)
  • file $(which exiv2)
  • otool -L $(which exiv2)
  • locale

@kmilos without a reproducer or actual evidence, just blaming it on std::regex isn't very productive. So let's focus on getting a reproducer that one can debug and fix.

@kmilos
Copy link
Collaborator

kmilos commented Mar 28, 2024

@hassec #2746 (comment)

@joaobrisson
Copy link

joaobrisson commented Mar 28, 2024

x86 64 bit
got exiv2 from https://macappstore.org/exiv2/

% exiv2 --version --verbose
exiv2 0.28.2
exiv2=0.28.2
platform=apple
compiler=Clang
bits=64
dll=1
debug=0
cplusplus=201703
version=Apple LLVM 14.0.0 (clang-1400.0.29.202)
date=Feb 13 2024
time=18:49:36
processpath=/usr/local/bin
localedir=../share/locale
package_name=exiv2
curlprotocols=dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
curl=1
executable=/usr/local/Cellar/exiv2/0.28.2/bin/exiv2
library=/usr/local/Cellar/exiv2/0.28.2/lib/libexiv2.0.28.2.dylib
library=/usr/local/Cellar/gettext/0.22.5/lib/libintl.8.dylib
library=/usr/local/Cellar/brotli/1.1.0/lib/libbrotlicommon.1.1.0.dylib
library=/usr/local/Cellar/brotli/1.1.0/lib/libbrotlidec.1.1.0.dylib
library=/usr/local/Cellar/inih/58/lib/libinih.0.dylib
library=/usr/local/Cellar/inih/58/lib/libINIReader.0.dylib
library=/usr/lib/system/libsystem_kernel.dylib
library=/usr/lib/system/libsystem_platform.dylib
library=/usr/lib/system/libsystem_pthread.dylib
have_inttypes=1
have_libintl=1
have_lensdata=1
have_iconv=1
have_memory=1
have_stdbool=1
have_stdint=1
have_stdlib=1
have_strlib=0
have_strerror_r=1
have_strings_h=0
have_mmap=1
have_munmap=1
have_sys_stat=1
have_unistd_h=0
have_sys_mman=1
have_libz=1
have_brotli=1
have_xmptoolkit=1
adobe_xmpsdk=0
have_bool=0
have_strings=1
have_sys_types=1
have_unistd=1
enable_bmff=1
enable_webready=1
enable_nls=1
enable_video=1
use_curl=1
config_path=/Users/brisson/.exiv2
uid=502
euid=502
gid=20

file $(which exiv2)
/usr/local/bin/exiv2: Mach-O 64-bit executable x86_64

% otool -L $(which exiv2)
/usr/local/bin/exiv2:
/usr/local/Cellar/exiv2/0.28.2/lib/libexiv2.28.dylib (compatibility version 28.0.0, current version 0.28.2)
/usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 13.0.0, current version 13.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.23.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

% locale
LANG="pt_PT.UTF-8"
LC_COLLATE="pt_PT.UTF-8"
LC_CTYPE="pt_PT.UTF-8"
LC_MESSAGES="pt_PT.UTF-8"
LC_MONETARY="pt_PT.UTF-8"
LC_NUMERIC="pt_PT.UTF-8"
LC_TIME="pt_PT.UTF-8"
LC_ALL=

@Exiv2 Exiv2 deleted a comment from joaobrisson Mar 28, 2024
@kmilos
Copy link
Collaborator

kmilos commented Mar 28, 2024

@zisoft is reproducing this on macOS as well

@hassec
Copy link
Member

hassec commented Mar 28, 2024

@zisoft also on a x86 mac?

@zisoft
Copy link

zisoft commented Mar 28, 2024

The weird thing for me is, that the command line gives fine results for my Sigma lens but darktable reports Unknown Lens (183).

~ exiv2 -pt --grep lens/i img_sigma.CR2
Exif.CanonCs.LensType                        Short       1  Sigma 105mm f/2.8 EX DG OS HSM Macro
Exif.CanonCs.Lens                            Short       3  105.0 mm
Exif.Canon.LensModel                         Ascii      74  105mm
Exif.CanonLiOp.DigitalLensOptimizer          SLong       1  Off
Exif.CanonLe.LensSerialNumber                Byte        5  0000000000
Exif.CanonAfC.USMLensElectronicMF            SLong       1  Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible      SLong       1  Continue Focus Search
Exif.Photo.LensSpecification                 Rational    4  105mm
Exif.Photo.LensModel                         Ascii       6  105mm
Exif.Photo.LensSerialNumber                  Ascii      11  0000000000
~ exiv2 -pe --grep lens/i img_sigma.CR2
Exif.CanonCs.LensType                        Short       1  183
Exif.CanonCs.Lens                            Short       3  105 105 1
Exif.Canon.LensModel                         Ascii      74  105mm
Exif.CanonLiOp.DigitalLensOptimizer          SLong       1  0
Exif.CanonLe.LensSerialNumber                Byte        5  0 0 0 0 0
Exif.CanonAfC.USMLensElectronicMF            SLong       1  0
Exif.CanonAfC.LensDriveWhenAFImpossible      SLong       1  0
Exif.Photo.LensSpecification                 Rational    4  105/1 105/1 0/1 0/1
Exif.Photo.LensModel                         Ascii       6  105mm
Exif.Photo.LensSerialNumber                  Ascii      11  0000000000

To me, it looks like Exiv2::ExifData &exifData = image->exifData(); is using -pe internally.

@zisoft
Copy link

zisoft commented Mar 28, 2024

@zisoft also on a x86 mac?

Yes, MacBook Pro 2019 (Intel)

@kmilos
Copy link
Collaborator

kmilos commented Mar 28, 2024

The weird thing for me is, that the command line gives fine results for my Sigma lens but darktable reports Unknown Lens (183).

How about locales, are both terminal and darktable running as de_DE for you?

@zisoft
Copy link

zisoft commented Mar 28, 2024

How about locales, are both terminal and darktable running as de_DE for you?

Good catch!
I have my terminal's locale usually set to en_US.UTF-8. Setting the locale to de_DE.UTF-8 now gives:

~ exiv2 -pt --grep lens/i img_sigma.CR2
Exif.CanonCs.LensType                        Short       1  Unknown Lens (183)
Exif.CanonCs.Lens                            Short       3  105.0 mm
Exif.Canon.LensModel                         Ascii      74  105mm
Exif.CanonLiOp.DigitalLensOptimizer          SLong       1  Aus
Exif.CanonLe.LensSerialNumber                Byte        5  0000000000
Exif.CanonAfC.USMLensElectronicMF            SLong       1  Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible      SLong       1  Continue Focus Search
Exif.Photo.LensSpecification                 Rational    4  105mm
Exif.Photo.LensModel                         Ascii       6  105mm
Exif.Photo.LensSerialNumber                  Ascii      11  0000000000

@joaobrisson
Copy link

Yeah!
It's the locale on macOS!
The DE locale yelds the same behaviour as the PT locale for both exiv2 and darktable (Refresh Exif): Unknown Lens!
But try switching to the UK and all of a sudden you get it right

So, for the time being, the workaround seems to be to set the locale to the UK or the US when importing to darktable while waiting for the exiv2 team to come up with some fix

@zisoft
Copy link

zisoft commented Mar 28, 2024

It's the locale on macOS!

Sure it is macOS only?

@joaobrisson
Copy link

joaobrisson commented Mar 28, 2024 via email

@zisoft
Copy link

zisoft commented Mar 28, 2024

My questions was, is this a general locale issue in exiv2 on each system, or is this macOS specific?

@kmilos
Copy link
Collaborator

kmilos commented Mar 28, 2024

@hassec std:stof seems to be locale dependent (expecting , as decimal separator in both de_DE and pt_PT). So it shouldn't be macOS only. Not std::regex it would seem after all, but it did come in w/ that commit.

@kmilos
Copy link
Collaborator

kmilos commented Mar 28, 2024

@hassec Maybe swicth to std::from_chars?

@zisoft
Copy link

zisoft commented Mar 28, 2024

We had a similar issue in darktable recently: Fix locale for aperture filter. There f/5.6 becomes f/5,6 in some locales.

Solution was to switch locale for LC_NUMERIC to C but this needs to be done thread-safe:

https://github.com/darktable-org/darktable/blob/aa84dc17a518caea3825f423091b034f60e48483/src/common/utility.c#L100-L119

@kmilos
Copy link
Collaborator

kmilos commented Mar 28, 2024

@zisoft Looks like it's better to avoid switching locale altogether, especially from a library... Since exiv2 requires C++17 anyway, std::from_chars (or some custom implementation) seems to be the best option for something this "simple":

Unlike other parsing functions in C++ and C libraries, std::from_chars is locale-independent, non-allocating, and non-throwing. Only a small subset of parsing policies used by other libraries (such as std::sscanf) is provided. This is intended to allow the fastest possible implementation that is useful in common high-throughput contexts such as text-based interchange (JSON or XML).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug i18n Internationalisation lens Issue related to lens detection
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants