Exiv2 does not find certain AF tags for the Nikon D850 #646
Comments
|
Can you provide a sample image with the D850. I have 7xD810 images, however if you have additional images, they will be useful. |
|
OK. A little progress. I downloaded DSC_6983.NEF from the sample gallery. I didn't write Exiv2, and don't understand how the makernote decoder performs its magic. However, that makes the puzzle more interesting. I'll update you when I've made some progress. Here are some thoughts for the moment. I'm fairly sure that the AF2 structure is tag 0xb7 And this is what's in it: When I dump the file with the recursive parser (which I wrote): So, I can see 84 bytes of AF2 data in the file. Story to be continued. |
|
I see that Exiftool is very interested in the same 84 bytes: And I can extract those bytes bytes with 13500 = Offset to the MakerNote And there the trail dies. There's nothing interesting in those 84 bytes. And exiv2 is reporting the same very uninteresting data. So. I'm back to square one. Can you provide a sample image. Can you extract your items of interest with exiftool and share that output with me, please? |
|
Hi Robin, It appears the RAW file you downloaded was not captured with "contrast detect" AF, and I believe that's the reason you're coming up empty. I have a file I'll share with you, I've uploaded the file to my Dropbox, and I'll send the link to the e-mail address you've linked to your Github profile. Thanks so much for your help with this. Michael |
|
Michael Thanks for sending your file. Here's what I see: The MakerNote is 203064 bytes at offset 13500. Tag 0x00b7 is 84 bytes at offset 26286 in the maker note. Once more, I have to add 10 to get the location of the data. The 10 is correct, my old brain can't explain it, yet!. It's clear that the As you can see. 84 bytes. However it's not the land of zeros we saw in the other file. This is the same binary block that exiftool reveals with the -v5 option: Decoded as: I have a little collection of home-made utilities (such as dmpf), and one is hex.cpp: And using this: And that's what's in the file (from bytes 70-80) in tag 0x007b So we've identified the location of the data in the file. And now I have to figure out how to correctly decode the key/values. Here's what we're reporting: You've noticed that "ContrastDetectAF" is set in your file, however we arge reporting lots of zeros. I didn't write the metadata decoder and Andreas (the author) made such a good job that I've hardly ever visited that code. It uses a "visitor" model. The parser descends through the file and calls functions to decode different types of tags. I have to discover where tag 0x007b is encountered and "fix it" to respect Exif.NikonAf2.ContrastDetectAF. Difficult? Unlikely. However with all puzzles, it'll take a little time. We're making progress. |
|
Glad to see my file was helpful, and you're making such excellent progress! |
|
@mallman I'm pleased to be working on this. I've been working on Exiv2 since 2008 and have wondered for some time: How does |
|
Almost there. The following table in src/nikonmn.cpp defines the layout of the binary metadata: Bytes 0-3, 4 and 5 are Version, ContrastDetectAF and AFAreaMode which are 1010, 1, 1 in Michael's file (and 1010,0,0 in the sample). When ContrastDetectAF and AFAreaMode are 1, (and the length is 84 bytes or more), metadata such as AFAreaWidth is not at byte 24, it's at 70 (or something like that). When TiffReader::visitBinaryArray() reads 0x007b, it calls TiffReader::readTiffEntry() which creates a binary object for the 84 bytes. visitBinaryArray pushes this object for later processing in Later on, TiffReader::postProcess() is called to decode the binary objects on Now I need to understand:
This is fun. I'll nail this one in the next few days. |
|
postProcess(), sends the visitor to inspect every deferred object, and the metadata is created here: I can get this to stop in the debugger at: constant nikonAf2Id == object->group() == 47 is defined in Phil has the 84 byte data structure documented here: https://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Nikon.html (Search "Nikon AFInfo2 Tags") We could length the table in nikonmn_int.cpp (to include the shorts at bytes 70+), however we'd have two entries for some keys such as: We know which to select from the value of ContrastDetectAF and AFAreaMode. If we can set decoderFct == 0, we can skip "early" shorts. This has to be achieved in a generic data-driven way to achieve the following effect: I'll work on something else for a little while to rest the brain, then run 10k to think about this. |
|
I had a two good ideas while running:
It's obviously possible to run the array of TagInfo and calculate the maximum offset expected. We should ensure that we never overflow the data read (in this case 84 bytes). I know the data is:
There's a rather complex dance with the visitor code. It appears to create an Array of values (called elements_ in the code), and then he runs a loop on those values to call The crux of this dance is nikonAf2Id (47). By the time we arrive in decodeTiffEntry(), this->group() is nikonAf2Id. However in the dance about, it's nikon3Id (40) which might be the maker note. Tomorrow: More time in the debugger to understand this code. Or run another 10k tomorrow and think. |
|
More progress. Tag 0x007b is born as a nikonAf2ld and cannot be modified after birth! It's defined in this array in tiffimage_int.cpp We have to study the declaration EXV_COMPLEX_BINARY_ARRAY. I suspect that the function nikonSelector() is in src/makernote_int.cpp I've always suspected that Andreas' tiff parsing engine was ingenious. And now I know that it is ingenious. |
|
Very exciting work! As an aside, I'll mention I'm migrating the way I extract image metadata for a Mac photo app I'm writing from the built-in ImageIO framework to Exiv2. Sadly, the amount of metadata we get through ImageIO is quite anemic. For example, I can extract the GPS time of day ( Another problem I've discovered is related to another discovery I've recently made. In all the years I've used Aperture to store my photos (since v1.1)—when I've adjusted the timezone in a file I assumed Aperture was just setting a value in its metadata database. And indeed it does. However, it also alters the exif data in the raw file itself! Heresy! Indeed, it alters other fields as well! I thought Aperture never modified the raw files, and this is a very disappointing discovery for me. One of the concerns I have regarding any modification of raw files is inadvertent file corruption, either because of a bug in the modifying code or because a bit gets flipped when the file is rewritten. The other problem is that—even though Aperture writes the original, unaltered date/time in another field (namely Which all goes to say I really appreciate that we have an open-source alternative such as Exiv2. Aside from basic metadata extraction, I want to overlay the contrast detect focus area over an image so I can see in post exactly where I set focus. This is where your work comes into play. Cheers. |
|
Open Source is much better than Apple/Adobe/Google/Microsoft products because you have direct access to the author/maintainer. If you report anything to Apple about Aperture, it's unlikely that anything will happen as I believe it's no longer maintained. However, Apple Radar support is OK. I've reported a couple of issues concerning Xcode/C++, and been pleased by their response. I maintain Exiv2 on a Mac and use Parallels to run VMs for Linux and Windows. About 2012, I wrote a Cocoa/Objective-C++ application to display metadata. Exiv2 was much much faster than Apple's code. About 100x faster, I think. I concluded that NSImage (or whatever I was used to read the file) built a frame-buffer with all the pixels. Exiv2 doesn't do that. Exiv2 reads the metadata and nothing else. A typical JPEG is about 4mb, of which about 10k is Metadata. Raw images, 40mb of which about 10k is Metadata. If you're looking for a commercial alternative to Aperture for raw images, Exiv2 is used by AlienSkin Exposure. They gave me a license. Oh, and they paid for lunch when my wife and I went to visit them on vacation in Raleigh, NC last year. Exiv2 is also used by FotoStation who told me to use their 7 day trial license to fix something for them! I wonder if they'll buy me lunch when I visit? We're thinking about Norway for our summer vacation. I hope to solve your puzzle today. For sure I can provide you with a bash script to extract the 0x007b tag using We are planning to rewrite the Tiff code to support BigTiff (64 bit tiff) for v0.28. Tiff is "the heart" of Exiv2 because Exif Metadata is stored as an embedded tiff. And most Raw formats are based on Tiff. Another member of the team has accepted this challenge. I'm highly motivated to work on your issue so that I can help Luis with BigTiff. We'd like to reuse Andreas' robust TiffVisitor code. To do that we have to understand it! |
|
Eureka: There's something not quite right. You'll see the key changed its name from Very happy. I'm going to run 10k to celebrate. It'll be after sunset when I get back. I should be able to get this into the code this week. |
👏
I can live with that! 😉
Sounds appropriate! (And healthy!) |
FWIW, the Apple API call I was using, namely I've switched over to Exiv2 with the help of an awesome ObjC wrapper I found on Github, import Exiv2
guard let metadata = FLTImageMetadata(imageAt: fileUrl) else { return }
let dateTimeOriginal = metadata.value(forMetadataKey: "Exif.Image.DateTimeOriginal",
metadataType: .exif) as? StringGreat success! 👍 I'll check out AlienSkin and FotoStation. Thanks for the tips! |
|
Thanks for the updates. Very pleased to learn that there's an Obj-C/swift/cocoa wrapper for Exiv2. I hope to get the code for this issue into Exiv2 in the next few days. I'm off to visit family in the States next week and have a vacation to get away from the weather in England. When I get home (late February), I'll work on Exiv2 v0.27.1. The change for this issue will be in Exiv2 v0.27.1. Scheduled 2019-03-31. Incidentally, there's a request to support FocusPositon from ARW files. The FocusPosition metadata in the Sony file is encrypted! (Why? Don't know. Phil helped me with the decryption code). Andreas' Tiff Engine supports decryption. With the knowledge I've gained from working on your issue, I hope to also fix FocusPosition in ARW. |
|
The Nikon maker note tag 0x00b7 (AFInfo2) data varies greatly by model. The latest ExifTool release (Jan 15) updates this decoding to add support for newer Nikon models (Z series). You can look at the "Condition" statements in the Exiftool source code to see which tags are valid under what conditions. |
Fantastic. Have a great break! |
|
|
@clanmills can you send me the image @mallman sent you to your personal email, so that I can continue with the research ? |
|
@clanmills Hi Robin. Please do not share that file. Instead, we can share the following raw file publicly: |
|
NP. Deleted. |
|
PR Submitted: #900 |
I'm looking for the location and size metadata of the contrast-detect AF point in my Nikon D850 raw files. Such tags include
Exif.NikonAf2.AFAreaXPosition,Exif.NikonAf2.AFAreaYPosition,Exif.NikonAf2.AFAreaWidth, etc. Exiv2 finds the correct values for the D810 but not for the D850.Exiftool does return this information for the D850, so I checked its source. Apparently the file offset of this metadata depends on the value of
Exif.NikonAf2.Version. For example, comparehttps://github.com/exiftool/exiftool/blob/9e0ce916748abb56dbb8593d2184445080f678d0/lib/Image/ExifTool/Nikon.pm#L3519-L3525
with
https://github.com/exiftool/exiftool/blob/9e0ce916748abb56dbb8593d2184445080f678d0/lib/Image/ExifTool/Nikon.pm#L3627-L3633.
Note specifically the different
Conditionkeys.The text was updated successfully, but these errors were encountered: