You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am working with the exiv2 program and manpage at the moment, so had a quick look at this.
A CR3 test file that contains preview images can be found here. It is worth noting that the thumbnail is included in the list of any previews.
You can list all the available previews in the file using:
$ exiv2 --print p IMG_0482_raw.CR3
but none are listed. I tried this with Exiv2 0.27.4 and 1.0.0.9 .
If I use Exiftool then I can extract all the previews (including a JPEG conversion):
exiftool -a -b -W %d%f_%t%-c.%s -preview:all IMG_0482_raw.CR3
The manpage states that Exiv2 supports CR3 reading thumbnails, but this doesn't work. I will update the manpage, but do not have time to investigate further.
@paolobenve Good to hear from you, Paolo. I hope all's well with you. I don't believe you received any assistance with localisation and that's a pity.
@postscript-dev Thank you for your helpful look at this, Peter.
I don't know the thumbnails/previews code as well I know other parts of Exiv2. There are two ways in which previews/thumbnails are stored in images. The first is in the Exif data (normally in IFD1) and the exiv2 preview manager works on that data. The second depends on the file format and is not part of the Exif standard. I don't think Exiv2 provides any support for thumbnails/previews outside of the Exif metadata. So this is not an oversight in Exiv2 BMFF/CR3 support, it's a general weakness in Exiv2 which requires more development to support format dependant thumbnails/previews.
In my book, I have an in-depth analysis of about 20 common image formats (Tiff, JPEG, PNG and others). The CR3 format has previews stored as illustrated in the uuid/canp and THMD boxes. The tvisitor code in my book understands these previews, although tvisitor does not provide a preview/thumbnail extractor. Here is the drawing from my book of the CR3 format .
There is a discussion in book of previews and thumbnails. This is an area of Exiv2 that would benefit from more development. As I have now retired, I'm not volunteering to get involved. https://clanmills.com/exiv2/book/#5
@paolobenve Paolo, when I was running today, I thought of something ultra-nerdy (anything to keep my mind off the pain).
I can prove my case that Exiv2 Preview Manager only understands Thumbnails in IFD1. In the test file Stonehenge.jpg in my book and http://clanmills.com/Stonehenge.jpg, we have:
There are three preview/thumbnails. One is in IFD1. It's in Exif.Image.JPEGInterchangeLength. There are another two which are 640x424 and 1620x1080 in the trailer of the JPG. Exiv2 only sees the one in IFD1:
@paolobenve I have spoken to @lbschenkel about his availability to work on Exiv2 and it is limited. I know you offered to get involved with localisation and asked about a Crowdin "template" or something. As a typical native English speaker, I know nothing about localisation. Can you ask Leonardo about this and I'm sure he'll help you.
Am I right in thinking that this is mostly a matter of patching BmffImage::boxHandler() so that it handles THMB, PRVW and the JPEG cases in the "trak"s, (as per https://github.com/lclevy/canon_cr3 ), and stashing the appropriate offsets, sizes and mime-types in nativePreviews_ ?
Yes, David @dhoulder, I think that's what you have to do. I have never really studied the preview code carefully. I believe it's designed to recover the previews in IFD1 of the Exif metadata. It might be easy to get it to recover images from the THMB and PRVW boxes. While on vacation last week, I started to think about updating tvisitor.cpp in my book to parse these images and haven't yet done that because I worked on Exiv2 v0.27.5 RC3 this week.
If you're interested in working on this, I will work on tvisitor.cpp and study the preview code. Two heads are better than one.
Hi Robin @clanmills . OK, I'll have a closer look at it over the next few days. As far as I can tell, the preview extraction for most camera raw files either grabs a preview straight out of an EXIF tag value directly, or reads a file offset and length out of an EXIF tag value and then grabs the preview from that section of the file. Since CR3 files don't store preview data in EXIF tags, some other mechanism will need to be used, but fortunately it looks like this is already catered for (for PhotoShop and EPS files I think) via Image::nativePreviews_.
Looks like a wet week ahead for us here in Hobart so this might be good use of my time :-)
I'll look at this during the week. There are a couple of parts of my book which need more work. One is Image Previews and the other is the convertor code (which converts Exif metadata into XMP etc).
I think the convertor code is an absurdity. We should only deal with what's in a file. We should never cook up anything. We should be like SuperMan and deal in "Truth, Justice and the American Way".
I've had a look at the tvisitor.cpp code in my book and it's now happily dumping Exif thumbnails and the THMB and PRVW boxes in CR3 files. I'll submit that code on Monday and update the text in the book. I'll also have a look at the preview code in Exiv2 tomorrow.
We're off on vacation on Friday (for a week). Let's have a talk mid-week on Zoom. By Friday, I think we'll have a good plan about how to get this done.
When working on the manpage, I used the Stonehenge.jpg file (from @clanmills) to demonstrate the different commands. When it came to PREVIEW IMAGES AND THUMBNAILS though, I was a bit stuck. Apart from the thumbnail, exiv2 cannot extract any other previews from the image. If you have time @dhoulder, would you consider adding previews for the Nikon JPEG? The manpage could then be updated and this would help others understand how the feature works. I am happy to update the manpage, if the previews are added.
Thanks, Peter @postscript-dev . You can read my thoughts about previews here: https://exiv2.org/book/#5 I think the preview code is both complex and a little weak.
The CR3 code is read-only, so we don't require code to insert/delete/modify thumbnails.
I have updated the tvisitor.cpp code in my book today to beautiful locate and dump the previews in JPEG and CR3 files. And I've updated some parts of the book.
I haven't yet totally understand the machinery for reading previews. However it appears to be searching for particular tags in the Exif metadata such as Exif.Thumbnail.JPEGInterchangeFormat and Exif.Thumbnail.JPEGInterchangeFormatLength.
However the image can hold thumbnails which are not in the Exif metadata. Tomorrow, I will implement NativePreviewList& BmffImage::nativePreviews() to return the THMB and PRVW images from the CR3 file. These are JPEG images. tvisitor reveals them as follows:
Based on my work with tvisitor.cpp today, I'm anticipating success tomorrow. I may also have time to implement NativePreviewList& JpegImage::nativePreviews().
tvisitor reveals previews which are not seen by exiv2:
tvisitor identifies 4 frames (JPEG/Start of Frame0 segments). So it finds the 160x120 image in the Exif metadata, the "Big Guy" (6000x4000) and two more previews (640x424 and 1620x1080) following the "Big Guy".
David: @dhoulder My discovery today about nativePreviews() fits perfectly with your observations about Image::nativePreviews_. I'm 99% confident that I'll have a strong prototype of this on Tuesday. Do you have time on Wednesday or Thursday to talk on Zoom?
If it's wet enough in Hobart on Tuesday, you might have this working before I get out of bed on Tuesday.
Tomorrow, I will implement NativePreviewList& BmffImage::nativePreviews() to return the THMB and PRVW images from the CR3 file.
I was assuming that the existing base class implementation in Image::nativePreviews() already does the job, and all that is required in BmffImage::boxHandler() is to fill out the nativePreviews_ vector in much the same way that readWriteEpsMetadata() in epsimage.cpp does. Am I missing something?
I think you're ahead of me in understanding this magic. Image::nativePreviews() is the base implementation and does exactly nothing. It returns an empty list. Today, I'm going to implement BmffImage::nativePreviews(). BmffImage::readMetadata() calls the box handler and I'll modify that to save the THMD and PRVW images (which are JPEGs) in the instance nativePreviews_. I think that'll fix it.
I got my 10 hour time change mental arithmetic wrong yesterday. 11am in England is 9pm in Hobart. Happy to make the meeting earlier if you prefer. You can edit the meeting time in Google Calendar. Kind of odd that I messed up because I have Melbourne in the world clock of my smart watch for speaking with my friend Penny.
Here's the patch to recover the THMB from a CR3 file on branch 0.27-maintenance. PRVW is very similar.
As this issue is to find previews in CR3 files, it sufficient to deal with THMB and PRVW in BmffImage::boxHandler(), test it and ask @paolobenve to test it. A PR should be submitted to both the 0.27-maintenance and main branches. This change is too late for submission to Exiv2 v0.27.5 which is frozen.
@postscript-dev: A new issue should be opened concerning previews in JPEGs. The changes to jpegimage.cpp will be similar to bmpffimage.cpp. Beware that the current implementation of JpegImage::readMetadata() does not search beyond the first SOS (Start of Scan) and we should retain that behaviour for performance reasons. We should only read the whole JPEG when asked to read the previews.
This is an important moment in my life. I believe (and hope) that I've written the last line of code I will ever contribute to Exiv2.
diff --git a/src/bmffimage.cpp b/src/bmffimage.cpp
index 7b53789b..b19362dc 100644
--- a/src/bmffimage.cpp+++ b/src/bmffimage.cpp@@ -76,6 +76,7 @@
#define TAG_colr 0x636f6c72 /**< "colr" */
#define TAG_exif 0x45786966 /**< "Exif" Used by JXL*/
#define TAG_xml 0x786d6c20 /**< "xml" Used by JXL*/
+#define TAG_THMB 0x54484d42 /*=< "THMB" Used by CR3*/
// *****************************************************************************
// class member definitions
@@ -442,6 +443,25 @@ namespace Exiv2
}
} break;
+ case TAG_THMB: {+ NativePreview nativePreview;+ int32_t header = 16 ;+ nativePreview.position_ = io_->tell()+header;+ byte buf[header];+ io_->read(buf,header);+ nativePreview.width_ = getShort(buf+4,bigEndian);+ nativePreview.height_ = getShort(buf+6,bigEndian);+ nativePreview.size_ = getLong (buf+8,bigEndian);+ nativePreview.filter_ = "";+ nativePreview.mimeType_ = "image/jpeg";+ nativePreviews_.push_back(nativePreview);++ if ( bTrace ) {+ out << Internal::stringFormat("width,height,size = %d,%d,%d"+ ,nativePreview.width_,nativePreview.height_,nativePreview.size_);+ }+ } break;+
case TAG_cmt1:
parseTiff(Internal::Tag::root, box_length);
break;
#1968 has been updated w/ the JPEG limitation as well, please feel free to test. Works for me standalone, and within darktable (now throwing the expected [exiv2 dt_exif_get_thumbnail] couldn't find thumbnail for ... in the HDR cases since exiv2 returns an empty list).
I was expecting the patches to be accumulative. They are not. They are exclusive. I applied Dave's BMFF/mdat/booster and removed your CR3/JPEG/Preview. Duuuuuuuu. My error.
@paolobenve I think we're going to release v0.27.5 tomorrow with CR3/preview support. I'm not going to fix #1977. Too risky without going to RC4. #1968 (comment)
I'd like to thank everybody for pulling together to get this done. Well done @dhoulder. A very valuable contribution.
For you, @kmilos. Of course. Submit a PR and I'll review it.
A totally different way to dig around in image files is the use the tvisitor (and dmpf) code in my book. It's similar to Exiv2, however it benefits from being a rewrite. If somebody wants to work with me to add a section about HDR to my book, even better. Only HDR metadata, I never want sucked into image-processing.
I've been eating dinner and thinking about everything. No need to modify the CR3 related PRs (1958, 1968, 1975, 1978). I'll approve them and merge them and we're done. It's an amazing feeling. No RC4, no 1977.
Exiv2 has been developed by trial and error. Two people have done exceptional work to get the metadata monster under control. Andreas Huggel for the Exiv2 Architecture. Phil Harvey for documenting metadata on his web site.
The complications of metadata are many. The MakerNotes are usually Tiffs (except Kodak), every manufacturer has different undocumented tags in their makernotes. Parts of the Exif Spec (IFD1) seem totally absurd to me. Then there are the image formats. Every one does more or less the same thing, but in a different way. And of course there are different versions of the unwritten specs. Let's not discuss lens recognition.
There's the usual open-source challenges of multi-platform build, test, documentation, security, recruitment, user-support, bug-fixing, release-engineering and so on. We mustn't discuss legal matters!
Anyway, with sufficient effort, we've done it. Exiv2 v0.27.5 will be released on 2021-10-22. I'm really happy about the status of Exiv2. I'm especially pleased that a gang of great engineers are now involved. Thank You Everybody for Contributing.
Please see the lclevy link above, all the ones that mention HDR under CNCV... It's a Canon proprietary format, and someone would need to reverse engineer it. Exiv2 can now extract it, so its job is done here.
Activity
postscript-dev commentedon Sep 6, 2021
I am working with the exiv2 program and manpage at the moment, so had a quick look at this.
A CR3 test file that contains preview images can be found here. It is worth noting that the thumbnail is included in the list of any previews.
You can list all the available previews in the file using:
but none are listed. I tried this with Exiv2 0.27.4 and 1.0.0.9 .
If I use Exiftool then I can extract all the previews (including a JPEG conversion):
The manpage states that Exiv2 supports CR3 reading thumbnails, but this doesn't work. I will update the manpage, but do not have time to investigate further.
clanmills commentedon Sep 6, 2021
@paolobenve Good to hear from you, Paolo. I hope all's well with you. I don't believe you received any assistance with localisation and that's a pity.
@postscript-dev Thank you for your helpful look at this, Peter.
I don't know the thumbnails/previews code as well I know other parts of Exiv2. There are two ways in which previews/thumbnails are stored in images. The first is in the Exif data (normally in IFD1) and the exiv2 preview manager works on that data. The second depends on the file format and is not part of the Exif standard. I don't think Exiv2 provides any support for thumbnails/previews outside of the Exif metadata. So this is not an oversight in Exiv2 BMFF/CR3 support, it's a general weakness in Exiv2 which requires more development to support format dependant thumbnails/previews.
In my book, I have an in-depth analysis of about 20 common image formats (Tiff, JPEG, PNG and others). The CR3 format has previews stored as illustrated in the uuid/canp and THMD boxes. The tvisitor code in my book understands these previews, although tvisitor does not provide a preview/thumbnail extractor. Here is the drawing from my book of the CR3 format .
There is a discussion in book of previews and thumbnails. This is an area of Exiv2 that would benefit from more development. As I have now retired, I'm not volunteering to get involved. https://clanmills.com/exiv2/book/#5
clanmills commentedon Sep 8, 2021
@paolobenve Paolo, when I was running today, I thought of something ultra-nerdy (anything to keep my mind off the pain).
I can prove my case that Exiv2 Preview Manager only understands Thumbnails in IFD1. In the test file Stonehenge.jpg in my book and http://clanmills.com/Stonehenge.jpg, we have:
There are three preview/thumbnails. One is in IFD1. It's in Exif.Image.JPEGInterchangeLength. There are another two which are 640x424 and 1620x1080 in the trailer of the JPG. Exiv2 only sees the one in IFD1:
In the file cr3.cr3 in my book, we have:
There are two preview/thumbnails. The first is a little JPEG 120x160 pixels:
The second is in the canon uuid/canp box at the end:
Exiv2 doesn't know about either of them because they are not in IFD1!
I must be a nerd to find this interesting!
Can this be fixed? Of course. It's only work. I'll mentor a volunteer.
clanmills commentedon Sep 24, 2021
@paolobenve I have spoken to @lbschenkel about his availability to work on Exiv2 and it is limited. I know you offered to get involved with localisation and asked about a Crowdin "template" or something. As a typical native English speaker, I know nothing about localisation. Can you ask Leonardo about this and I'm sure he'll help you.
dhoulder commentedon Oct 3, 2021
Am I right in thinking that this is mostly a matter of patching BmffImage::boxHandler() so that it handles THMB, PRVW and the JPEG cases in the "trak"s, (as per https://github.com/lclevy/canon_cr3 ), and stashing the appropriate offsets, sizes and mime-types in nativePreviews_ ?
clanmills commentedon Oct 3, 2021
Yes, David @dhoulder, I think that's what you have to do. I have never really studied the preview code carefully. I believe it's designed to recover the previews in IFD1 of the Exif metadata. It might be easy to get it to recover images from the THMB and PRVW boxes. While on vacation last week, I started to think about updating tvisitor.cpp in my book to parse these images and haven't yet done that because I worked on Exiv2 v0.27.5 RC3 this week.
If you're interested in working on this, I will work on tvisitor.cpp and study the preview code. Two heads are better than one.
dhoulder commentedon Oct 3, 2021
Hi Robin @clanmills . OK, I'll have a closer look at it over the next few days. As far as I can tell, the preview extraction for most camera raw files either grabs a preview straight out of an EXIF tag value directly, or reads a file offset and length out of an EXIF tag value and then grabs the preview from that section of the file. Since CR3 files don't store preview data in EXIF tags, some other mechanism will need to be used, but fortunately it looks like this is already catered for (for PhotoShop and EPS files I think) via Image::nativePreviews_.
Looks like a wet week ahead for us here in Hobart so this might be good use of my time :-)
clanmills commentedon Oct 3, 2021
I'll look at this during the week. There are a couple of parts of my book which need more work. One is Image Previews and the other is the convertor code (which converts Exif metadata into XMP etc).
I think the convertor code is an absurdity. We should only deal with what's in a file. We should never cook up anything. We should be like SuperMan and deal in "Truth, Justice and the American Way".
Speak later in the week.
clanmills commentedon Oct 3, 2021
I've had a look at the tvisitor.cpp code in my book and it's now happily dumping Exif thumbnails and the THMB and PRVW boxes in CR3 files. I'll submit that code on Monday and update the text in the book. I'll also have a look at the preview code in Exiv2 tomorrow.
We're off on vacation on Friday (for a week). Let's have a talk mid-week on Zoom. By Friday, I think we'll have a good plan about how to get this done.
postscript-dev commentedon Oct 4, 2021
When working on the manpage, I used the Stonehenge.jpg file (from @clanmills) to demonstrate the different commands. When it came to PREVIEW IMAGES AND THUMBNAILS though, I was a bit stuck. Apart from the thumbnail, exiv2 cannot extract any other previews from the image. If you have time @dhoulder, would you consider adding previews for the Nikon JPEG? The manpage could then be updated and this would help others understand how the feature works. I am happy to update the manpage, if the previews are added.
clanmills commentedon Oct 4, 2021
Thanks, Peter @postscript-dev . You can read my thoughts about previews here: https://exiv2.org/book/#5 I think the preview code is both complex and a little weak.
The CR3 code is read-only, so we don't require code to insert/delete/modify thumbnails.
I have updated the tvisitor.cpp code in my book today to beautiful locate and dump the previews in JPEG and CR3 files. And I've updated some parts of the book.
I haven't yet totally understand the machinery for reading previews. However it appears to be searching for particular tags in the Exif metadata such as Exif.Thumbnail.JPEGInterchangeFormat and Exif.Thumbnail.JPEGInterchangeFormatLength.
However the image can hold thumbnails which are not in the Exif metadata. Tomorrow, I will implement
NativePreviewList& BmffImage::nativePreviews()to return the THMB and PRVW images from the CR3 file. These are JPEG images. tvisitor reveals them as follows:.../book/build $ ./tvisitor -pR ../files/cr3.cr3 ........... 8680 | 11864 | THMB | | _____._x__.=_.__.... 0 0 0 0 0 160 0 120 0 0 46 61 0 1 0 0 255 216 255 219 STRUCTURE OF JPEG FILE (II): ../files/cr3.cr3:32->22784:24->20552:8704->11840 address | marker | length | signature 0 | 0xffd8 SOI 2 | 0xffdb DQT | 132 | _....................................... 136 | 0xffc0 SOF0 | 17 | ._x_...!_........ = h,w = 120,160 155 | 0xffc4 DHT | 418 | __........________............_......... 575 | 0xffda SOS 11835 | 0xffd9 EOI END: ../files/cr3.cr3:32->22784:24->20552:8704->11840 END: ../files/cr3.cr3:32->22784:24->20552 20568 | 108 | mvhd | | ____..4...4.___.___. 0 0 0 0 214 228 52 142 214 228 52 142 0 0 0 1 0 0 0 1 20676 | 484 | trak | | ___\tkhd___...4...4. 0 0 0 92 116 107 104 100 0 0 0 7 214 228 52 142 214 228 52 142 21160 | 584 | trak | | ___\tkhd___...4...4. 0 0 0 92 116 107 104 100 0 0 0 7 214 228 52 142 214 228 52 142 21744 | 600 | trak | | ___\tkhd___...4...4. 0 0 0 92 116 107 104 100 0 0 0 7 214 228 52 142 214 228 52 142 22344 | 440 | trak | | ___\tkhd___...4...4. 0 0 0 92 116 107 104 100 0 0 0 7 214 228 52 142 214 228 52 142 END: ../files/cr3.cr3:32->22784 22816 | 65560 | uuid | xmp | <?xpacket begin='... 60 63 120 112 97 99 107 101 116 32 98 101 103 105 110 61 39 239 187 191 88376 | 264929 | uuid | canp | _______._...PRVW____ 0 0 0 0 0 0 0 1 0 4 10 193 80 82 86 87 0 0 0 0 STRUCTURE OF JPEG FILE (II): ../files/cr3.cr3:88432->264865 address | marker | length | signature 0 | 0xffd8 SOI 2 | 0xffdb DQT | 132 | _....................................... 136 | 0xffc0 SOF0 | 17 | ..8.T..!_........ = h,w = 1080,1620 155 | 0xffc4 DHT | 418 | __........________............_......... 575 | 0xffda SOS 264871 | 0xffd9 EOI END: ../files/cr3.cr3:88432->264865 353305 | 13789912 | mdat | | _____.j....._._..... 0 0 0 0 0 210 106 216 255 216 255 219 0 132 0 6 4 4 6 4 END: ../files/cr3.cr3Based on my work with tvisitor.cpp today, I'm anticipating success tomorrow. I may also have time to implement
NativePreviewList& JpegImage::nativePreviews().tvisitor reveals previews which are not seen by exiv2:
$ exiv2 -pp ~/Stonehenge.jpg Preview 1: image/jpeg, 160x120 pixels, 10837 bytestvisitor identifies 4 frames (JPEG/Start of Frame0 segments). So it finds the 160x120 image in the Exif metadata, the "Big Guy" (6000x4000) and two more previews (640x424 and 1620x1080) following the "Big Guy".
David: @dhoulder My discovery today about
nativePreviews()fits perfectly with your observations aboutImage::nativePreviews_. I'm 99% confident that I'll have a strong prototype of this on Tuesday. Do you have time on Wednesday or Thursday to talk on Zoom?If it's wet enough in Hobart on Tuesday, you might have this working before I get out of bed on Tuesday.
dhoulder commentedon Oct 4, 2021
I was assuming that the existing base class implementation in Image::nativePreviews() already does the job, and all that is required in BmffImage::boxHandler() is to fill out the nativePreviews_ vector in much the same way that readWriteEpsMetadata() in epsimage.cpp does. Am I missing something?
clanmills commentedon Oct 5, 2021
I think you're ahead of me in understanding this magic. Image::nativePreviews() is the base implementation and does exactly nothing. It returns an empty list. Today, I'm going to implement BmffImage::nativePreviews(). BmffImage::readMetadata() calls the box handler and I'll modify that to save the THMD and PRVW images (which are JPEGs) in the instance nativePreviews_. I think that'll fix it.
I got my 10 hour time change mental arithmetic wrong yesterday. 11am in England is 9pm in Hobart. Happy to make the meeting earlier if you prefer. You can edit the meeting time in Google Calendar. Kind of odd that I messed up because I have Melbourne in the world clock of my smart watch for speaking with my friend Penny.
dhoulder commentedon Oct 5, 2021
OK, getting there…
Now I just need to extract the big JPEG from inside "trak" 1
clanmills commentedon Oct 5, 2021
Here's the patch to recover the THMB from a CR3 file on branch 0.27-maintenance. PRVW is very similar.
As this issue is to find previews in CR3 files, it sufficient to deal with THMB and PRVW in
BmffImage::boxHandler(), test it and ask @paolobenve to test it. A PR should be submitted to both the 0.27-maintenance and main branches. This change is too late for submission to Exiv2 v0.27.5 which is frozen.@postscript-dev: A new issue should be opened concerning previews in JPEGs. The changes to jpegimage.cpp will be similar to bmpffimage.cpp. Beware that the current implementation of
JpegImage::readMetadata()does not search beyond the first SOS (Start of Scan) and we should retain that behaviour for performance reasons. We should only read the whole JPEG when asked to read the previews.This is an important moment in my life. I believe (and hope) that I've written the last line of code I will ever contribute to Exiv2.
58 remaining items
kmilos commentedon Oct 20, 2021
#1968 has been updated w/ the JPEG limitation as well, please feel free to test. Works for me standalone, and within darktable (now throwing the expected
[exiv2 dt_exif_get_thumbnail] couldn't find thumbnail for ...in the HDR cases since exiv2 returns an empty list).clanmills commentedon Oct 20, 2021
@kmilos I know why my test didn't show your CR3/JPEG didn't appear to work. I update my pr_1958 branch with:
I was expecting the patches to be accumulative. They are not. They are exclusive. I applied Dave's BMFF/mdat/booster and removed your CR3/JPEG/Preview. Duuuuuuuu. My error.
@paolobenve I think we're going to release v0.27.5 tomorrow with CR3/preview support. I'm not going to fix #1977. Too risky without going to RC4. #1968 (comment)
I'd like to thank everybody for pulling together to get this done. Well done @dhoulder. A very valuable contribution.
kmilos commentedon Oct 20, 2021
Great Robin, thanks for confirming.
If you permit, I'd like to squeeze in one more commit to expose the HDR preview as a binary buffer so someone can dump it and study it further...
clanmills commentedon Oct 20, 2021
For you, @kmilos. Of course. Submit a PR and I'll review it.
A totally different way to dig around in image files is the use the tvisitor (and dmpf) code in my book. It's similar to Exiv2, however it benefits from being a rewrite. If somebody wants to work with me to add a section about HDR to my book, even better. Only HDR metadata, I never want sucked into image-processing.
I've been eating dinner and thinking about everything. No need to modify the CR3 related PRs (1958, 1968, 1975, 1978). I'll approve them and merge them and we're done. It's an amazing feeling. No RC4, no 1977.
dhoulder commentedon Oct 20, 2021
@kmilos wrote:
Oh no 😞 …
I guess this is what happens when there's no published spec 😬.
Thanks @kmilos for #1978, and thanks @clanmills for all the testing and coordination!
kmilos commentedon Oct 20, 2021
I have added my idea to #1978 now - if you're ok with it, we could cherry-pick it for #1968 as well.
clanmills commentedon Oct 20, 2021
Exiv2 has been developed by trial and error. Two people have done exceptional work to get the metadata monster under control. Andreas Huggel for the Exiv2 Architecture. Phil Harvey for documenting metadata on his web site.
The complications of metadata are many. The MakerNotes are usually Tiffs (except Kodak), every manufacturer has different undocumented tags in their makernotes. Parts of the Exif Spec (IFD1) seem totally absurd to me. Then there are the image formats. Every one does more or less the same thing, but in a different way. And of course there are different versions of the unwritten specs. Let's not discuss lens recognition.
There's the usual open-source challenges of multi-platform build, test, documentation, security, recruitment, user-support, bug-fixing, release-engineering and so on. We mustn't discuss legal matters!
Anyway, with sufficient effort, we've done it. Exiv2 v0.27.5 will be released on 2021-10-22. I'm really happy about the status of Exiv2. I'm especially pleased that a gang of great engineers are now involved. Thank You Everybody for Contributing.
dhoulder commentedon Oct 20, 2021
@kmilos
Sounds good to me. Good point about fullBox(): I guess THMB and PRVW qualify. Thanks
kmilos commentedon Oct 21, 2021
With #1981 now merged, I think we can close again.
JPEG previews are fully supported, HDR ones just exposed as unknown binary.
darktable log now includes the more informative message straight from exiv2:
paolobenve commentedon Oct 21, 2021
What cr3 have them? don't we know anything of the image type?
kmilos commentedon Oct 21, 2021
Please see the lclevy link above, all the ones that mention HDR under CNCV... It's a Canon proprietary format, and someone would need to reverse engineer it. Exiv2 can now extract it, so its job is done here.
Canon cr3 previews (Exiv2#1958)