Improved charset handling in UserComment: what does it mean?

[norbertj42]

One of the highlights of Exiv2 v0.27.3 is "Improved charset handling in UserComment". I scrolled through the change list, but did not find details to this item.
What I observed: in previous version toString returned the value including the leading "charset=...", whereas print() returned it without. Now in both cases I get the leading "charset=...".
Is this changed behaviour the improved handling or did something else change?
From my point of view the previous behaviour was the better one. print() returned a value, which is good for users which just want to see the value, whereas toString() gives the additional information about charset for users interested in charset.

[clanmills]

Good Question. I recommend that you read the discussion with Phil Harvey of ExifTool about this: #1046

I updated the man page exiv2.1 to explain how this is intended to work. Before v0.27.3 it was not documented.

I find charset (and all unicode/non-ascii) matters difficult to understand. I have a new contributor from China working with me at the moment on #1215 (currently PR #1257) and I have been thinking about asking ask him to review and test the Charset code after he completes his current assignment.

To address your concern about the API toString(), let me make a couple of observations:

1 In times gone by toString() could emit binary which was causing trouble in the test suite which uses the utility diff to compare files. Not only are arguments to diff platform dependent, binary diff on some platforms (Solaris I think) requires the utility bdiff. These horrors are being addressed in #1215.

2 You can get the "raw" value of stored bytes with another API. I think it's toValue(). I don't remember, however I believe this is discussed in #1046.

If you'd like to get involved in this, I would be delighted to accept your help.

[norbertj42]

Thanks for the details and references. I have read them, but my C++ is too poor to understand it completely. Some time ago I also read the thread #662 "Incorrect Unicode encoding of Exif UserComment tag", because I noticed that often UserComment written by one program is not displayed correct by other programs, if the text contains German umlauts (ä. ö, ü). But using toString() and if needed converting enconding I was able to display UserComment in readable way from most of the images written by other meta data editors I tried.

Having a closer look now, I detected, that this "compatibilty" was lost when I migrated to 0.27.3, because in some cases I see now only "charset=Ascii binary comment". Based on the discussion in #1046, I will check, if the other programs write the tag according specification. Even if the error is on their side, it would be great to have exiv2 as tolerant as it was in 0.27.2.

I would like to help and give something back to this project, which enabled my program. But I have just basics in C++ and my environment is Windows and Microsoft Visual Studio. If this helps, please let me know.

[clanmills]

Well the list of things TODO never gets shorter. Not because I'm not making progress, it's because we live in an expanding universe!

Here's something you could explore with your tools: #1255 By default VCPKG builds 32-bit exiv2 and:
1 Doesn't build the sample applications
2 Doesn't copy the DLLs into build/bin

It would be good to figure out how to run the test suite on that - especially when Leo finishes #1215.

A matter closely related to #1255 is building libiconv on Windows. I see that vcpkg builds libiconv and links it into Exiv2. I'd like to test that with the UserComment. I recently added CMake code to guarantee that Exiv2/MSVC never links libiconv on 0.27-maintenance. I did that because a user kept linking Cygwin64/libiconv with Exiv2 and sending me bug reports.

My priority at the moment is to write a book Image Metadata and Exiv2 Architecture and then retire (I'll be 70 in January). I hope to give a talk about the book at LGM in Rennes in May 2021 and to run an afternoon workshop. Current draft: https://clanmills.com/book/exiv2

[tester0077]

Hi all, this issue is very dear to my heart as well as I also have to deal with metadata text string in other than plain ASCII. I my case, these strings were not part of UserComment, but rather of some of the other metadata fields. As it turns out, I am likely the guy who Robin is thinking of & who sent him all those bug reports encountered during my 32-bit Windows build of Exiv2lib for my app. Some of these build problems were resolved by my using vcpkg to install some of the dependencies, but I have meanwhile backed away from vcpkg until I can sort out some of the issues I have run into in my setup. Vcpkg, when installed the default way, takes over all instances of all IDEs. For my environment, that is a problem & I have contacted MS, but have not had any resolution yet. Still, I have built a 32-bit exiv2lib for my app and if necessary, I might be able to help with getting it compiled for someone else. If there are any images with such problem string, I would not mind testing them on my app - mainly intended to be able to compare the metadata in two images. It is nowhere near finished, but the basic reading and display of the metadata works for my example files. Arnold

…

On 2020-08-04 3:53 AM, Robin Mills wrote: Well the list of things TODO never gets shorter. Not because I'm not making progress, it's because we live in an expanding universe! Here's something you could explore with your tools: #1255 <#1255> By default VCPKG builds 32-bit exiv2 and: 1 Doesn't build the sample applications 2 Doesn't copy the DLLs into build/bin It would be good to figure out how to run the test suite on that - especially when Leo finishes #1215 <#1215>. A matter closely related to #1255 <#1255> is building libiconv on Windows. I see that vcpkg builds libiconv and links it into Exiv2. I'd like to test that with the UserComment. I recently added CMake code to guarantee that Exiv2/MSVC never links libiconv on 0.27-maintenance. I did that because a user kept linking Cygwin64/libiconv with Exiv2 and sending me bug reports. My priority at the moment is to write a book *Image Metadata /and/ Exiv2 Architecture* and then retire (I'll be 70 in January). I hope to give a talk about the book at LGM in Rennes in May 2021 and to run an afternoon workshop. Current draft: https://clanmills.com/book/exiv2 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1258 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFCLPCR2TET4BKGOHSDKQLR67SBFANCNFSM4PTLERJQ>.

[clanmills]

Thanks Arnold (@tester0077). I butchered cmake/FindIconv.cmake specially for you! I am curious about using libiconv on Windows - however I don't want to get distracted from working on ISOBMFF (CR3 and HEIF) for the book this week.

By the way, I researched and documented both PSD and IPTC last week. I documented the mysterious "Iptc.Envelope.CharacterSet" https://clanmills.com/exiv2/book/#IPTC

I will revisit UserComment for the book. I haven't spoken the Chinese Contributor yet about doing CharSet testing. I'll speak to him when/if he finishes his current assignment.

The book is of course more work than I planned or expected. Initially, I though "if I document TiffVisitor, that's enough.". That's done, and now I'm documenting every last nook and cranny. I've no idea how Exiv2 interacts with the Adobe XMPsdk and only a vague idea about preview images. Lots of work ahead.

[tester0077]

Hi Robin

On 2020-08-04 9:33 AM, Robin Mills wrote: Thanks Arnold ***@***.*** <https://github.com/tester0077>). I butchered cmake/FindIconv.cmake specially for you! I am curious about using libiconv on Windows - however I don't want to get distracted from working on ISOBMFF (CR3 and HEIF) for the book this week.

Yes, there is some work to be done yet and I very much appreciate your support to get my version to compile & link. As I mentioned, I am a bit disappointed by vcpkg, at least in my environment. When it is installed and hooked up to the MSVC IDE, it takes over all tool chains it knows about, or so it seems. But when it is unhooked or uninstalled, to does not restore the state to anything like it was before and since it seems to do all its work 'under the covers', it takes some work to restore a sane status after it is out of the loop.

By the way, I researched and documented both PSD and IPTC last week. I documented the mysterious "Iptc.Envelope.CharacterSet" https://clanmills.com/exiv2/book/#IPTC

Those character sets are quite a handful. My understanding is that there are code sequences which allow shifting back and forth between character sets within one image. The biggest problem for me is to find images with these various character sets for testing and checking the values extracted. FWIW, an image I believe I found on the Exiv2 site of the USSRonaldReagan has, what I believe to be a bad Exif string for the 'Artist' caption. No idea how it was entered, but it can't be right.

The book is of course more work than I planned or expected.

It is always so ;-)

[clanmills]

I'm really good at bringing my projects in on schedule and to budget. And I seldom cut the spec. However I resist feature creep. In this case of the book, I shipped the v1 with v0.27.3. The book at that time was more-or-less what I had in mind to be the finished product. For LGM in Rennes and it'll be v5 and will document everything I know about Exiv2 and Metadata.

1157 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance $ exiv2 -pa --grep artist/i test/data/Reagan.jpg 
Exif.Image.Artist                            Ascii      34  Photographerís Mate 3rd Class (A
1158 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance $ 

I have the hi-res original of that in my Wallpaper's folder. I found it on the US Navy Website. Beautiful Photo. Using my utility dmpf.cpp (from the book) and tvisitor.cpp, here's what's in the file:

678 rmills@rmillsmm-local:~/gnu/exiv2/team/book/build $ ./dmpf '/Users/rmills/Google Drive/Wallpapers/USS Ronald Reagan.jpg' skip=$((30+654)) count=100 width=20
   0x2ac      684: Photographer..s Mate  ->  50 68 6f 74 6f 67 72 61 70 68 65 72 c3 ad 73 20 4d 61 74 65
   0x2c0      704:  3rd Class (A__&.._.  ->  20 33 72 64 20 43 6c 61 73 73 20 28 41 00 00 26 82 9a 00 05
   0x2d4      724: ___.__.~.._.___.__..  ->  00 00 00 01 00 00 04 7e 82 9d 00 05 00 00 00 01 00 00 04 86
   0x2e8      744: ."_.___._.__.__.___.  ->  88 22 00 03 00 00 00 01 00 01 00 00 90 00 00 07 00 00 00 04
   0x2fc      764: 0220.._.___.__...._.  ->  30 32 32 30 90 03 00 02 00 00 00 14 00 00 04 8e 90 04 00 02
679 rmills@rmillsmm-local:~/gnu/exiv2/team/book/build $ 

Somebody has used c3 ad as an apostrophe. I suspect "Adobe PhotoShop CS Macintosh".

I installed vcpkg by downloading the code from GitHub and building it myself. Visual Studio didn't seem to be impacted. Perhaps you installed it in a different way, or you've uncovered something I haven't noticied as I hardly ever use use Visual Studio these days. I used it all-day/every-day at Adobe for 10 years.

Although I've documented "Iptc.Envelope.CharacterSet", I don't think we should mess with that. However using charset encoding in UserComment is important and I'd like to be certain that it really works. I've tested it with UNICODE on Windows. However, it needs a serious workout by a native Nippon, Hindu, Mandarin or Arabic speaker.

[norbertj42]

I have now checked the image, which gives "charset=Ascii binary comment" as UserComment. The problem: the text contains German umlauts (ä, ö, ü) - and they are no Ascii characters. So the value does not fit to the charset.
Then I made a test with exiv2 0.27.3. I was able to write Usercomment with "charset=Ascii comment äöü". When reading with 0.27.3, it gave binary comment, whereas exiv2 0.27.2 returned the value with wrong representation of the non-Ascii-characters.
So for command line the new behaviour might be ok, but I still think it is a step backward for those using the library in a GUI, where the non-Ascii-characters can be displayed correct.

As I already mentioned before, I personally would prefer to have the behaviour of 0.27.2 back (not returning binary comment, interpreted value without leading charset information). But as I did not understand the details about #1046, I am not able to judge, if going back is a good solution. Anyhow, as exiv2 is Open Source, I am able to adjust the code for my needs.

An idea: makes is sense to reject writing charset=Ascii with non-Ascii-characters? This would ensure that exiv2 does not violate the specification and it is somehow strange, that exiv2 is able to write something, which it cannot read. As opinions my differ about this idea, the check could be disabled by a #define.

[tester0077]

Hi Norbert, Looks like we have the same issue with Umlauts ;-) Are you needing the command line version to handle these Umlauts or are you using exiv2lib in a program? I know the library can extract the string properly, though I have not yet tried to use it to write anything to an image, nor have I used the command line very much. Arnold

…

On 2020-08-05 11:06 AM, norbertj42 wrote: I have now checked the image, which gives "charset=Ascii binary comment" as UserComment. The problem: the text contains German umlauts (ä, ö, ü) - and they are no Ascii characters. So the value does not fit to the charset. Then I made a test with exiv2 0.27.3. I was able to write Usercomment with "charset=Ascii comment äöü". When reading with 0.27.3, it gave binary comment, whereas exiv2 0.27.2 returned the value with wrong representation of the non-Ascii-characters. So for command line the new behaviour might be ok, but I still think it is a step backward for those using the library in a GUI, where the non-Ascii-characters can be displayed correct. As I already mentioned before, I personally would prefer to have the behaviour of 0.27.2 back (not returning binary comment, interpreted value without leading charset information). But as I did not understand the details about #1046 <#1046>, I am not able to judge, if going back is a good solution. Anyhow, as exiv2 is Open Source, I am able to adjust the code for my needs. An idea: makes is sense to reject writing charset=Ascii with non-Ascii-characters? This would ensure that exiv2 does not violate the specification and it is somehow strange, that exiv2 is able to write something, which it cannot read. As opinions my differ about this idea, the check could be disabled by a #define. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1258 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFCLPGCOFKR26TM6FLGCW3R7GNTPANCNFSM4PTLERJQ>.

[clanmills]

Gentlemen. My understanding is that you can put any sequence of 8byte values into an Exif ascii tag. I believe the standard says that the ascii type is intended for 7-bit ascii values (32-127) and the should be nul terminated. (The nul is included in the count).

In this test file, the mysterious 2-byte encoding of the apostrophe came with the file from the internet.

1193 rmills@rmillsmbp:~/gnu/github/isobmff/pyke369/isobmffdump $ exiv2 -pa --grep Artist ~/gnu/github/exiv2/0.27-maintenance/test/data/Reagan.tiff 
Exif.Image.Artist                            Ascii      34  Photographerís Mate 3rd Class (A
1194 rmills@rmillsmbp:~/gnu/github/isobmff/pyke369/isobmffdump $ exiv2 -pa --grep Artist ~/gnu/github/exiv2/0.27-maintenance/test/data/Reagan.tiff  | od -a 
0000000    E   x   i   f   .   I   m   a   g   e   .   A   r   t   i   s
0000020    t  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp
0000040   sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp   A   s   c
0000060    i   i  sp  sp  sp  sp  sp  sp   3   4  sp  sp   P   h   o   t
0000100    o   g   r   a   p   h   e   r   ?   ?   s  sp   M   a   t   e
0000120   sp   3   r   d  sp   C   l   a   s   s  sp   (   A  nl        

Exiv2 is a metadata tool and not a metadata policemen. Exiv2 will not prevent you from putting other bytes into an ascii string. For that matter, you could define the tag "Exif.Image.Artist" to have Rational or other values:

$ exiv2 -M'set Exif.Image.Artist Rational 3/2 1/2' Reagan.tiff 
1206 rmills@rmillsmbp:~/temp $ exiv2 -pa --grep Artist Reagan.tiff 
Exif.Image.Artist                            Rational    2  3/2 1/2
$

UserComment is defined to have a Charset and a binary stream. The use of this feature is documented in the man page exiv2.1 which shipped with Exiv2 v0.27.3

@norbertj42 has raised the topic of interoperability with other applications. My first concern is to correctly implement the standard. If other applications are in conflict with the Exiv2 implementation, I am willing to investigate ways to accommodate them. However before dealing with interoperability, I would like the Exiv2 implementation to be tested by a native speaker of a language which requires UNICODE or other charset support.

[clanmills]

Guys: I'm out of sync. I've seen your messages in the wrong order. I'll investigate tomorrow.

[tester0077]

@clanmills
0x2ac 684: Photographer..s Mate -> 50 68 6f 74 6f 67 72 61 70 68 65 72 c3 ad 73 20 4d 61 74 65
===
FWIW, These 2 bytes are the encoding for the UTF-8 character sequence for 0xC3 0xAF.
See: https://www.utf8-chartable.de/ , which corresponds to the character of an i with accent aigu.

@norbertj42 can you share a test image with the UserComment and what string you expect to find in that field, as well as which app placed the text there?
I have been looking for a usable and trustworthy test image for the user comment, but have had no luck

[clanmills]

Reagan.tiff came from the internet with the UTF-8 data which is being correctly reported by Exiv2. I don't see any case to discuss here.

The specification of UserComment is a very different . It supports CharSet. I believe exiftool also supports CharSet. Can somebody compare the two, please? And I can't simply restore the previous behaviour because that would break the fix to #1046.