Update database from selected xmp files not working for tags

[craigtaranto]

Describe the bug
When I update tags within XMP sidecar files in another application (eg. Digikam), darktable does not appear to update its database even though I select "update datebase from selected xmp files" within the "updated xmp sidecar files found" dialog when starting darktable.

This is also reproducible without another application. See steps below.

To Reproduce
A pre-requisite is that both Digikam and darktable have their own libraries generated from the images in the same folder. I created the Digikam library first (no XMP files were created or existed). I then created darktable's library on the same folder, which created XMP files for each image. darktable is configured to look for updated XMP files on start. Digikam is configured to read and write XMP sidecar files. For tags in particular, Digikam is configured to read and write both Xmp.dc.subject and Xmp.lr.hierarchicalSubject.

1. Ensure both Digikam and darktable are closed.
2. In Digikam, select a photo and add two tags.
3. Close Digikam. At this time, I confirm the tags are in both sections of the XMP file noted above.
4. Open darktable
5. At the "updated xmp sidecar files found" dialog, select the row and click "update database from selected xmp files".
6. darktable does the right thing and imports the two tags.
7. Close darktable
8. Open Digikam
9. Select the same image
10. Remove one of the tags and select Apply
11. Close Digikam. At this time, I confirm that both sections of the XMP file contain the one tag.
12. Open darktable
13. At the "updated xmp sidecar files found" dialog, select the row and click "update database from selected xmp files".
14. darktable continues to show two tags.
15. Close darktable
16. The XMP sidecar file has both tags in both sections.

An alternative way to reproduce without another application:

1. Open darktable
2. Assign two tags to an image
3. Close darktable
4. Edit the XMP sidecar file (using vi or similar editor) and remove one of the tags from both Xmp.dc.subject and Xmp.lr.hierarchicalSubject sections
5. Open darktable
6. At the "updated xmp sidecar files found" dialog, select the row and click "update database from selected xmp files".
7. darktable continues to show two tags
8. Close darktable
9. The XMP sidecar file has both tags in both sections.

It appears that darktable didn't update the database. It certainly updated the XMP files when it shouldn't have.

Expected behavior
darktable should update the database with the new XMP sidecar file content. In the case above, the darktable database should have been updated to reflect one tag and should not have written the 2nd tag back to the XMP file.

Screenshots
Let me know if needed

Platform (please complete the following information):

• Darktable Version: 3.0.2
• OS: Manjaro Linux
• OpenCL activated
• Asus GTX Strix 1660 Super. Nvidia proprietary drivers.

Additional context
All packages have been installed from the Manjaro repo (nothing compiled by me).

Sample XMP:

<?xml version="1.0" encoding="UTF-8"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0-Exiv2">
 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <rdf:Description rdf:about=""
    xmlns:MicrosoftPhoto="http://ns.microsoft.com/photo/1.0/"
    xmlns:acdsee="http://ns.acdsee.com/iptc/1.0/"
    xmlns:exif="http://ns.adobe.com/exif/1.0/"
    xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/"
    xmlns:tiff="http://ns.adobe.com/tiff/1.0/"
    xmlns:xmp="http://ns.adobe.com/xap/1.0/"
    xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:lr="http://ns.adobe.com/lightroom/1.0/"
    xmlns:darktable="http://darktable.sf.net/"
   MicrosoftPhoto:Rating="0"
   acdsee:rating="0"
   exif:ApertureValue="196608/65536"
   exif:ColorSpace="1"
   exif:CustomRendered="0"
   exif:DateTimeOriginal="2020:01:26 21:22:42"
   exif:ExifVersion="0230"
   exif:ExposureBiasValue="0/1"
   exif:ExposureMode="0"
   exif:ExposureProgram="3"
   exif:ExposureTime="1/160"
   exif:FNumber="28/10"
   exif:FlashpixVersion="0100"
   exif:FocalLength="50/1"
   exif:FocalPlaneResolutionUnit="2"
   exif:FocalPlaneXResolution="5184000/907"
   exif:FocalPlaneYResolution="3456000/595"
   exif:MeteringMode="5"
   exif:PixelXDimension="5184"
   exif:PixelYDimension="3456"
   exif:SceneCaptureType="0"
   exif:ShutterSpeedValue="483328/65536"
   exif:WhiteBalance="0"
   photoshop:DateCreated="2020-01-26T21:22:42.92"
   tiff:Compression="6"
   tiff:ImageLength="3456"
   tiff:ImageWidth="5184"
   tiff:Make="Canon"
   tiff:Model="Canon EOS 7D"
   tiff:Orientation="1"
   tiff:ResolutionUnit="2"
   tiff:XResolution="72/1"
   tiff:YResolution="72/1"
   xmp:CreateDate="2020-01-26T21:22:42.92"
   xmp:ModifyDate="2020-01-26T21:22:42.92"
   xmp:Rating="0"
   xmpMM:DerivedFrom="IMG_1229.CR2"
   darktable:xmp_version="3"
   darktable:raw_params="0"
   darktable:auto_presets_applied="0"
   darktable:history_end="0"
   darktable:iop_order_version="0">
   <exif:ComponentsConfiguration>
    <rdf:Seq>
     <rdf:li>1</rdf:li>
     <rdf:li>2</rdf:li>
     <rdf:li>3</rdf:li>
     <rdf:li>0</rdf:li>
    </rdf:Seq>
   </exif:ComponentsConfiguration>
   <exif:Flash
    exif:Fired="True"
    exif:Function="False"
    exif:Mode="1"
    exif:RedEyeMode="False"
    exif:Return="0"/>
   <exif:ISOSpeedRatings>
    <rdf:Seq>
     <rdf:li>1250</rdf:li>
    </rdf:Seq>
   </exif:ISOSpeedRatings>
   <tiff:BitsPerSample>
    <rdf:Seq>
     <rdf:li>8 8 8</rdf:li>
    </rdf:Seq>
   </tiff:BitsPerSample>
   <dc:subject>
    <rdf:Bag>
     <rdf:li>tag1</rdf:li>
     <rdf:li>tag2</rdf:li>
    </rdf:Bag>
   </dc:subject>
   <lr:hierarchicalSubject>
    <rdf:Bag>
     <rdf:li>tag1</rdf:li>
     <rdf:li>tag2</rdf:li>
    </rdf:Bag>
   </lr:hierarchicalSubject>
   <darktable:masks_history>
    <rdf:Seq/>
   </darktable:masks_history>
   <darktable:history>
    <rdf:Seq/>
   </darktable:history>
  </rdf:Description>
 </rdf:RDF>
</x:xmpmeta>

[phweyland]

What does happen if you add a new tag instead of removing one ?

[craigtaranto]

Adding a new tag (in vi) works.

I added "tag3" in vi, started darktable, and it showed up. I then closed darktable and deleted "tag3" in vi. When I reopened darktable, "tag3" still appeared. Closed darktable and "tag3" was back in the file.

[phweyland]

Adding a new tag (in vi) works.

I added "tag3" in vi, started darktable, and it showed up. I then closed darktable and deleted "tag3" in vi. When I reopened darktable, "tag3" still appeared. Closed darktable and "tag3" was back in the file.

Thanks for trying.
If I understand correctly, the xmp tags are correctly read, but a prior cleaning is missing.
I'll look at this.

[phweyland]

BTW, we could have the same issue for metadata ...

[craigtaranto]

Let me know if there's something you'd like me to try.

[phweyland]

Let me know if there's something you'd like me to try.

For the time being no, thank you. If I succeed in reproducing this I think that shouldn't be too difficult.

[phweyland]

This is not only the case for tags, but also for metadata and colors. Removing an item from xmp is not updated by the crawler detection (only add new item). I'll fix that.

@TurboGit but ... history stack / load sidecar file doesn't read the xmp data (except history) and worse, rewrites dt database into the xmp file. The xmp file changes are then lost. This because of the history_only flag which is set.
I think we should remove the history_only flag in that case. Do you know any reason NOT to read all xmp information there ?
EDIT I see a reason. load sidecar file can be done over several images. The user may want to copy only the history, the copy of other xmp data may not be valid. So load sidecar file MUST not be used for updating xmp metadata.... :(