2008-09 to 2009-09: Move CustomAttributes transforms to top level

[dgault]

This is a follow up to #142 and should see the unit tests in ome/bioformats#3695 turn green.

The issue appears to have been due to CustomAttributes being able to be located in 4 different locations:

top level under OME
under images
under images/feature
under dataset
The non top level CustomAttributes were being transformed into XMLAnnotations but embedded in the original element, be it Image or Dataset, this is an invalid location for it and thus it was not being recognised when converted back to an OME-Model object. This PR aims to move all CustomAnnotations to the top OME level before transforming them. If you have multiple CustomAttributes at different levels then you will get multiple XMLAnnotations inside a single StructuredAnnotations.

Fixes #142

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/bio-formats-returns-all-metadata-except-ome-meta-data/52295/14

[sbesson]

Quickly tested using a few examples including the one from the original thread and the first impression is positive. For instance using the representative file from the original image.sc thread:

(base) sbesson@ls30630:~ $ diff cells_6.6.1.ome.xml cells_pr145.ome.xml 
13a14,76
>    <StructuredAnnotations>
>       <XMLAnnotation ID="Annotation:1">
>          <Value>
>             <OriginalMetadata ID="OriginalMetadata:0" Name="Acquisition Parameters/Acquisition GUID" Value="{2329CC5F-23E5-4D61-8AB1-526F349A3C9E}"/>
>             <OriginalMetadata ID="OriginalMetadata:1" Name="Acquisition Parameters/Channel 1" Value="Fluorescence                                                    "/>
>             <OriginalMetadata ID="OriginalMetadata:2" Name="Acquisition Parameters/Channel 1 Dye" Value="FITC                                                            "/>
>             <OriginalMetadata ID="OriginalMetadata:3" Name="Acquisition Parameters/Channel 2" Value="Fluorescence                                                    "/>
>             <OriginalMetadata ID="OriginalMetadata:4" Name="Acquisition Parameters/Channel 2 Dye" Value="Cy3                                                             "/>
>             <OriginalMetadata ID="OriginalMetadata:5" Name="Acquisition Parameters/Confocal Aperature-Ch2" Value="95.0�m"/>
>             <OriginalMetadata ID="OriginalMetadata:6" Name="Acquisition Parameters/Confocal Aperature-Ch3" Value="105.0�m"/>
>             <OriginalMetadata ID="OriginalMetadata:7" Name="Acquisition Parameters/Date" Value="02-17-2004"/>
>             <OriginalMetadata ID="OriginalMetadata:8" Name="Acquisition Parameters/DelayToFirstImagePixelInSecs" Value="0.02144"/>
>             <OriginalMetadata ID="OriginalMetadata:9" Name="Acquisition Parameters/Frame Filter" Value="2 frame Kalman Filter"/>
>             <OriginalMetadata ID="OriginalMetadata:10" Name="Acquisition Parameters/Gain Ch2" Value="1.000000"/>
>             <OriginalMetadata ID="OriginalMetadata:11" Name="Acquisition Parameters/Gain Ch3" Value="1.000000"/>
>             <OriginalMetadata ID="OriginalMetadata:12" Name="Acquisition Parameters/Intensity Name" Value="Intensity"/>
>             <OriginalMetadata ID="OriginalMetadata:13" Name="Acquisition Parameters/Intensity Resolution" Value=" 1"/>
>             <OriginalMetadata ID="OriginalMetadata:14" Name="Acquisition Parameters/Intensity Units" Value=" "/>
>             <OriginalMetadata ID="OriginalMetadata:15" Name="Acquisition Parameters/Laser-ND HeNe-G" Value="50.1% transmittance"/>
>             <OriginalMetadata ID="OriginalMetadata:16" Name="Acquisition Parameters/Laser-ND M_Ar2" Value="12.3% transmittance"/>
>             <OriginalMetadata ID="OriginalMetadata:17" Name="Acquisition Parameters/Laser-Shutter HeNe-G" Value="Open"/>
>             <OriginalMetadata ID="OriginalMetadata:18" Name="Acquisition Parameters/Laser-Shutter HeNe-R" Value="Closed"/>
>             <OriginalMetadata ID="OriginalMetadata:19" Name="Acquisition Parameters/Laser-Shutter LD405" Value="Closed"/>
>             <OriginalMetadata ID="OriginalMetadata:20" Name="Acquisition Parameters/Laser-Shutter M_Ar1" Value="Closed"/>
>             <OriginalMetadata ID="OriginalMetadata:21" Name="Acquisition Parameters/Laser-Shutter M_Ar2" Value="Open"/>
>             <OriginalMetadata ID="OriginalMetadata:22" Name="Acquisition Parameters/Laser-Shutter M_Ar3" Value="Closed"/>
>             <OriginalMetadata ID="OriginalMetadata:23" Name="Acquisition Parameters/Magnification" Value="40X"/>
>             <OriginalMetadata ID="OriginalMetadata:24" Name="Acquisition Parameters/Objective Lens" Value="UPLAPO 40XO"/>
>             <OriginalMetadata ID="OriginalMetadata:25" Name="Acquisition Parameters/Offset Ch2" Value="9"/>
>             <OriginalMetadata ID="OriginalMetadata:26" Name="Acquisition Parameters/Offset Ch3" Value="15"/>
>             <OriginalMetadata ID="OriginalMetadata:27" Name="Acquisition Parameters/PMT Voltage Ch2" Value="510"/>
>             <OriginalMetadata ID="OriginalMetadata:28" Name="Acquisition Parameters/PMT Voltage Ch3" Value="595"/>
>             <OriginalMetadata ID="OriginalMetadata:29" Name="Acquisition Parameters/Pan X" Value="76"/>
>             <OriginalMetadata ID="OriginalMetadata:30" Name="Acquisition Parameters/Pan Y" Value="0"/>
>             <OriginalMetadata ID="OriginalMetadata:31" Name="Acquisition Parameters/Scan Speed" Value="8000"/>
>             <OriginalMetadata ID="OriginalMetadata:32" Name="Acquisition Parameters/Scan Start Mode" Value="Scan started immediately"/>
>             <OriginalMetadata ID="OriginalMetadata:33" Name="Acquisition Parameters/ScanMode" Value="Standard Scan"/>
>             <OriginalMetadata ID="OriginalMetadata:34" Name="Acquisition Parameters/ScanRotationAngle" Value="1�"/>
>             <OriginalMetadata ID="OriginalMetadata:35" Name="Acquisition Parameters/SecondsPerScanLine" Value="0.010368"/>
>             <OriginalMetadata ID="OriginalMetadata:36" Name="Acquisition Parameters/Sequential Scanning Mode" Value="Line-sequential"/>
>             <OriginalMetadata ID="OriginalMetadata:37" Name="Acquisition Parameters/SourceOfData" Value="PMT 2 PMT 3 "/>
>             <OriginalMetadata ID="OriginalMetadata:38" Name="Acquisition Parameters/System Configuration" Value="FV500"/>
>             <OriginalMetadata ID="OriginalMetadata:39" Name="Acquisition Parameters/Time" Value="11:54:50"/>
>             <OriginalMetadata ID="OriginalMetadata:40" Name="Acquisition Parameters/XY Resolution-Ch2" Value="0.23�m"/>
>             <OriginalMetadata ID="OriginalMetadata:41" Name="Acquisition Parameters/XY Resolution-Ch3" Value="0.25�m"/>
>             <OriginalMetadata ID="OriginalMetadata:42" Name="Acquisition Parameters/Z Resolution-Ch2" Value="0.57�m"/>
>             <OriginalMetadata ID="OriginalMetadata:43" Name="Acquisition Parameters/Z Resolution-Ch3" Value="0.62�m"/>
>             <OriginalMetadata ID="OriginalMetadata:44" Name="Acquisition Parameters/Zoom Size" Value="3.000000"/>
>             <OriginalMetadata ID="OriginalMetadata:45" Name="Display Settings/DisplayMode" Value=" 545"/>
>             <OriginalMetadata ID="OriginalMetadata:46" Name="Display Settings/DisplayZoom" Value=" 1"/>
>             <OriginalMetadata ID="OriginalMetadata:47" Name="Display Settings/EnabledSourceForView0" Value=" 1 2"/>
>             <OriginalMetadata ID="OriginalMetadata:48" Name="Display Settings/EnabledSourceForView1" Value=" 2"/>
>             <OriginalMetadata ID="OriginalMetadata:49" Name="Display Settings/Gamma 0" Value="1"/>
>             <OriginalMetadata ID="OriginalMetadata:50" Name="Display Settings/Gamma 1" Value="1"/>
>             <OriginalMetadata ID="OriginalMetadata:51" Name="Display Settings/NumberOfViews" Value=" 1"/>
>             <OriginalMetadata ID="OriginalMetadata:52" Name="Display Settings/NumberOfViewsAvailable" Value=" 2"/>
>             <OriginalMetadata ID="OriginalMetadata:53" Name="Intensity Mapping/Map Ch0: Range" Value="00000 to 04095"/>
>             <OriginalMetadata ID="OriginalMetadata:54" Name="Intensity Mapping/Map Ch1: Range" Value="00000 to 04095"/>
>             <OriginalMetadata ID="OriginalMetadata:55" Name="Version Info/FLUOVIEW Version" Value="Version 4.3.53, w/ TIEMPO"/>
>             <OriginalMetadata ID="OriginalMetadata:56" Name="Version Info/File Version" Value="4."/>
>          </Value>
>       </XMLAnnotation>
>    </StructuredAnnotations>

The CustomAttributes previously lost during the transformation is now successfully converted into an XMLAnnotation.
I will perform additional testing on the representative samples.

Since the StructuredAnnotation is a top-level element, my only concern at this stage is that the transformation might still be losing the relationship with the parent element, in the use case above this would be the Image.CustomAttributes relationship. Do you think creating aAnnotationRef as part of the transform would be feasible?

[sbesson]

A few failures in https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-test-folder/127065. Additionally testing locally gives the unfortunate error

cvc-pattern-valid: Value 'CustomAttributesAnnotation:1' is not facet-valid with respect to pattern '(urn:lsid:([\w\-\.]+\.[\w\-\.]+)+:Annotation:\S+)|(Annotation:\S+)' for type 'AnnotationID'.

so only LSID-style prefixes seem to be valid

[sbesson]

Tested against a variety of sample OME-TIFF files from our curated QA repository including the sample OME-TIFF from the original image.sc thread. Below is a summary table including the file paths, number and type of CustomAttributes and the transformed elements with and without this PR:

File path	OME-XML	Bio-Formats 6.6.1	with this PR
christian/lsfm_110404_005_x00_y00_ch00.tif	OME.CustomAttributes (1)	XMLAnnotation (1)	XMLAnnotation (1)
ruben/Test_Ruben_001/W0002/P002/	Image.CustomAttributes (3 empty)
peter/LA_ProtocolCombination_23.01.2012_16_29_52_1.ome.tif	Region.CustomAttributes (1) Image.CustomAttributes (1 empty) OME.CustomAttributes (1)	XMLAnnotation (1)	XMLAnnotation (2)
oliver/LA_ProtocolCombination_12.10.2009_21_40_58_Marker_4__6.tif	Region.CustomAttributes (1) Image.CustomAttributes (1 empty) OME.CustomAttributes (1)	XMLAnnotation (1)	XMLAnnotation (2)
oliver/DIV0Ctrl-2_1.tif	Region.CustomAttributes (1) Image.CustomAttributes (1 empty) OME.CustomAttributes (1)	XMLAnnotation (1)	XMLAnnotation (2)
oliver/DIV0Ctrl-2_3.tif	Region.CustomAttributes (1) Image.CustomAttributes (1 empty) OME.CustomAttributes (1)	XMLAnnotation (1)	XMLAnnotation (2)
prairie/TSeries-12062007-1400-093/	Image.CustomAttributes (1)		XMLAnnotation (1) AnnotationRef (1)
cells.ome.tiff	Image.CustomAttributes (1)		XMLAnnotation (1) AnnotationRef (1)

Apart from Dataset.CustomAttributes, this provides a good sampling of the various elements that can be annotated with CustomAttributes

• OME.CustomAttributes was properly handled prior to this change, generated XMLAnnotations without AnnotationRef and the behavior is unchanged by this PR
• Region.CustomAttributes were previously lost and are now transformed into XMLAnnotations without AnnotationRef as expected since the OME.Region element has been dropped
• ImageCustomAttributes were previously lost and are now transformed into XMLAnnotations with AnnotationRef on the Image

The ID of the XMLAnnotation is now transformed into Annotation:CustomAttributes1. Although this is a bit more convoluted than a simple indexing, it should cater for the case where both XMLAnnotation and CustomAttributes might be present.

Overall, I think this addresses the outstanding issue of the metadata loss during transformation, preserving the Image/Annotation linkage where appropriate. Barring further objections from @ome/formats, I propose to get this merged tomorrow and cut a patch release of ome-model for inclusion in Bio-Formats 6.7.0.