Fix the OME-XML writer to write valid files

[melissalinkert]

See #246 and
https://www.openmicroscopy.org/community/viewtopic.php?f=13&t=3499

Testing involves everything from PR #246, but also verifying that xmlvalid on the converted file shows no errors.

[sbesson]

Using the command-line tools provided with the 4.4.5 Homebrew formula

sbesson:david sebastien$ bfconvert -version
Version: 4.4.5
VCS revision: 03770c0
Build date: 1 December 2012
sbesson:david sebastien$ bfconvert Image0016.oib Image0016_4_4_5.ome
...
19.379s elapsed (2.1666667+7.5ms per plane, 515ms overhead)
sbesson:david sebastien$ grep Image\ ID Image0016_4_4_5.ome 
<Image ID="Image:0" Name="Series 1">
sbesson:david sebastien$ xmlvalid Image0016_4_4_5.ome 
Parsing schema path
http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd
Validating Image0016_4_4_5.ome
cvc-complex-type.2.4.a: Invalid content was found starting with element 'Channel'. One of '{"http://www.openmicroscopy.org/Schemas/BinaryFile/2012-06":BinData, "http://www.openmicroscopy.org/Schemas/OME/2012-06":Plane, "http://www.openmicroscopy.org/Schemas/SA/2012-06":AnnotationRef}' is expected.
Error validating document: 1 errors found

Using BIOFORMATS-per-commit-4.4#52 artifacts (see #246)

sbesson:david sebastien$ curl -O http://hudson.openmicroscopy.org.uk/view/4.4/job/BIOFORMATS-per-commit-4.4/52/artifact/artifacts/bftools.zip
...
sbesson:david sebastien$ unzip bftools.zip -d bftools_246/
...
sbesson:david sebastien$ curl -O http://hudson.openmicroscopy.org.uk/view/4.4/job/BIOFORMATS-per-commit-4.4/52/artifact/artifacts/loci_tools.jar
...
sbesson:david sebastien$ mv loci_tools.jar bftools_246/
sbesson:david sebastien$ ./bftools_246/bfconvert -version
Version: 4.4.6-DEV
VCS revision: d2e1325
Build date: 28 November 2012
sbesson:david sebastien$ ./bftools_246/bfconvert Image0016.oib Image0016_246.ome
...
19.291s elapsed (1.8333334+7.3333335ms per plane, 499ms overhead)
sbesson:david sebastien$ ./bftools_246/xmlvalid 
sbesson:david sebastien$ grep Image\ ID Image0016_246.ome 
<Image ID="Image:0" Name="Series 1">
<Image ID="Image:1" Name="Series 2">
sbesson:david sebastien$ ./bftools_246/xmlvalid Image0016_246.ome 
Parsing schema path
http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd
Validating Image0016_246.ome
cvc-complex-type.2.4.a: Invalid content was found starting with element 'BinData'. One of '{"http://www.openmicroscopy.org/Schemas/ROI/2012-06":ROIRef, "http://www.openmicroscopy.org/Schemas/OME/2012-06":MicrobeamManipulationRef, "http://www.openmicroscopy.org/Schemas/SA/2012-06":AnnotationRef}' is expected.
cvc-complex-type.2.4.a: Invalid content was found starting with element 'BinData'. One of '{"http://www.openmicroscopy.org/Schemas/ROI/2012-06":ROIRef, "http://www.openmicroscopy.org/Schemas/OME/2012-06":MicrobeamManipulationRef, "http://www.openmicroscopy.org/Schemas/SA/2012-06":AnnotationRef}' is expected.
Error validating document: 2 errors found

Using BIOFORMATS-per-commit-4.4#62 artifacts

sbesson:david sebastien$ curl -O http://hudson.openmicroscopy.org.uk/view/4.4/job/BIOFORMATS-per-commit-4.4/62/artifact/artifacts/bftools.zip
...
sbesson:david sebastien$ unzip bftools.zip -d bftools_265/
...
sbesson:david sebastien$ curl -O http://hudson.openmicroscopy.org.uk/view/4.4/job/BIOFORMATS-per-commit-4.4/62/artifact/artifacts/loci_tools.jar
...
sbesson:david sebastien$ mv loci_tools.jar bftools_265
sbesson:david sebastien$ ./bftools_265/bfconvert -version
Version: 4.4.6-DEV
VCS revision: a7e34a4
Build date: 4 December 2012
sbesson:david sebastien$ ./bftools_265/bfconvert Image0016.oib Image0016_265.ome
...
46.745s elapsed (2.6666667+9.0ms per plane, 548ms overhead)
sbesson:david sebastien$ grep Image\ ID Image0016_265.ome 
<Image ID="Image:0" Name="Series 1">
<Image ID="Image:1" Name="Series 2">
sbesson:david sebastien$ ./bftools_265/xmlvalid Image0016_265.ome 
Parsing schema path
http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd
Validating Image0016_265.ome
No validation errors found

The test describe above passes with this PR merged in.
Note the conversion time increased significantly between #246 and this PR. Is that expected?

[melissalinkert]

A slight increase in time is expected, but 30 seconds seems excessive. I'll try to rewrite the fix so that it is faster.

[melissalinkert]

Should be faster now.

[sbesson]

lifesci-2635:david sebastien$ cd ~/code/openmicroscopy/components/bioformats/
lifesci-2635:bioformats sebastien$ git fetch origin && git fetch melissalinkert
lifesci-2635:bioformats sebastien$ git checkout origin/dev_4_4 && git merge melissalinkert/valid-ome-xml 
...
lifesci-2635:bioformats sebastien$ git graph HEAD^..HEAD
* e93b8f1 (HEAD) Merge remote-tracking branch 'melissalinkert/valid-ome-xml' into HEAD
* b4388ba (melissalinkert/valid-ome-xml) Make setId a no-op if called on the currently initialized file
* 23d3941 Fix the OME-XML writer to write validating files
lifesci-2635:bioformats sebastien$ ant clean tools utils-formats
...
lifesci-2635:david sebastien$ unzip ~/code/openmicroscopy/components/bioformats/artifacts/bftools.zip -d bftools_265/
Archive:  /Users/sebastien/code/openmicroscopy/components/bioformats/artifacts/bftools.zip
  inflating: bftools_265/bfconvert   
  inflating: bftools_265/bfconvert.bat  
  inflating: bftools_265/bfview      
  inflating: bftools_265/bfview.bat  
  inflating: bftools_265/domainlist  
  inflating: bftools_265/domainlist.bat  
  inflating: bftools_265/editor      
  inflating: bftools_265/editor.bat  
  inflating: bftools_265/formatlist  
  inflating: bftools_265/formatlist.bat  
  inflating: bftools_265/ijview      
  inflating: bftools_265/ijview.bat  
  inflating: bftools_265/notes       
  inflating: bftools_265/notes.bat   
  inflating: bftools_265/omeul       
  inflating: bftools_265/omeul.bat   
  inflating: bftools_265/showinf     
  inflating: bftools_265/showinf.bat  
  inflating: bftools_265/tiffcomment  
  inflating: bftools_265/tiffcomment.bat  
  inflating: bftools_265/xmlindent   
  inflating: bftools_265/xmlindent.bat  
  inflating: bftools_265/xmlvalid    
  inflating: bftools_265/xmlvalid.bat  
  inflating: bftools_265/log4j.properties  
lifesci-2635:david sebastien$ cp ~/code/openmicroscopy/components/bioformats/artifacts/loci_tools.jar bftools_265/
lifesci-2635:david sebastien$ ./bftools_265/bfconvert -version
Version: 4.4.6-DEV
VCS revision: e93b8f1
Build date: 7 December 2012
lifesci-2635:david sebastien$ ./bftools_265/bfconvert Image0016.oib Image0016_265.ome
Image0016.oib
Initializing helper readers
Reading additional metadata
Populating metadata
Reading bitmap header
Populating metadata
Unknown LaserMedium value 'Alexa Fluor 405' will be stored as "Other"
Expected positive value for EmissionWavelength; got 0
Expected positive value for ExcitationWavelength; got 0
Unknown LaserMedium value '' will be stored as "Other"
Expected positive value for EmissionWavelength; got 0
Expected positive value for ExcitationWavelength; got 0
Unknown LaserMedium value '' will be stored as "Other"
[Olympus FV1000] -> Image0016_265.ome [OME-XML]
    Series 0: converted 1/3 planes (33%)
    Series 0: converted 3/3 planes (100%)
    Series 1: converted 3/3 planes (100%)
[done]
8.455s elapsed (1.6666666+8.0ms per plane, 553ms overhead)
lifesci-2635:david sebastien$ grep Image\ ID Image0016_265.ome
<Image ID="Image:0" Name="Series 1">
<Image ID="Image:1" Name="Series 2">
lifesci-2635:david sebastien$ ./bftools_265/xmlvalid Image0016_265.ome
Parsing schema path
http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd
Validating Image0016_265.ome
No validation errors found.

Looks good now. bfconvert even increased in performance thanks to b4388ba . Unless you want BIOFORMATS-per-commit-4.4 to be green again (which depends on other PRs), I think this is ready to merge.

[melissalinkert]

BIOFORMATS-per-commit-4.4 is green; merging Monday afternoon.