Micromanager: fix multi-file image stacks and dimensionorder

[sbesson]

Fixes #2308

This Pull Request fixes the MicroManagerReader reader to handle large datasets saved as image stack and split over multiple files (i.e. datasets larger than 4GB). While testing a sample dataset, a second issue was fixed by improving the way each image dimensionality is parsed from the metadata file.

Summary of changes

• in parsePosition(), the number of IFDs were added to the current plane at the end of each loop as well as in the call to parseKeyValue() causing an ArrayIndexOutOfBoundsException when dealing with multiple files
• the dimensionOrder was previously unconditionally set to XYZCT causing wrong files to be returned by Position.getFile() under some conditions. For the sample multi-file dataset, this caused parsePosition() to scan twice the IFDs of the same file and parseKeyValue() to fail with an ArrayIndexOutOfBoundsException. Instead we make use of the SlicesFirst metadata value as documented in https://micro-manager.org/wiki/Micro-Manager_File_Formats#Writing_Images to store the correct DimensionOrder

Testing

In addition to the existing datasets available under curated/micromanager, a new large fileset was generated using Micro-Manager 1.4.22 with the demo config with 3 channels, 21 z-sections and 140 timepoints composed of 2 OME-TIFF files and a metadata text file. This fileset will be migrated to the public folder and be available under http://downloads.openmicroscopy.org/images/Micro-Manager/1.4.22/.

This fileset should fail to open without this PR and open properly with this PR included with the dimensions indicated above (might require to bump the default memory if using the command-line tools). Additionally, the OMERO workflow should be tested as the fileset should be importable and viewable.

The configuration files for a couple of filesets will be updated separately to match the DimensionOrder changes described above. Some attention should be put on curated/micromanager/nico/Untitled_4_multi_file/ where the Plane metadata was previously incorrectly populated and should now be fixed by this change. The expected deltaT and position value can be checked using Micro-Manager.

[dgault]

The code changes here look fine and I can reproduce and see the fixes in the plane metadata for incorrect ordering.

I am unable to reproduce the ArrayIndexOutOfBoundsException though when using the dataset from http://downloads.openmicroscopy.org/images/Micro-Manager/1.4.22/sebastien/ or samples from curated. Am I testing with the wrong filesets?

[sbesson]

@dgault: to reproduce the exception, you could either use the 5.1.10 command-line tools (you might have to up BF_MAX_MEM) or try to import the dataset into OMERO without this PR included. Since this is a MM image file stack, you need to point at the *_metadata.txt file from the dataset to pick the correct reader, e.g.

$ BF_MAX_MEM=4g ~/bftools/showinf -nopix /ome/data_repo/public/Micro-Manager/1.4.22/sebastien/Test_Dataset_MultiPosition_Stack_MMStack_Pos0_metadata.txt 
Checking file format [Micro-Manager]
Initializing reader
MicromanagerReader initializing /ome/data_repo/public/Micro-Manager/1.4.22/sebastien/Test_Dataset_MultiPosition_Stack_1_MMStack_Pos0_metadata.txt
Reading metadata file
Populating metadata
Finding image file names
Building list of TIFFs
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 66015625
    at loci.formats.in.MicromanagerReader.parseKeyAndValue(MicromanagerReader.java:576)
    at loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:544)
    at loci.formats.in.MicromanagerReader.initFile(MicromanagerReader.java:298)
    at loci.formats.FormatReader.setId(FormatReader.java:1426)
    at loci.formats.ImageReader.setId(ImageReader.java:835)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:651)
    at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1028)
    at loci.formats.tools.ImageInfo.main(ImageInfo.java:1117)

[dgault]

Ah that makes sense now, I wasn't pointing to the txt file. I can now reproduce the ArrayIndexOutOfBoundsException on 5.1.10 and verify that it is no longer present with this PR in place.

I've also retested the Plane metadata on curated/micromanager/nico/Untitled_4_multi_file and can confirm that it has been updated correctly. Lastly the images also display as expected.

PR should be good to merge.

[julou]

hi sebastien,
can I test this by adding the OME repository in fiji's downloader? or how to get a preview of BF5.2?

thanks,
Thomas

[dgault]

Hi Thomas,

The Fiji updater at the moment only supports 5.1 builds. If you wish to test this you can download the latest 5.2 build from http://downloads.openmicroscopy.org/bio-formats/5.2.0-m5/

To update Fiji you will need to replace the jars in the folder jars/bioformats and the bio-formats_plugins.jar under the plugins directory.

Note: The 5.2 builds are still in internal development and are not fully supported

David