Micromanager: reader selection issue

[julou]

Hello,

Following your fixes to parse config-specific metadata in MM datasets, I realised that I really don't understand why you rely on the *_metadata.txt in order to parse the config-specific metadata…

In fact (and as we discussed at large), the exact same metadata is present in the TIF tag 51123 of the tif files… Did I get it right that you want to keep the OME-TIF reader as the default one for full MM datasets, i.e. when the input path is the first file of the dataset? If this is the case, this comes with 2 major drawbacks:

• the config-specific metadata are not parsed for datasets without *_metadata.txt files although they're valid MM datasets
• there is no way to use the MM reader for the first position and hence to open this position only using the same syntax as another position

Could we come up with a filename pattern criteria that would allow to use the MM reader for MM files (and parse the tif tag 51123)?
Maybe this was already discussed but because I faced this other issue with the new reader and large datasets spread over several files, I couldn't even make sense of this basic choice anymore…

Thank you for your attention. Best,
Thomas

[julou]

btw this would also fix my (yet unanswered) question on the OME-users mailing list a while ago: http://lists.openmicroscopy.org.uk/pipermail/ome-users/2016-March/005886.html

[sbesson]

Hi Thomas,

Following your fixes to parse config-specific metadata in MM datasets, I realised that I really
don't understand why you rely on the *_metadata.txt in order to parse the config-specific
metadata…

To be precise, as a result of the PR linked above, the config-specific metadata is actually parsed directly from the TIFF tag and not the metadata file. But you are right that the *_metadata.txt file is effectively used to identify the fileset as a Micro-Manager fileset and not an a OME-TIFF fileset.

In fact (and as we discussed at large), the exact same metadata is present in the TIF tag 51123 of the tif files… Did I get it right that you want to keep the OME-TIF reader as the default one for full MM datasets, i.e. when the input path is the first file of the dataset?
If this is the case, this comes with 2 major drawbacks:

• the config-specific metadata are not parsed for datasets without *_metadata.txt files although they're valid MM datasets
• there is no way to use the MM reader for the first position and hence to open this position only using the same syntax as another position

On a technical level, how a reader is selected for a given fileset is defined via rules implemented in IFormatReader.isThisType() for each reader. In the case of Micro-Manager image file stacks without metadata files, the first file is recognized as a TIFF file with an .ome.tiff extension and a valid OME-XML header stored in the ImageDescription tag. This condition is necessary and sufficient to identify it as an OME-TIFF fileset. As you rightfully mention, an implication is that the MM config-specific metadata is not parsed since the Micro-Manager TIFF tag is not part of the OME-TIFF specification.

Could we come up with a filename pattern criteria that would allow to use the MM reader for MM files (and parse the tif tag 51123)?

As mentioned in #2213 (comment), the main caveat of such a proposal is that it has larger implications in terms of support and maintenance of readers for any custom use of the OME-TIFF specification in the community and the cost of doing so is simply too high. Adding proper extension points for extended metadata like the acquisition one defined by Micro-Manager is certainly on the OME roadmap but implementing it still requires some discussion and scoping.

btw this would also fix my (yet unanswered) question on the OME-users mailing list a while ago: http://lists.openmicroscopy.org.uk/pipermail/ome-users/2016-March/005886.html

Some suggestions have been proposed in the answer to your post. Did you have any chance to explore any of these options and see if they would fit your purposes?

Best,
Sebastien

[julou]

Thanks for the clarifications. I guess that fixing the reader inconsistency discussed in #2308 (comment) will greatly simplify the situation!

Could we come up with a filename pattern criteria that would allow to use the MM reader for MM files (and parse the tif tag 51123)?

As mentioned in #2213 (comment), the main caveat of such a proposal is that it has larger implications in terms of support and maintenance of readers for any custom use of the OME-TIFF specification in the community and the cost of doing so is simply too high. Adding proper extension points for extended metadata like the acquisition one defined by Micro-Manager is certainly on the OME roadmap but implementing it still requires some discussion and scoping.

Not sure that I really understand the problem here. I was not asking to extend the OME-TIF reader, rather to use the MM reader for those datasets. I now realise that this can be done using an importer option.

Also, thanks for pointing me to josh's answer on the ome-user mailing list: i unfortunately overlooked it :(
Best,

Thomas

[sbesson]

No worries about the thread. Closing this reader selection issue as #2314 should unify the behavior.

[julou]

still commenting on a closed issue. hope it's not considered a terrible practice :(

Now that i've become a bit more familiar with the way OME-TIF is used by MM, I might be able to phrase a better request. What I find very confusing is that the active series is always the first one independent on which file has been given as input.
In my opinion, the OMETifReader should make all series found in the input file active (instead of the first series only). NB: this/these series might indeed be spread over several files…
Does this makes sense? is this a too much MM-centered viewpoint?

Thanks,
Thomas

[sbesson]

Hi Thomas,

no worries. The issue is mostly closed to remind us the multiple reader selection issue was addressed. Keeping the discussion opened for new proposed designs is completely fine.

At the moment, the strategy is to have the default series set to 0 at initialization across readers. We can certainly see your viewpoint but it has a few downsides at the Bio-Formats level:

• changing the default series policy should definitely be considered as a breaking change
• changing the default series for a particular reader would likely mean reviewing all readers and consider doing the same for consistency which is a high-cost implementation and maintenance burden
• concretely, setting the series based on the input file can be complex and very specific to each file format. For instance, limiting the discussion to the OME-TIFF format, your proposal has a unique non-null solution in the context of the Micro-Manager implementation where each OME-TIFF file maps to one series. It is unclear what should happen in the case of other implementations where files contain data belonging to multiple series, metadata-only files or others?

In the meantime, as you started in #2308 (comment), implementing a macro/wrapper to solve your use case might be the easiest. Note you could also use of the FormatReader#getSeriesUsedFiles() API to list all files used by a given series and compare it to the current id.

Best,
Sebastien