Import Plate/Image name metadata

[chris-allan]

The issue

The current state of how Image and Plate name are treated in the importer and the historical lack of trust put in the metadata coming out of Bio-Formats is becoming a problem.

With respect to Plate name, unless it is explicitly chosen by the user using OMERO.cli, it will be set to the Bio-Formats target filename and any Plate name metadata is ignored. This is an issue when the desired name is deep within the metadata, requiring users to perform their own metadata extraction in order to set the name. It is also particularly problematic when the Plate count being imported is large effectively necessitating that a user write shell scripts in order to import their data correctly.

The same holds true for a singular Image. If the Image is part of a multi-image fileset (MIF) its series number is used as part of the name and any Bio-Formats name metadata is ignored. This is an issue when, using images of the digital pathology domain as an example, each Image has a self describing name which ends up effectively being lost.

History and state of play

The processor that enforces the aforementioned rule on Image:

• https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.5/components/blitz/src/ome/formats/model/PixelsProcessor.java

For OMERO 5.0.0 (ome/openmicroscopy#1710) the filename, rather than the absolute path became used as the default name unless the user specifies one explicitly.

This is also when the Bio-Formats Image name metadata began being ignored for multi-image filesets. The rationale for this is not completely clear to me. Length restrictions were also put in place, substituting ellipses when the name is longer than 255 characters.

From OMERO 4.3.0, (ome/openmicroscopy@98691605), the user specified name has defaulted to the absolute path and became just the filename in ome/openmicroscopy#1710. The rationale, albeit not a particularly thorough, is outlined in this corresponding ticket:

• https://trac.openmicroscopy.org/ome/ticket/5575

The processor that enforces the aforementioned rule on Plate:

• https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.5/components/blitz/src/ome/formats/model/WellProcessor.java

Also for OMERO 5.0.0 (ome/openmicroscopy#577; ome/openmicroscopy@92ec095b), the user specified Image and Plate name were unified. This had the potential unintended side effect, as the user specified Image name was the absolute path and became just the filename in ome/openmicroscopy#1710, of always setting a user specified Plate name regardless of whether the user had conciously made that decision or not.

From OMERO 4.2.1 (ome/openmicroscopy@87db20ab), a user specified Plate name has been possible.

Current work on the issue

• https://trac.openmicroscopy.org/ome/ticket/11894
• https://trac.openmicroscopy.org/ome/ticket/13231
• Rework plate naming priorities so that reader-supplied names can be used glencoesoftware/openmicroscopy#2

Suggestions for moving forward

OMERO 5.2.x

No change.

OMERO 5.3.x

• Trust Plate name metadata provided by Bio-Formats unless the user has specifically overriden it. If such metadata is missing or blank use the current default: filename.
• Trust Image name metadata provided by Bio-Formats when imported under multi-image fileset (MIF) conditions. If such metadata is missing or blank use the current default: filename.

/cc @jburel, @joshmoore, @sbesson, @melissa-linkert, @mtbc, @will-moore

/cc @emilroz

Edits

Mon 19 Sep 2016 16:23:36 BST: Add ongoing work from @melissa-linkert, @joshmoore, @mtbc
Wed 21 Sep 2016 09:22:21 BST: Reference to ongoing work on relaxing length restrictions on names

[jburel]

The length restriction was previously discussed but not removed
this will require some db adjustments
see https://trac.openmicroscopy.org/ome/ticket/11894

[mtbc]

http://trac.openmicroscopy.org/ome/ticket/11894 discusses the length restrictions. Happy to switch to TEXT in the DB any properties that we like. (See distinction between string and text on http://www.openmicroscopy.org/site/support/omero5.3-staging/developers/Model/EveryObject.html) Perhaps one may reasonably prioritize properties for which the corresponding OME-XML attribute is already of unrestricted length.

[chris-allan]

Current "named" objects, which consequently have a character varying database column of length 255 reserved to store a name property, are as follows:

• Image
• OriginalFile
• StageLabel
• Project
• Dataset
• Folder (yes, @sbesson :))
• Namespace (also displayname)
• Screen
• Plate

Exceptions to this include ExperimenterGroup which is not "named" but has a name character varying database column of length 255.

[sbesson]

Update on this design issue:

• OMERO 5.3.0 (via Use reader-supplied plate names by default openmicroscopy#4992) included the first part of the suggestion in the issue description i.e. trust the Plate name read by Bio-Formats at import time

• a few threads have been recently raised recently about the remaining issue i.e. trusting the Image name read by Bio-Formats at import time

  • for non-screen data, idr0040 is an example where the image name (read by MicroManager: use PositionName key/value for storing the ImageName (rebased onto metadata54) bioformats#3092) needs to be overriden by -n option via a bulk file

  • for HCS data, see http://lists.openmicroscopy.org.uk/pipermail/ome-users/2018-April/007001.html

[mtbc]

Created [trello] ignores plate names in naming fields for the issue in PixelsProcessor relating to the above community thread.

[sbesson]

See also https://www.openmicroscopy.org/community/viewtopic.php?f=4&p=19976#p19976 for another community use case where allowing to read the ImageName from Bio-Formats in the non-HCS behavior would be useful.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/upload-issue-with-metadata-ome-xml-companion-file/59290/8