Evaluate Omero as a storage backend

[jluethi]

Evaluate the Omero architecture to check whether we want to use it to store our OME-Zarr files and to manage metadata, access etc.

https://www.openmicroscopy.org/omero/
https://omero-guides.readthedocs.io/en/latest/

One interesting point in this (also interesting) blog post from Glencoe Software:

OMERO Plus will support the import and management of OME-NGFF image data by mid-2022

[gusqgm]

Finally some words from my side on this.

OMERO is intended as a full platform for managing images in a secure central repository where data can be viewed, organized, analyzed and shared online. I installed initially just the server part, but many other apps exist which allow more interactive visualization / sharing and organization of the data (Omero.insight, Omero.iviewer, etc).

I have been doing quite some reading and went through installation of the omero-server in a local CentOS 8 VM. Installation per se is not very complicated, but it becomes easier if some reading is done before to understand the server in the greater context of things.
Basically, from the server you can add users, groups (or have them imported via LDAP), and users can log in to it either directly via the CLI or via one of the apps (OMERO.insight for example). There are many ways that one can add different level of admins with particular set of privileges so that work is distributed. Files can be added to the server in different ways, either via making an extra copy of them into the server storage, or via making links to them using paths. There are also many different ways to import metadata, and from my understanding it seems that currently anything which could be Bioformats compatible should also work here "out of the box".

From a facility point of view, Omero-server can be a great help in keeping the data of entire groups/departments or institutions within clear organizational rules, making people be vigilant about their metadata, minimizing data duplication and improving accessibility and share-ability.

However, in our case, I was hoping to find a way how we could only have the data manager part of OMERO (more towards the server part) alone that we could use for Fractal, but this was not the case, unfortunately. By installing the server we also need to set up a postgresql database for storing all relevant metadata associated with the images, along with permissions, etc. Clearly, the database needs always to be in sync with any changes in the images/metadata associated with the omero server.

On another note, checking around, I came across an interesting discussion about the future of omero-server in being able to read ome-zarr and their metadata natively. There seems to still be an issue with floating point bypass pyramids, but otherwise it seems like things are moving in the right direction. Furthermore, discussions both internally with the facility as well as with Christian Tischer from EMBL point towards directions where, instead of diving head on towards a full blown installation of an omero server, people first try to solver the problem of metadata from more complex microscope outputs by creating Omero Companion Files from that at least it can be read by Bioformats, which in turn also means being compatible with Omero as well.

All in all, I believe that working on an instance of an omero-server right now might be a bit too early. This opinion is voiced especially considering the current advances in having omero being capable of reading ome-zarr as well as the rather complex nature of the omero-server itself. That said, I also think that the omero-server can become a powerful tool that can be used on the side with all output from Fractal, for the sake of keeping records of processed data either in the form of publication ready of for general lab/institutional reference. This means that we do not need to think on how to adapt an omero server to fit Fractal.

Our bridge here is the OME-Zarr NGFF omero metadata fields, which can facilitate the possibility to migrate processed fractal hcs data onto an instance of an omero server for the sake of more static data collection and management.

I would therefore close this issue for now.