Plate conversion

[sbesson]

Ongoing set of improvements allowing to use these tools to convert plates

• bumps bioformats2raw to the current master (pre 0.3.0) including the HCS specification implementation Initial OME Zarr HCS implementation glencoesoftware/bioformats2raw#69
• refactors merge.py to consume a top-level rendering file and apply it recursively to all subfolders of a Zarr plate

[joshmoore]

@sbesson : do you know if this is (more or less) ready to go in? Just thinking about the needed bumps for bioformats2raw 0.3.0 (see https://forum.image.sc/t/issue-with-opening-zarr-inside-napari/56089/15). Perhaps this needs adding to ome_zarr_test_suite as well.

[sbesson]

I have tested it to generate sample data so I am confident the conversion works as a proof-of-concept (and potentially a candidate for ome_zarr_test_suite)

There is a fundamental limitation we should discuss before moving forward and using this in production. Like for the non-HCS image use case, the rendering settings need to be exported as JSON first and injected via command-line option. The current state of this PR uses a single JSON file for all the images. While this will do an fair job for some plates, this effectively assumes all images have the same rendering settings, which is not always true, and the same name which is definitely not true.

I can think of two routes, both of them require more discussion and design:

• introduce some logic to download rendering files associated with all images on a plate and support parsing this new structure instead of a single file when generating the data
• investigate some form of hybrid conversion where the base OME-Zarr would be generated via bioformats2raw and ahve the OMERO specific fields updated by omero-cli-zarr

[sbesson]

Looking at how the logic is implemented in omero-cli-zarr, I was reminded of the omero zarr export --bf option in https://github.com/ome/omero-cli-zarr#images-and-plates. This makes me wonder whether this should be the preferred approach and all the logic in merge.py should be handled in the CLI plugin instead.

[joshmoore]

Discussed today: @sbesson giving it a try and then we can adjust, refactor, and roll out.

[sbesson]

ome/omero-cli-zarr#76 would be a first step towards migrating the merge.py logic as part of omero zarr export --bf for images. If approved, the next action will be to look at updating omero zarr export -bf to support HCS.

[sbesson]

From my side, omero-cli-zarr 0.1.0 should now include all the metadata logic in merge.py for non-HCS datasets.

Opened ome/omero-cli-zarr#85 to handle the HCS use case. My vote would be to close this draft PR and keep pushing as much as the generic export as possible to the OMERO plugin.

[joshmoore]

Is a change here needed to consume ome/omero-cli-zarr#85?

[sbesson]

Quite likely. While this repository is using submodules + Docker, I have been primarily using Conda with an environment.yml to install the set of dependencies required for conversion + transfer.

I could look into updating this repository accordingly so that any new conversion would benefit from the latest releases of the tools?