Axes v0.4 #93

will-moore · 2021-10-28T09:36:46Z

Starting to implement Axes and Transformations (see ome/ngff#57).
cc @constantinpape

This requires https://pypi.org/project/ome-zarr/0.3a1/ for mask writing and new HCS metadata (row/column Indexes).

To test:

$ omero zarr export Image:6001240     # /6001240.zarr/.zattrs  should have new axes (see below)
$ omero zarr masks Image:6001240      # /6001240.zarr/labels/0/.zattrs should also have new axes

$ omero zarr export Plate:1234           # 1234.zarr/.zattrs each 'well' should have rowIndex and columnIndex

Some data exported with this PR can be viewed with vizarr:
https://deploy-preview-139--vizarr.netlify.app/?source=https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-05/idr0062/6001240.zarr (with PR at hms-dbmi/vizarr#139)

Plate data: https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-24/plates/idr0004/1752.zarr

Exported NGFF includes:

axes - eg. at https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-05/idr0062/6001240.zarr/.zattrs :

    "axes": [
                {
                    "name": "c",
                    "type": "channel"
                },
                {
                    "name": "z",
                    "type": "space",
                    "units": "micrometer"
                },
                {
                    "name": "y",
                    "type": "space",
                    "units": "micrometer"
                },
                {
                    "name": "x",
                    "type": "space",
                    "units": "micrometer"
                }
            ]

and each dataset has transformations. E.g. czyx image is scaled along axis z, y, x or [1, 2, 3]:
NB: this format is still being discussed at ome/ngff#57 (review)

                  "transformations": [
                        {
                            "axisIndices": [
                                1,
                                2,
                                3
                            ],
                            "scale": [
                                0.5002025531914894,
                                0.3603981534640209,
                                0.3603981534640209
                            ],
                            "type": "scale"
                        }
                    ]

constantinpape · 2021-11-04T15:19:10Z

For me installation of cfunits via conda install -c conda-forge cfunits went smooth and without any issues. What system did you try this on? Maybe we can raise this issue with the conda-forge maintainers of cfunits / uduints2.

will-moore · 2021-11-04T15:46:36Z

Hah! OK, it worked this time (on an aging Mac 10.14).
But I still don't know if I want to make this a dependency. I might try to validate all the units that we get from OMERO for length (and if not valid then map to a valid unit).

It seems that a lot of things are valid (see below), so if the spec allows all of these, then it becomes very hard to read OME-NGFF data without having cfunits installed. If the spec were more restrictive, then we might not have to make cfunits a dependency of napari-ome-zarr, which would be nice. I'll add a comment to the spec PR...

>>> Units("micrometer").isvalid
True
>>> Units("micrometre").isvalid
True
>>> Units("micrometres").isvalid
True
>>> Units("micrometress").isvalid
False
>>> Units("micrometres")
<Units: micrometres>
>>> Units("micrometre")
<Units: micrometre>
>>> Units("nm")
<Units: nm>
>>> Units("nm").isvalid
True
>>> Units("nanom").isvalid
True
>>> Units("nanome").isvalid
True
>>> Units("nanomet").isvalid
False
>>> Units("nanometer").isvalid
True
>>> Units("nanometre").isvalid
True
>>> Units("nanometre").formatted()
'1e-09 m'
>>> Units("nanome").formatted()
'1.602176487e-31 C'
>>> Units("nanome").units
'nanome'
>>> Units("nanom").units
'nanom'
>>> Units("nano").units
'nano'
>>> Units("nano").formatted()
>>> Units("nanom").formatted()
'1e-09 m'
>>> Units("nanometer").formatted()
'1e-09 m'
>>> Units("nanometre").formatted()
'1e-09 m'
>>> Units("nanom").formatted()
'1e-09 m'

src/omero_zarr/raw_pixels.py

will-moore · 2022-01-24T11:49:32Z

Plate data exported today with current state of this branch and https://pypi.org/project/ome-zarr/0.3a1/
https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-24/plates/idr0004/1752.zarr

sbesson

A few API comments. I have not carried out any functional testing yet

src/omero_zarr/__init__.py

src/omero_zarr/masks.py

src/omero_zarr/raw_pixels.py

src/omero_zarr/util.py

sbesson · 2022-01-24T15:37:21Z

src/omero_zarr/raw_pixels.py

@@ -91,6 +91,12 @@ def planeGen() -> np.ndarray:
        cache_file_name_func=get_cache_filename,
    )

+    metadata = marshal_axes(image, len(paths))
+
+    write_multiscales_metadata(parent, paths, **metadata)


Now that I see this API (introduced in ome/ome-zarr-py#124) in action, I am slightly confused. I think this is primarily because transformations is generated from the knowledge of paths above and then later re-zipped with paths when writing the metadata.

An alternative workflow would be to support datasets either as lists of string or dictionaries in write_multiscales_metadata.

Perhaps slightly related to my thoughts in ome/ome-zarr-py#161 (review)

since datasets would include paths, we would just have write_multiscales_metadata(group, datasets) and remove support for paths? Or if datasets is a list of strings, treat it as paths (wouldn't break the API, but could be more confusing).

Similar questions and concerns around the usability came up in ome/ome-zarr-py#157 when adding support for plate.wells as list of dictionaries alongside list of strings (also corresponding to individual well paths).

At least, I find List[Union[str, dict]] to be a fairly good compromise that allows to handle both the simple use case where a minimal valid spec is generated from a list of paths and the extensible scenario where the caller wants to store additional metadata (and not necessarily specified by the OME-NGFF spec) to the datasets element.

I also wonder about the API for reading the transformations in ome-zarr-py. These are currently a 2D list in the node.metadata (not in the form of datasets) since paths are not returned by the reader:

paths = [d["path"] for d in datasets] self.datasets: List[str] = paths transformations = [d.get("transformations") for d in datasets] if any(trans is not None for trans in transformations): node.metadata["transformations"] = transformations

See ome/ome-zarr-py#162

sbesson · 2022-01-25T12:11:41Z

In addition to the various comments above, following this morning's discussion, I have been thinking of ways to exercise the translation transformation that is being introduced in ome/ngff#57.

In the OMERO -> OME-NGFF workflow, a potential candidate might be the WellSample.PositionX and WellSample.PositionY i.e. within the context of a well specification have the individual multiscales images registering the X/Y position relative to some well origin so that a client would be able to display a well grid using the absolute positions.

Unless this is something straightforward to implement, I would definitely consider this outside the scope of this PR and limit the transformations support to scale for now. The primary reason for bringing it up is that it might something to keep in mind alongside the API questions raised in #93 (comment). Since this translation metadata would be not be available from the ImageWrapper itself and would either need to be passed to add_image or be added via a follow-up call which would update the metadata.

sbesson · 2022-01-25T14:38:45Z

Trying to use the latest version of this PR, I get

(zarr) [sbesson@pilot-zarr1-dev ~]$ omero zarr export Plate:9512 --help

Error loading: /home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/omero/plugins/zarr.py
Traceback (most recent call last):
  File "/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/omero/cli.py", line 1690, in loadpath
    execfile(str(pathobj), loc)
  File "/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/past/builtins/misc.py", line 87, in execfile
    exec_(code, myglobals, mylocals)
  File "/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/omero/plugins/zarr.py", line 1, in <module>
    from omero_zarr.cli import HELP, ZarrControl
  File "/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/omero_zarr/cli.py", line 15, in <module>
    from .raw_pixels import (
  File "/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/omero_zarr/raw_pixels.py", line 10, in <module>
    from ome_zarr.writer import (
ImportError: cannot import name 'write_multiscales_metadata' from 'ome_zarr.writer' (/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/ome_zarr/writer.py)

Should the ome-zarr-py dependency be upgraded similarly to ome/napari-ome-zarr@dc4f0bd?

will-moore · 2022-01-25T16:04:06Z

@sbesson Fixed ome-zarr requirement.

will-moore · 2022-01-25T17:42:23Z

That last commit de7703a uses PR: ome/ome-zarr-py#162

will-moore · 2022-01-26T16:30:13Z

@joshmoore @sbesson I guess we probably don't want that last commit if this PR goes into a pre-release of omero-cli-zarr that works with ome-zarr>=0.3a1?
So, I can revert that and then we can make those changes in a follow-up PR?
I also have a commit locally that uses "scale": {"x": 0.5, "y":0.5: "z": 0.3} format and I guess that should go into a later PR too (so that the pre-releases are all aligned on the axisIndices format?

sbesson · 2022-01-26T16:33:33Z

Yes 👍 for getting a pre-release of omero-cli-zarr with ome-zarr-py 0.3.0a that matches the current proposal and focus on the new writing API and new variant for scale as a follow-up set of pre-releases.

will-moore · 2022-01-26T16:39:23Z

Reverted de7703a

sbesson

The most important concerns above have been addressed and new PRs have been opened upstream to introduce new metadata writing API. No objection from my side to getting this merged and releases as 0.3.0a1. @joshmoore is there anything from your side that should be captured as an issue and addressed in the next development segment?

will-moore · 2022-01-26T22:44:00Z

As discussed today, a screenshot illustrating the use of different names for Z/T axes and how it looks in Vizarr:
cc @joshmoore @sbesson @constantinpape

constantinpape · 2022-01-27T08:57:36Z

Thanks @will-moore. Is this the dataset from https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-05/idr0062/6001240.zarr?
I can open that one with Fiji/MoBIE now, but I am not sure if it's quite correct yet:

will-moore · 2022-01-27T16:50:16Z

Looking at https://deploy-preview-139--vizarr.netlify.app/?source=https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-05/idr0062/6001240.zarr (use Firefox - bugs on Chrome just now).
The .zattrs has:

"datasets": [
                {
                    "path": "0",
                    "transformations": [
                        {
                            "axisIndices": [
                                1,
                                2,
                                3
                            ],
                            "scale": [
                                0.5002025531914894,
                                0.3603981534640209,
                                0.3603981534640209
                            ],
                            "type": "scale"
                        }
                    ]
                },

which should mean it applies to both channels? It looks like your example has different transforms applied to the 2 channels?

sbesson · 2022-01-31T12:10:04Z

As discussed this morning, merging and tagging this as 0.3.0a1 alongside all the other Python libraries pre-releases created ahead of the NGFF community call.
Ongoing work is happening in #103 to support the latest proposed changes to the 0.4 specification and the conversation on how to validate transformations started above can happen there.

will-moore mentioned this pull request Nov 2, 2021

Extend the axes fields in multiscales metadata ome/ngff#57

Merged

will-moore added 3 commits December 20, 2021 12:37

Export axes as dict of name and type for Image

3d0c7a2

Update version to 0.4

288a1d6

Add units info to axes

9f2297e

will-moore force-pushed the axes_v0.4 branch from aebcb30 to 9f2297e Compare December 20, 2021 12:37

will-moore added 4 commits December 20, 2021 12:46

Move axes creation to add_image()

d079905

add_image() also does add_multiscales_metadata()

b3db05d

Define path = str(level) in 1 place only

9568de9

Add 'transformations' support for x, y, z

3ec5f56

will-moore marked this pull request as ready for review January 6, 2022 14:53

sbesson reviewed Jan 13, 2022

View reviewed changes

src/omero_zarr/raw_pixels.py Outdated Show resolved Hide resolved

Use ome-zarr-py write_multiscales_metadata()

70975a4

sbesson mentioned this pull request Jan 19, 2022

v0.4 axes transforms ome/napari-ome-zarr#31

Merged

3 tasks

will-moore added 2 commits January 19, 2022 12:40

Use ome-zarr .write_plate_metadata()

cc44688

use same axes and transf creation for image and labels

b4ecba4

sbesson reviewed Jan 24, 2022

View reviewed changes

will-moore added 3 commits January 25, 2022 13:31

ngff_version = CurrentFormat().version

9194fdd

Use ome_zarr.writer.write_well_metadata()

27df32b

Remove hard-coded version in masks.py

c9447be

will-moore added 3 commits January 25, 2022 14:59

Split marshal_transformations() into separate function

0346685

Fix marshal_axes/transformations in MaskSaver

b4ab862

setup.py requires ome-zarr>=0.3a1

adbe8f5

will-moore mentioned this pull request Jan 25, 2022

write_multiscales_metadata no_transformations ome/ome-zarr-py#162

Merged

will-moore force-pushed the axes_v0.4 branch from de7703a to adbe8f5 Compare January 26, 2022 16:38

will-moore mentioned this pull request Jan 26, 2022

Scale dict #103

Merged

sbesson approved these changes Jan 26, 2022

View reviewed changes

sbesson merged commit ab56eca into ome:master Jan 31, 2022

sbesson added this to Done in OME-NGFF 0.4 Feb 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Axes v0.4 #93

Axes v0.4 #93

will-moore commented Oct 28, 2021 •

edited

constantinpape commented Nov 4, 2021

will-moore commented Nov 4, 2021

will-moore commented Jan 24, 2022

sbesson left a comment

sbesson Jan 24, 2022

joshmoore Jan 25, 2022

will-moore Jan 25, 2022

sbesson Jan 25, 2022

will-moore Jan 25, 2022

will-moore Jan 25, 2022

sbesson commented Jan 25, 2022 •

edited

sbesson commented Jan 25, 2022

will-moore commented Jan 25, 2022

will-moore commented Jan 25, 2022 •

edited

will-moore commented Jan 26, 2022

sbesson commented Jan 26, 2022 •

edited

will-moore commented Jan 26, 2022

sbesson left a comment

will-moore commented Jan 26, 2022

constantinpape commented Jan 27, 2022

will-moore commented Jan 27, 2022

sbesson commented Jan 31, 2022

Axes v0.4 #93

Axes v0.4 #93

Conversation

will-moore commented Oct 28, 2021 • edited

constantinpape commented Nov 4, 2021

will-moore commented Nov 4, 2021

will-moore commented Jan 24, 2022

sbesson left a comment

Choose a reason for hiding this comment

sbesson Jan 24, 2022

Choose a reason for hiding this comment

joshmoore Jan 25, 2022

Choose a reason for hiding this comment

will-moore Jan 25, 2022

Choose a reason for hiding this comment

sbesson Jan 25, 2022

Choose a reason for hiding this comment

will-moore Jan 25, 2022

Choose a reason for hiding this comment

will-moore Jan 25, 2022

Choose a reason for hiding this comment

sbesson commented Jan 25, 2022 • edited

sbesson commented Jan 25, 2022

will-moore commented Jan 25, 2022

will-moore commented Jan 25, 2022 • edited

will-moore commented Jan 26, 2022

sbesson commented Jan 26, 2022 • edited

will-moore commented Jan 26, 2022

sbesson left a comment

Choose a reason for hiding this comment

will-moore commented Jan 26, 2022

constantinpape commented Jan 27, 2022

will-moore commented Jan 27, 2022

sbesson commented Jan 31, 2022

will-moore commented Oct 28, 2021 •

edited

sbesson commented Jan 25, 2022 •

edited

will-moore commented Jan 25, 2022 •

edited

sbesson commented Jan 26, 2022 •

edited