From 7bf97d454cd42f6bdb1d0bff8e2e7ffa35c07f9d Mon Sep 17 00:00:00 2001 From: Norman Rzepka Date: Tue, 2 Jul 2024 15:40:45 +0200 Subject: [PATCH 1/6] RFC2 revision --- rfc/2/index.md | 94 +++++++++++++++++++++++--------------------------- 1 file changed, 44 insertions(+), 50 deletions(-) diff --git a/rfc/2/index.md b/rfc/2/index.md index 67e1cbb4..1158fc74 100644 --- a/rfc/2/index.md +++ b/rfc/2/index.md @@ -54,17 +54,18 @@ Adopting Zarr v3 in OME-Zarr is a precondition for using sharding. Library support for Zarr v3 is already available for several languages: +- [zarr-python (Python)](https://github.com/zarr-developers/zarr-python) - [tensorstore (C++/Python)](https://github.com/google/tensorstore) -- [zarrita (Python)](https://github.com/scalableminds/zarrita) +- [zarr-java (Java)](https://github.com/zarr-developers/zarr-java) - [zarrita.js (JS)](https://github.com/manzt/zarrita.js) - [zarr3-rs (Rust)](https://github.com/clbarnes/zarr3-rs) -Visualization tools with integrated Zarr implementations are also available: +Visualization tools with integrated Zarr v3 implementations are also available: - [neuroglancer](https://github.com/google/neuroglancer) - [WEBKNOSSOS](https://github.com/scalableminds/webknossos) -Support for other languages is under active development, including C, Java and Python. +Support for other languages is under active development. Libraries will likely prioritize support for v3 over previous versions in the near future. OME-Zarr should therefore adopt the new version for future-proofing. @@ -87,26 +88,7 @@ Implementations can read inner chunks individually. Depending on the choice of codecs and the underlying storage backends, it may be possible to write inner chunks individually. However, in the general case, writing is limited to entire shards. -## Proposal - -This RFC proposes to adopt version 3 of the Zarr format for OME-Zarr. -Images that use the new version of OME-Zarr metadata MUST NOT use Zarr version 2 any more. - -The motivation for making this hard cut is to reduce the burden of complexity for implementations. -Currently, many Zarr library implementations support both versions. -However, in the future they might deprecate support for version 2 or deprioritize it in terms of features and performance. -Additionally, there are OME-Zarr implementations that have their own integrated Zarr stack. -With this hard cut, implementations that only support OME-Zarr versions > 0.5 (TODO: update assigned version number) will not need to implement Zarr version 2 as well. - -From a OME-Zarr user perspective, the hard cut also makes things simpler: ≤ 0.5 => Zarr version 2 and > 0.5 => Zarr version 3 (TODO: update assigned version number). -If users wish to upgrade their data from one OME-Zarr version to another, it would be easy to also migrate the core Zarr metadata to version 3. -This is a fairly cheap operation, because only json files are touched. -Zarr version 2 and 3 metadata could even live side-by-side in the same hierarchy. -There are [scripts available](https://github.com/scalableminds/zarrita/blob/8155761/zarrita/array_v2.py#L452-L559) that can migrate the metadata automatically. - -It is RECOMMENDED that implementations support a range of OME-Zarr versions, including versions based on Zarr version 2. - -### Notable changes in Zarr v3 +### Other notable changes in Zarr v3 There are a few notable changes that Zarr v3 brings for OME-Zarr: @@ -126,15 +108,40 @@ There are a few notable changes that Zarr v3 brings for OME-Zarr: The Zarr specification does not prescribe the support stores for Zarr hierarchies. HTTP(S), File system, S3, GCS, and Zip files are commonly used stores. +## Proposal + +This RFC proposes to adopt version 3 of the Zarr format for OME-Zarr. +Images that use the new version of OME-Zarr metadata MUST NOT use Zarr version 2 any more. + With this proposal all features of the Zarr specification are allowed in OME-Zarr. In the future, the OME-Zarr community MAY decide to restrict the allowed feature set. +The motivation for making this hard cut is to reduce the burden of complexity for implementations. +Currently, many Zarr library implementations support both versions. +However, in the future they might deprecate support for version 2 or deprioritize it in terms of features and performance. +Additionally, there are OME-Zarr implementations that have their own integrated Zarr stack. +With this hard cut, implementations that only support OME-Zarr versions > 0.5 will not need to implement Zarr version 2 as well. + +From a OME-Zarr user perspective, the hard cut also makes things simpler: ≤ 0.5 => Zarr version 2 and > 0.5 => Zarr version 3. +If users wish to upgrade their data from one OME-Zarr version to another, it migration tools will be available ([prototype here](https://github.com/scalableminds/zarrita/blob/8155761/zarrita/array_v2.py#L452-L559)). +Migration is a fairly computationally cheap operation, because only json files are touched. + +Due to the existance of large quantities of image in OME-Zarr 0.4, it is RECOMMENDED that implementations continue to support OME-Zarr 0.4 with the underlying Zarr v2. + +OME-Zarr images MUST be consistent in their OME-Zarr and Zarr version. +With this constraint, implementations only need to detect the version of a provided URL or file path once and can assume that all multiscale levels, wells, series images etc. use the same version. + +While technically possible, OME-Zarr 0.5 (with Zarr v3) and OME-Zarr 0.4 (with Zarr v2) metadata could exist side-by-side in a Zarr hierarchy, it is NOT RECOMMENDED. +This may be useful for short periods of time (i.e. during migrations from 0.4 to 0.5), but should not be used longer term. +Multiple metadata versions can lead to conflicts, which may be hard to resolve by implementations. +If implementations encounter 0.4 and 0.5 metadata side-by-side, 0.5 SHOULD be treated preferentially. + ### Changes to the OME-Zarr metadata -While the adoption of Zarr v3 does not strictly require changes to the OME-Zarr metadata, this proposal contains changes to align with community conventions and ease implementation: +While the adoption of Zarr v3 does not strictly require changes to the OME-Zarr metadata, this proposal contains changes to align with [community conventions](https://zarr.dev/zeps/draft/ZEP0004.html#namespacing) and ease implementation: -- OME-Zarr metadata will be stored under a dedicated key in the Zarr array or group attributes. The key will be a well-known URI of the OME-NGFF specification with a version number, e.g. `https://ngff.openmicroscopy.org/0.6`. -- Since the version is already encoded in the new metadata key, the `version` keys in `multiscale`, `plate`, `well` etc. are removed. +- OME-Zarr metadata will be stored under a dedicated `ome` key in the Zarr array or group attributes. +- The version information will be moved from the `multiscale`, `plate`, `well` etc. sections into the new `ome` section. - The `dimension_names` attribute in the Zarr metadata must match the axes names in the OME-Zarr metadata. Finally, this proposal changes the title of the OME-Zarr specification document to "OME-Zarr specification". @@ -145,7 +152,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [IETF RFC 2119][IETF RFC 2119] -## Stakeholders (Recommended Header) +## Stakeholders Preliminary work of this RFC has been discussed in: @@ -156,30 +163,12 @@ Preliminary work of this RFC has been discussed in: - several Zarr community calls - several recent OME-NGFF community calls. - - ## Implementation -OME-Zarr implementations can rely on existing Zarr libraries to implement the adoption of Zarr v3. +OME-Zarr implementations can rely on existing Zarr libraries to implement the adoption of Zarr v3. +See [Background](#background) for a list of v3-capable Zarr libraries. -TODO: Provide a reference implementation +Support for the OME-Zarr 0.5 metadata is under development in [ome-zarr-py](https://github.com/ome/ome-zarr-py/pull/383/files) and other implementations. ## Drawbacks, risks, alternatives, and unknowns @@ -216,7 +205,7 @@ In particular, the chunk sizes can be made small to facilitate interactive visua The metadata of Zarr v3 arrays are not backwards compatible with Zarr v2. -It is RECOMMENDED that implementations of OME-Zarr specify the version of the OME-Zarr specification that they support. +Implementations of OME-Zarr MUST specify the version(s) of the OME-Zarr specification that they support. It is RECOMMENDED that implementations of OME-Zarr that support both v2 and v3-based OME-Zarr versions auto-detect the underlying Zarr version. @@ -230,6 +219,10 @@ Zarr v3 and v2 metadata can exist side-by-side within a Zarr hierarchy. Previous versions of this proposal contained changes to referencing `labels` in the OME-Zarr metadata. This has been delayed to future RFCs. +Previous versions have used a versioned namespace, e.g. `https://ngff.openmicroscopy.org/0.5`, in the Zarr attributes instead of a simple `ome` namespace with dedicated `version` attribute. +This has been abandoned because it makes discovery of versions more difficult. +Additionally, handling of multiple versions may be ill-defined. + ## Examples File hierarchy of one multi-scale OME-Zarr image `456.zarr`: @@ -260,7 +253,8 @@ File hierarchy of one multi-scale OME-Zarr image `456.zarr`: "zarr_format": 3, "node_type": "group", "attributes": { - "https://ngff.openmicroscopy.org/0.6": { + "ome": { + "version": "0.5", "multiscales": [ { "coordinateSystems": [ From 2a05226690be52a2af6ac2fe667e18bab7541672 Mon Sep 17 00:00:00 2001 From: Norman Rzepka Date: Tue, 2 Jul 2024 17:31:59 +0200 Subject: [PATCH 2/6] RFC2 revision --- rfc/2/index.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/rfc/2/index.md b/rfc/2/index.md index 1158fc74..eddba3c8 100644 --- a/rfc/2/index.md +++ b/rfc/2/index.md @@ -174,6 +174,11 @@ Support for the OME-Zarr 0.5 metadata is under development in [ome-zarr-py](http While it is clear that Zarr v3 will become the predominant version of the specification moving forward, current library support for v3 is still under active development. +An alternative to this proposal would be to [add Zarr v3 support to OME-Zarr 0.4](https://github.com/ome/ngff/pull/249) without changes to the OME-Zarr Metadata. +The contents of the `.zattrs` would simply move to the `attributes` within the `zarr.json`. +There would need to be some transparency for users to know what Zarr versions are supported by an implementation. +Additionally, there would be no opportunity to introduce a `ome` namespace in the attributes that is useful for composibility. +