Hierarchical chunk storage #29
Comments
|
Here's what I've been working on:
Conversations starting https://gitter.im/zarr-developers/community?at=601d471fc83ec358be27944f and during the 2021-02-10 community call suggest that even before the V3 spec (which defaults to nested storage) it might be possible for us to transition to using it more heavily with the V2 spec. A requirement would be a heuristic or metadata entry to determine if chunks are nested or flat. zarr-python is waiting on a PR from me. |
|
This issue has been mentioned on Image.sc Forum. There might be relevant details there: https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386/9 |
|
Status update:
With all of this in mind, I'd propose that we go ahead and bump to OME-Zarr version to 0.2.0 stating in the spec that nested storage is a MUST. A client can then detect whether the fileset is 0.1.0 or 0.2.0 from the metadata and make a choice as to the chunk index separator. cc: @manzt (since I've failed to engage on the JS front) |
This defines the dimension separator (sometimes called "key separator") for chunks in the NGFF file to be "/" rather than ".". The benefit of storing hierarchically is that for very large volumes the number of files in a directory is limited to `shape / chunk` for a given dimension. close: ome#29
This defines the dimension separator (sometimes called "key separator") for chunks in the NGFF file to be "/" rather than ".". The benefit of storing hierarchically is that for very large volumes the number of files in a directory is limited to `shape / chunk` for a given dimension. close: ome#29
This defines the dimension separator (sometimes called "key separator") for chunks in the NGFF file to be "/" rather than ".". The benefit of storing hierarchically is that for very large volumes the number of files in a directory is limited to `shape / chunk` for a given dimension. close: ome#29
|
Feels like most implementations have now implemented this new strategy. Likely a solid next step would be to add v0.1 and v0.2 implementations to https://github.com/ome/ome_zarr_test_suite |
|
This issue has been mentioned on Image.sc Forum. There might be relevant details there: https://forum.image.sc/t/open-an-image-from-omero-in-imagej-js/47747/17 |
Currently, the ome zarr specification demands that the chunks are stored in a single directory.
This causes issues on the file system for large images / volumes with many chunks.
N5 stores chunks in a hierarchy natively and zarr also supports this with the NestedDirectoryStore (although this is currently not working according to @joshmoore).
The text was updated successfully, but these errors were encountered: