we now have a few subpackages that spin out core functionality into self-contained python packages:
zarr-indexing handles lazy, chunked array indexing. It's a drop-in replacement for indexing.py in zarr, and it supports lazy indexing.
zarr-metadata handles the type-safe representation of zarr v2 and v3 metadata documents. it can add a missing layer to our metadata handling and ultimately replace our ArrayMetadataV3 and ArrayMetadataV2 classes.
zarr-chunk-key-encodings is unreleased but proposed in this PR. It would be a drop-in replacement for our chunk_key_encodings.py
There are more subpackages I want to make, namely something for stores, codecs and data types. But before I start those, we should sketch out how we bring any subpackage into zarr.
here's my current plan:
- we first depend on the subpackage in our test suite
- we then bring drop-in replacements, if any, that package provides to runtime
- new features (like lazy indexing) are gated behind a feature flag
- with the perspective from 1-3 we can plan out major changes (like sunsetting
ArrayV3Metadata).
@zarr-developers/python-core-devs let me know if this direction makes sense. I'm also happy to explain why we are doing this. If nobody objects, I will open PRs soon with step 1 for the existing subpackages.
we now have a few subpackages that spin out core functionality into self-contained python packages:
zarr-indexinghandles lazy, chunked array indexing. It's a drop-in replacement forindexing.pyinzarr, and it supports lazy indexing.zarr-metadatahandles the type-safe representation of zarr v2 and v3 metadata documents. it can add a missing layer to our metadata handling and ultimately replace ourArrayMetadataV3andArrayMetadataV2classes.zarr-chunk-key-encodingsis unreleased but proposed in this PR. It would be a drop-in replacement for ourchunk_key_encodings.pyThere are more subpackages I want to make, namely something for stores, codecs and data types. But before I start those, we should sketch out how we bring any subpackage into
zarr.here's my current plan:
ArrayV3Metadata).@zarr-developers/python-core-devs let me know if this direction makes sense. I'm also happy to explain why we are doing this. If nobody objects, I will open PRs soon with step 1 for the existing subpackages.