Skip to content

how to integrate subpackages #4317

Description

@d-v-b

we now have a few subpackages that spin out core functionality into self-contained python packages:

  • zarr-indexing handles lazy, chunked array indexing. It's a drop-in replacement for indexing.py in zarr, and it supports lazy indexing.
  • zarr-metadata handles the type-safe representation of zarr v2 and v3 metadata documents. it can add a missing layer to our metadata handling and ultimately replace our ArrayMetadataV3 and ArrayMetadataV2 classes.
  • zarr-chunk-key-encodings is unreleased but proposed in this PR. It would be a drop-in replacement for our chunk_key_encodings.py

There are more subpackages I want to make, namely something for stores, codecs and data types. But before I start those, we should sketch out how we bring any subpackage into zarr.

here's my current plan:

  1. we first depend on the subpackage in our test suite
  2. we then bring drop-in replacements, if any, that package provides to runtime
  3. new features (like lazy indexing) are gated behind a feature flag
  4. with the perspective from 1-3 we can plan out major changes (like sunsetting ArrayV3Metadata).

@zarr-developers/python-core-devs let me know if this direction makes sense. I'm also happy to explain why we are doing this. If nobody objects, I will open PRs soon with step 1 for the existing subpackages.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions