Thinning down zarr-python

[MSanKeys963]

This came up during one of the conversations with @joshmoore. The idea is to extract functionalities not in V2 Spec and their respective tests from zarr-python. E.g. Zarr Specification V2 has the following stores:

• Directory Store
• Nested Directory Store
• Zip file Store
• Object Store
• HTTP Store

But in zarr-python, we have additional stores like SQLiteStore, MongoDBStore etc. If we could remove these additional stores from the zarr-python codebase and host them in a separate repo under zarr-developers, it'll help us:

• Making the code run faster and more efficiently
• Run fewer tests and faster completion of them
• Reduce the noise from dependabot
• Add maintainers for specific stores
• Focus on a more concise codebase, thus helping us to add additional features when migrating from V2 to V3

If a user/developer wishes to use the removed functionality, they can shoot pip install zarr-sqlitestore or similar.

FYI: A similar repo is in the works for Numcodecs, which is taken care of by @joshmoore and @jakirkham.

Please let me know your thoughts on this. Thanks!

CC: @zarr-developers/python-core-devs

[jakirkham]

Yeah we have had similar discussions before

• Store refactor #764
• Refactoring stores into a separate library #414

[MSanKeys963]

Thanks for linking the previous discussions, @jakirkham. Some interesting points there.

Also, we have PyData Global 2022 Sprints coming up next week so we're thinking to kick this off over there.

[jakirkham]

The main thing to consider is currently we are testing these Stores pretty regularly (particularly with the general code here and dependency updates). Would be good to handle this somehow when refactoring (cron jobs on CI, running with all Store extensions installed here, etc.)

[joshmoore]

Updates to the description:

• s/eliminate/extract/
• dropped FSStore from the list for the moment
• Added benefits:
  • Reduce the noise from dependabot
  • Add maintainers for specific stores

To #1274 (comment), @jakirkham, I'd say with all the work on easing releases, we should really push for faster iterations, and then have dependabot test each of the refactor stores (daily?)

[rabernat]

The main thing to consider is currently we are testing these Stores pretty regularly (particularly with the general code here and dependency updates)

If the interface between the Zarr core and the store layer is clearly and cleanly defined, it should in theory be sufficient to test each store thoroughly. If there are edge cases that only arise when testing a store against the full Zarr test suite, that might indicate leaky abstractions. (Partial chunk decompression comes to mind as a sketchy area.)

In any case, we should probably be setting up more integration testing of the entire stack, including testing against real cloud storage. I would be happy to work on that.

[martindurant]

we should probably be setting up more integration testing of the entire stack

I think this is far more important than trimming the zarr-python codebase!

Would it be simpler to move the not-so-used storage classes into a contrib/ directory, as a mark that they are not maintained by the core team?

I definitely think we need to keep in all the stores that may be instantiated directly in the main open_* functions (DirStore, FSStore, ConsolidatedStore, ...?).

[jakirkham]

Yeah we had a similar discussion about refactor contents out within Zarr-Python as different modules and adding code owners ( #764 )

Honestly agree this may be an easier first step to address the issue identified. Once complete this change may be useful for further library refactoring. It may also be easier for a new contributor to try than setting up a new repo

On a different note perhaps we should also think about an entry point mechanism like what was done in Numcodecs ( zarr-developers/numcodecs#300 ). This may make it easier for developers wanting to add their own Store without needing changes in Zarr-Python

[d-v-b]

I'd be happy seeing the n5 functionality pulled out into its own package.

[MSanKeys963]

More info here: https://github.com/zarr-developers/cookiecutter-zarr-store

[MSanKeys963]

@jhamman - do you think this should be a part of V3 design doc (#1583) goals?

[d-v-b]

see https://github.com/zarr-developers/n5py (private atm) for a start

[jhamman]

I've started taking steps to deprecate many of the more exotic stores. See #1756.