-
-
Notifications
You must be signed in to change notification settings - Fork 267
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactoring stores into a separate library #414
Comments
Thoughts @zarr-developers/core-devs? |
https://github.com/martindurant/filesystem_spec/blob/master/fsspec/mapping.py ? :) Yes, I know, not all stores are file-systems, but I really have been trying to make the implementation more general. |
I really like this idea. |
@jakirkham Well, hopefully |
Hi @mbr, simplekv does look nice, and there's clearly a lot of duplication of functionality between that and what we've implemented in the zarr storage module. There are two technical differences between simplekv's API and the zarr store API that I can see. The first is that zarr uses the MutableMapping interface for getting and setting key/value pairs (i.e., The second is that zarr stores supports some optional methods, including some which are "hierarchy-aware", i.e., when using keys that include a '/' character and wanting to list keys at a given level, akin to listing the contents of a directory if using a file system. Here's what we say about these optional methods in the docs:
|
It's worth pointing out this really helps as we can leverage other methods of |
So far, I have deliberately omittied these kinds of hierarchies;
I believe that |
Over time the Zarr library accumulated more compressors and even some Cython code related to that effort until it made sense for these to go out and live as their own library, Numcodecs. Similarly we seem to be accumulating many stores, perhaps they would be best suited in another library that Zarr depends on.
The stores themselves actually don't really need to know about compression, Groups, or Arrays (with the exception of some layers on top of the stores). Really stores are just key-value stores that allow associating some blob with some key. As such it may be possible for this library to have fewer requirements (if any) and be accessible to a larger group of people that are just interested in things that provide the
MutableMapping
interface to their storage format of choice.By breaking out the stores as a separate library, we gain another benefit. Namely we can release the stores at a different time scale from Zarr itself. So if we get a new store or a bug fix for a store, we can do a new release even while Zarr itself has not changed much.
There are probably more reasons to consider this move. Would be curious to hear what others think about this?
The text was updated successfully, but these errors were encountered: