Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Describe the zarr file format in the documentation #5

Closed
shoyer opened this issue Dec 22, 2015 · 2 comments
Closed

Describe the zarr file format in the documentation #5

shoyer opened this issue Dec 22, 2015 · 2 comments
Milestone

Comments

@shoyer
Copy link
Contributor

shoyer commented Dec 22, 2015

It looks like the design is very similar to bcolz, but this would be nice to have as a point of reference.

@alimanfoo
Copy link
Member

Yep, good idea.

FWIW I basically copied bcolz but made a couple of changes.

(1) I dropped the bloscpack header from the chunk data files, because it's not needed and I was trying to strip everything down to bare essentials. So each chunk data file contains the compressed data that c-blosc returns and nothing else.

(2) I used pickle instead of JSON for the metadata because it was simpler and easier to guarantee correct round-trip to/from Python objects. However I'd be happy to consider changing to JSON if it would help.

All details of the storage format are open to discussion.

@alimanfoo alimanfoo changed the title Describe the zarr file format in the documentation? Describe the zarr file format in the documentation Dec 22, 2015
@alimanfoo
Copy link
Member

Some documentation like this would be good: https://github.com/Blosc/bcolz/blob/master/DISK_FORMAT_v1.rst

@alimanfoo alimanfoo modified the milestone: v0.4 Apr 11, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants