Zarr 3.0 Codec Registry

[bruniss]

Good afternoon!

I work with the Vesuvius Challenge on very large / high resolution xray ct volumes. We created a compression codec (a per-chunk axis dependent delta filter followed up by an entropy encoder, in this case rANS) because the lossless compressors used by the regular pypi installed zarr perform rather poorly on our data. We've been able to use this with zarr given that we include it in a pyproject.toml to register it w/ numcodecs, but would like to store our s3 open data volumes this way.

This unfortunately would create a bit of a hurdle for people who see that our data is zarr and go to use it (using zarr-python) only to find out they cannot read it without a few more steps.

I noticed this page https://zarr.dev/codecs-registry/ , however it looks like it might not be applicable any longer.

Is that page / instructions still valid? or is there another way to get a codec registered like this?

Or perhaps there is a way to composite filters like the one described and have regular zarr-python read them?

Thank you for your help!

[d-v-b]

Hi @bruniss! thanks for using zarr!

This unfortunately would create a bit of a hurdle for people who see that our data is zarr and go to use it (using zarr-python) only to find out they cannot read it without a few more steps.

Yeah that's a bummer. This is a general issue when people extend zarr, e.g. with a custom codec. If your goal is for folks with just a plain zarr installation to get connected to the data, the two options here are:

1. bundling your custom codec with the base zarr installation. generally we only bundle codecs that have a published spec in either the zarr v3 core or on zarr-extensions (yall might want to write the spec up on zarr-extensions in any case). The guiding principle here (which I don't think we have actually articulated anywhere in text...) is that whatever codecs we support by default should be ones folks are likely to find in another zarr implementation. That discourages us from shipping any default codecs that aren't very well established.
2. including a useful error message that points people to the package they need to install the codec. A templated string like "An implementation for codec was not found. Try installing one of the following packages: " seems like it would be helpful, and the maintenance burden of maintaining this is low.

I noticed this page https://zarr.dev/codecs-registry/ , however it looks like it might not be applicable any longer.

wow I did not know about this page! it was last updated in 2022, so we probably need to have a think about what this page should be doing in the modern era.