Adds index_codecs to the sharding codec

[normanrz]

This PR adds the index_codecs configuration to the sharding codec. It also adds the crc32c checksum bytes-to-bytes codec.

Index codecs have been proposed by @jbms: #152 (comment)
There is a prototype implementation in zarrita: https://github.com/scalableminds/zarrita/blob/codec-pipeline/zarrita/sharding.py#L507-L518

[normanrz]

@jbms @jstriebel Please take a look. If we decide to go with this, it would be good to merge this before we send ZEP2 to the ZIC.

[jbms]

Thanks! Looks good to me other than the comment I raised.

[mkitti]

I'm a bit confused how c.compute_encoded_size will work. To use this, I have to compute the byte size of the decoded representation? Is there some function to compute the size of the decoded representation?

Rather I think the input should just be n, the number of inner chunks. Perhaps there should also be a c.compute_decoded_size as well which also takes n as an input.

[jbms]

I'm a bit confused how c.compute_encoded_size will work. To use this, I have to compute the byte size of the decoded representation? Is there some function to compute the size of the decoded representation?

I think this could indeed be clarified, since we have "array -> array", "array -> bytes", and "bytes -> bytes" codecs to consider. For "array -> array" and "array -> bytes", the input is an array and therefore the relevant size would be the shape of the array, not its size in bytes. For "bytes -> bytes" codecs the input size would be in bytes.

Rather I think the input should just be n, the number of inner chunks. Perhaps there should also be a c.compute_decoded_size as well which also takes n as an input.

[normanrz]

I'm a bit confused how c.compute_encoded_size will work. To use this, I have to compute the byte size of the decoded representation? Is there some function to compute the size of the decoded representation?

The size of the decoded representation is product(shape) * dtype.itemsize. The shape would be the chunk shape or index shape depending on the context. c.compute_encoded_size can be implemented for all codecs that produce non-variable-length outputs.

Please note, that these are just implementation suggestions. Some implementations may choose to do this differently.

[normanrz]

I'm a bit confused how c.compute_encoded_size will work. To use this, I have to compute the byte size of the decoded representation? Is there some function to compute the size of the decoded representation?

I think this could indeed be clarified, since we have "array -> array", "array -> bytes", and "bytes -> bytes" codecs to consider. For "array -> array" and "array -> bytes", the input is an array and therefore the relevant size would be the shape of the array, not its size in bytes. For "bytes -> bytes" codecs the input size would be in bytes.

That makes sense. I'll clarify that.

[normanrz]

For "array -> array" and "array -> bytes", the input is an array and therefore the relevant size would be the shape of the array, not its size in bytes.

Actually, this can be complicated for array -> array codecs that change the dtype. For these case, c.compute_encoded_size would need to track the dtype.

[jbms]

Yes, the "array -> array" codec should perhaps also indicate the output dtype.

Possibly it is easier to just leave these implementation details unspecified.

[jstriebel]

LGTM, the generalization of the shard index representation is great 🎉

[normanrz]

Thanks for the feedback! I added some more text to clarify the c.compute_encoded_size signatures.

[mkitti]

The documentation looks improved.

What is the current method to identify if a codec is one of array -> array, array -> bytes, or bytes -> bytes?

[jbms]

The documentation looks improved.

What is the current method to identify if a codec is one of array -> array, array -> bytes, or bytes -> bytes?

You can't tell just from looking at the json, but any supported codec must be known to the implementation, and it is up to the implementation exactly how this is managed.

In tensorstore there is a class for each codec, which inherits from one of ZarrArrayToArrayCodecSpec, ZarrArrayToBytesCodecSpec, or ZarrBytesToBytesCodecSpec; these in turn all inherit from ZarrCodecSpec, which has a virtual method ZarrCodecKind kind() where ZarrCodecKind is enum ZarrCodecKind { kArrayToArray, kArrayToBytes, kBytesToBytes };.

To parse the json codec list, it first parses as a generic list of ZarrCodecSpec and then sorts them into a separate list of array -> array codecs, the array -> bytes codec, and a list of bytes -> bytes codecs.

[normanrz]

Please review and merge if approved. Thanks!