Array position aware Bytes to Bytes Codec

[carloshorn]

I am experimenting with OpenZL as Zarr codec, see https://github.com/carloshorn/openzl4zarr/.

OpenZL is an interesting compression library because it provides methods for training specialized compressors while including a universal decompressor. Consequently, no additional instructions or configuration parameters are required for deserialization within the Zarr codec, as would be the case with trained compression dictionaries.

My current implementation covers the case where I use a pre-trained compressor that I apply to all chunks during writing.
I can use different compressors if I write the respective chunks using different pipelines. However, this is not very user-friendly and also does not allow for sharding.

The precise question I have:
Is there a way to get the array position of a chunk within the BytesBytesCodec._encode_sync method?

I am aware that this is not trivial, as an array-array codec can scramble positions, for instance, via a Fourier transform, resulting in a downstream sharding codec providing chunks that no longer necessarily bear a relationship to the original positions.

One possible solution I can envision is a custom sharding codec that will take at least the "position within the shard" into account and executes my pipeline; subsequently, the attributes in the zarr.json would be adjusted so that it appears as though the data had been generated using standard sharding and my openZL bytes-bytes codec. Sounds a bit hacky to me...

Thank you all in advance for any ideas or input.

[d-v-b]

The precise question I have: Is there a way to get the array position of a chunk within the BytesBytesCodec._encode_sync method?

no this is not possible, and we would not want to add this without coordination with all other implementations. the current model of chunk encoding as defined by the spec keeps the chunk index out of the signature.

One possible solution I can envision is a custom sharding codec that will take at least the "position within the shard" into account and executes my pipeline;

the sharding codec effectively does this today when fetching a subchunk. but it applies the exact same set of inner codecs to decode subchunks, no matter where they are.

A basic model of zarr so far is that a chunk decoding procedure is fully specified by a single JSON document. additional external resources like dictionaries that vary per-chunk are thus not in the picture right now. What we could do is define constructs for concatenating separate Zarr arrays, which each have their own separate chunk encodings. This would be useful more generally.

[carloshorn]

A basic model of zarr so far is that a chunk decoding procedure is fully specified by a single JSON document. additional external resources like dictionaries that vary per-chunk are thus not in the picture right now.

That is exactly the advantage of OpenZL over dictionary training. The decoder does not need any encoder specific parameters.
Therefore, I could live with a not too portable encoder implementation at the moment.

[d-v-b]

I think we could enable this if we model the training as runtime parametriztion for the codec that isn't part of the serialized representation. this requires some new APIs but I think it's interesting enough to pursue. The simplest version is just adding parameters to the routine that writes chunks, something like:

def write_chunks(self, selection, data, codec_options: Sequence[Mapping[str, object]]) -> WriteResult: ...

CodecOptions would then be an ordered sequence of arguments fed to each of the codecs in the codec chain. Because the composition of a given chain of codecs is dynamic, we can't get a narrow type hint here.

An alternative is to imbue each codec with a runtime configuration, and make it cheap to copy codecs while modifying the runtime config, and to chain this under copying an array with new codecs e.g:

array.with_codecs(array.codecs[:3], array.codecs[3].with_config(new_config), array.codecs[4:])

I'm less keen on this API

[d-v-b]

one complication here is that, broadly, the JSON configuration for the codec found in zarr.json declares how another implementation can encode the same chunk to get ~the same encoded bytes. There are a few exceptions (i think gzip puts a timestamp in the bytestream, so that can't be replicated), and as far as I know this principle isn't actually codified in the spec.

How would this work for openzl? What would we need to express in the JSON configuration to make training reproducible?

[carloshorn]

Specifying a reproducible encoding within zarr.json is not straightforward to implement. In an extreme scenario, each chunk could use a different compressor. Updating zarr.json every time a new compressor is used would be highly inefficient; furthermore, a mapping between compressors and chunks would need to be defined.
In theory, it should be possible to extract the compressor from the specific chunk and use it to re-compress the data to obtain the exact same byte sequence. After all, OpenZL stores the compression graph, which is precisely what enables the use of a universal decompressor. However, I find it hard to imagine that such reverse engineering of the compressor is intended within OpenZL.
In this theoretical sense, it is possible to derive reproducible compressors from the compressed data to generate the exact same bytes. To me, this sounds similar enough to draw a comparison with the timestamps found in gzip compressed chunks.
What do you think?

[d-v-b]

In this theoretical sense, it is possible to derive reproducible compressors from the compressed data to generate the exact same bytes. To me, this sounds similar enough to draw a comparison with the timestamps found in gzip compressed chunks.
What do you think?

I think the spirit of the spec so far has been to treat the configuration of codecs as a locus for all the parameters required for deterministically encoding the same array into the same chunks on different platforms, within the limitations of the encoding method (e.g., the gzip timestamp thing). I believe the proposal here -- a codec where the encoding method was only reproducible after examining an encoded chunk -- would violate this spirit, and would ultimately be problematic for the ecosystem. The potential storage savings are large, so this seems like a problem zarr needs to solve e.g. with some kind of efficient representation of per-chunk metadata. I have one proposal that essentially allows nesting zarr arrays which could be useful here, but I would love to explore other options.