Define system architecture

[alimanfoo]

Zarr and n5 naturally decouple different components of their architecture, in a way that allows clear and simple interfaces to be defined between them, and that allows for pluggability. For example, both define a storage layer interface, based on storage and retrieval of key/value pairs, which can then have pluggable implementations (file system, S3, mongodb, ...). Zarr (via numcodecs) also defines a codec API which enables new filters or compressors to be plugged in.

It could be very helpful to draw a picture of this system architecture, and then to discuss what aspects of the architecture should be covered in a "core" spec, versus where additional specs can be layered on top or plugged in.

For example, we might have a core spec that formally defines the system architecture, defines abstract APIs for storage and chunk encoding/decoding, and defines how essential metadata are formatted. Then we might have a separate spec for each storage layer implementation, that defines how keys and values are mapped into concrete storage entities like file paths and file contents. And we might have a separate spec for each codec, that defines the encoding process and format.

In other words, a modular spec architecture, that allows new specs for things like storage or encoding to be plugged in without affecting the core spec.

[alimanfoo]

OK, here's a first go at drawing something, based on our conversation today on the zarr/n5 call:

This is not complete and a bad drawing (apologies) but hopefully captures a couple of useful concepts, including:

• Array metadata
  • Filter chain metadata
• Codec interface
• Codec implementations
• Storage interface
• Storage transformations
• Storage implementations

We discussed all of the above today, except for storage transformations, which I've added in - this is just to capture the idea that you may want to define transformations on the keys and/or values, before they hit the storage implementation.

[alimanfoo]

Just adding another thought, the consolidated metadata feature could also be an example of a storage transformation.

[joshmoore]

On the call that @alimanfoo refers to, @axtimwalde brought up the idea of having the minimum API be simply the writing of byte arrays/streams. I still find this intriguing, so a few thoughts that have been rattling around:

• I think I gave up too early on endianness. I disagree that byte order is not important for portability. It should either be specified in the spec even at this low-level or detectable in the byte stream itself.
• There was a statement that the shape does not apply at this "no-memory-layout". At least conceptually, that leads me to believe we're talking about a "zobject" as opposed to a "zarray". chunks still apply and is worded nicely since you're saying, "break this stream into num_chunks = d1 x d2 x ... x dn".
• "zarrays" then conceptually subclass "zobject" to turn those num_chunks byte arrays into an ND-array by applying a memory layout on top of the streams.
• But I still have the inclination to say that an array of memory layouts could be placed on top of the stream to product multiple objects: List<Object> applyMemoryLayouts(InputStream...)

[jstriebel]

There are now two informative illustrations in the v3 spec which go into this direction:
https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#concepts-and-terminology

Defining a system architecture in the spec more concretely might also be imposing too many restrictions on impelementations.

@alimanfoo @joshmoore Do you feel like the important parts are covered for now?

[joshmoore]

Having one image that gets all the base concepts across is probably something to shoot for eventually but need not be exactly the one above nor done immediately. So 👍 if you want to close from my side.