unlistable stores

[rabernat]

What would it take to get zarr to work with a store that did not support any list operations? I'm not talking about implementing listdir...I mean stores that don't even support list(store.keys()).

For a given zarr group, once you know the names of the arrays, in principle you don't need to list the objects in the store at all. You just go get the .zattrs and .zarray data for each array, which tells you where the chunks should be.

I'm thinking about this in the context of cloud storage (#290), where operations can be expensive and complicated to implement.

One way to implement this would be to update the spec to allow .zgroup to contain a list of arrays contained in the group. If present, the step to list the store contents could be bypassed.

Just an idea I had that I wanted to write down before I forgot.

[martindurant]

I'm sure this is totally doable now - at least, you can have a contains that is implemented by trying to fetch a key and returnign False if that fails; but I would instead go the route of consolidated metadata in #268 . That metadata can also be stored in a separate location from the data itself.

[alimanfoo]

Just to echo @martindurant, using consolidated metadata should I think mean that all calls to listdir() or store.keys() could be avoided.