Open Multiple Zarrs

[peterdudfield]

Thanks so much for this package - it has been super useful for us. We are currently using this instead of dask backend to xarray, and its gives us some massive speeds ups - so thank you. It should mean we can stream the data, rather than having to make lots of samples first.

OCF internal github issue and (currently draft) PR

One thing we had to implement was opening multiple zarr files at once. This is because we currently like to store data in either monthly or yearly zarrs - this makes data processing slightly easier. Of course one option is we could process them into one big zarr.

The code we've written is here and seems to work.

In order to benefit the community, is it worth trying to add this feature into this repo?
We could do a PR for you, but I wanted to check before making one.

Thanks again and credit to @dfulu for lots of work on this

[shoyer]

Thanks Peter for sharing and glad you have found xarray-tensorstore useful! Apologies for not noticing this issue earlier.

Making yearly/monthly Zarrs feels like a bit of an anti-pattern to me, although I'm sure I'm missing some nuance.

That said, your implementation is quite minimal and I like the usage of ts.concat(), so I would be happy to review PR. Maybe call this open_concatenated_zarrs()?

[peterdudfield]

Thanks @shoyer for this. Yea, I know what you mean year/montly zarrs seems a bit of an anti-pattern. I think there's still enough reasons (sometimes, not always) to open multiple zarrs.

Thanks, Ill let @dfulu, make a PR for this