Prototype rust implementation

[clbarnes]

What started as some fiddling with the sharding binary format has sort of snowballed into an implementation of most of the v3 spec. It is dirty, with lots of rough edges where I've punted on a bunch of API decisions, and extremely verbose (whether that's necessary is TBD). It's serialised and synchronous, arrays are copied willy-nilly, allocations are everywhere, and despite rust's best efforts there are a bunch of race conditions in there, but it at least implements:

• file system and in-memory stores, which both support partial IO
• default and V2 chunk key encoding
• 100% of specified storage transformers ;)
• types for convenient and safe key handling
• all codecs, including sharding (BUT no partial en/decode for any, so there's not much value in the sharding yet)
• array IO for chunks and non-chunk-aligned regions

https://github.com/clbarnes/zarr3-rs

(N.B. most of the (scarce) hints of sanity in the design can probably be traced to the earlier prototype led by @aschampion https://github.com/sci-rs/zarr)

[rabernat]

Thanks for sharing this @clbarnes!

I did not see this post until just now, which makes me think that we should not have a discussion forum on the zarr-specs repo.

[Am3ra]

same, very interested in rust zarr :)

[clbarnes]

Is image.sc the right place for this kind of thing, do you think?

[ivirshup]

Also just seeing this. Maybe the zarr-developers discussions (here)? Maybe @MSanKeys963 has a suggestion?

image.sc seems too domain specific for this to me.

[joshmoore]

• Seconded that image.sc is too domain specific.
• Seconded that the multi-discussions-verse is confusing.
• Propose to move as proposed by @ivirshup (I'll do so this week if there are no objections).
• Then @MSanKeys963 and I will look into a strategy for the discussions. (Comments and help welcome!)

[joshmoore]

Edit: but announcing ALSO on image.sc (and if @rabernat concurs, pangeo, etc.) is "A Good Thing" ™️. Maybe there's a bot in our future to do the re-posting for us...