Dispatch on uri protocol

[mrocklin]

We might consider a convenience method to build a mutable mapping that consumes strings of various forms:

• /path/to/directory
• /path/to/file.zip
• s3://bucket/directory
• gcs://bucket/directory
• ...

My first instinct is to say that this is out of scope for Zarr, however I don't know where exactly it is in scope (maybe zict?) I thought I'd bring it up here for feedback.

We handle something like this for file systems in Dask by registering functions that import the appropriate module when necessary. See https://github.com/dask/dask/blob/master/dask/bytes/core.py#L478

[alimanfoo]

This is a nice idea, I have wondered about doing this. There is the beginnings of it here https://github.com/alimanfoo/zarr/blob/master/zarr/creation.py#L120. For cloud stores, is the URL-style string enough, or will the user commonly need to provide more configuration parameters (e.g., for authentication)? I guess I'm wondering if there is a need to be able to accept and pass extra keywords through to the store, or just supporting the URL-style string is enough.

…

On Wednesday, December 6, 2017, Matthew Rocklin ***@***.***> wrote: We might consider a convenience method to build a mutable mapping that consumes strings of various forms: - /path/to/directory - /path/to/file.zip - s3://bucket/directory - gcs://bucket/directory - ... My first instinct is to say that this is out of scope for Zarr, however I don't know where exactly it is in scope (maybe zict?) I thought I'd bring it up here for feedback. We handle something like this for file systems in Dask by registering functions that import the appropriate module when necessary. See https://github.com/dask/dask/blob/master/dask/bytes/core.py#L478 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/214>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QpYe8-lPj4nGfEV1dVppk8U_-4Qzks5s9d-kgaJpZM4Q3LbB> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[alimanfoo]

Btw I am conscious there is some duplication of functionality between zict and zarr.storage at the moment. Happy to contribute anything back to zict if you think appropriate. On Wednesday, December 6, 2017, Alistair Miles <alimanfoo@googlemail.com> wrote:

…

This is a nice idea, I have wondered about doing this. There is the beginnings of it here https://github.com/alimanfoo/zarr/blob/master/ zarr/creation.py#L120. For cloud stores, is the URL-style string enough, or will the user commonly need to provide more configuration parameters (e.g., for authentication)? I guess I'm wondering if there is a need to be able to accept and pass extra keywords through to the store, or just supporting the URL-style string is enough. On Wednesday, December 6, 2017, Matthew Rocklin ***@***.***> wrote: > We might consider a convenience method to build a mutable mapping that > consumes strings of various forms: > > - /path/to/directory > - /path/to/file.zip > - s3://bucket/directory > - gcs://bucket/directory > - ... > > My first instinct is to say that this is out of scope for Zarr, however I > don't know where exactly it is in scope (maybe zict?) I thought I'd bring > it up here for feedback. > > We handle something like this for file systems in Dask by registering > functions that import the appropriate module when necessary. See > https://github.com/dask/dask/blob/master/dask/bytes/core.py#L478 > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <https://github.com/alimanfoo/zarr/issues/214>, or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAq8QpYe8-lPj4nGfEV1dVppk8U_-4Qzks5s9d-kgaJpZM4Q3LbB> > . > -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: ***@***.*** Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[mrocklin]

In Dask we pass a storage_options={...} keyword argument any time we accept a URI

[mrocklin]

Btw I am conscious there is some duplication of functionality between zict and zarr.storage at the moment. Happy to contribute anything back to zict if you think appropriate.

We have Zip and Directory MutableMappings at the moment, but I suspect that they differ a bit. It would be interesting to see how they differ and how Zict might learn from Zarr.

[joshmoore]

This is largely covered now by FSStore, no?

cc: @martindurant

[martindurant]

Yes, zarr can essentially do what dask was starting to do when this conversation was still live. Dispatch happens via fsspec if the URI looks fsspec-lke ("protocol://path"), we don't dispatch to any of the other storage types.