-
-
Notifications
You must be signed in to change notification settings - Fork 271
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proof of concept: CloudFilesStore #767
Comments
Thanks for this Ryan! I'm having trouble running the notebook on my M1 (several dependencies are x86 only even though I have Rosetta2), but I'll try again with my x86 machine later. I'm very interested to see what the profile says. I'm always interested in increasing performance. One minor quick tip. def setitems(self, values_dict):
self.cf.puts(( (k, v) for k, v in values_dict.items() )) |
Also, it just occurred to me that you mentioned distributed read performance. I've found on virtualized servers that the Python threads jump around between the different cores in a way that doesn't happen on real hardware. Try using the green threads option and you might be pleasantly surprised. You can achieve similar results using |
Specifically, Dask.distributed. In our group, we are usually doing our scaling out via dask. One point I don't understand is how CloudFiles' thread-based parallelism would interact with Dask's multi-faceted parallelism (workers, processes, threads, etc.) |
One way to cope with the additional threading would be to write
n_threads=1. Then the dask threads will determine how many connections are
live.
…On Fri, Jun 4, 2021, 4:19 PM Ryan Abernathey ***@***.***> wrote:
Also, it just occurred to me that you mentioned *distributed* read
performance.
Specifically, Dask.distributed. In our group, we are usually doing our
scaling out via dask. One point I don't understand is how CloudFiles'
thread-based parallelism would interact with Dask's multi-faceted
parallelism (workers, processes, threads, etc.)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#767 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AATGQSOV26TDUCWVSBQDRHLTREYL3ANCNFSM46B7NMZA>
.
|
This looks like a great idea! 👍 Generally you want to use many more parallel threads for IO-bound work (like reading/writing Zarr chunks) than from for compute-bound work (for which you want roughly one thread per core). So I think this nested parallelism is actually probably a good thing -- you want more threads for the IO tasks than you want for the compute tasks. |
Somewhat related discussion about using multiple |
Note that fsspec uses asyncio to fetch multiple chunks concurrently, so this can greatly increase performance by setting each dask partition to be larger than the zarr chunksize. |
We currently rely 100% on fsspec and its implementations for accessing cloud storage (s3fs, gcsfs, adlfs). Cloud storage is complicated, and for debugging purposes, it could be useful to have an alternative. Since I met @william-silversmith a few years ago, I have been curious about CloudFiles:
https://github.com/seung-lab/cloud-files/
Today I coded up a quick CloufFiles-based store for Zarr
In my test with GCS, it works just fine with Zarr, Xarray, and Dask: https://nbviewer.jupyter.org/gist/rabernat/dde8b835bb7ef0590b6bf4034d5e0b2f
Distributed read performance was about 50% slower than gcsfs, but my benchmark is probably biased.
It might be useful to have the option to switch between the fsspec-based stores and this one. If folks are interested, we could think about adding this to zarr-python as some kind of optional alternative to fsspec.
The text was updated successfully, but these errors were encountered: