Document changes to async.concurrency defaults and performance considerations. #3528
Conversation
| Lower concurrency values may be beneficial when: | ||
| - Working with local storage with limited I/O bandwidth | ||
| - Memory is constrained (each concurrent operation requires buffer space) | ||
| - Using Zarr within a parallel computing framework (see below) |
There was a problem hiding this comment.
This is exactly this use case I ran into 😄 - thanks a lot for putting this together, it's a really nice summary.
There was a problem hiding this comment.
@dstansby Could you share a bit of specifics as to what you saw that might have indicated this setting as the culprit?
I like having examples, especially :)
There was a problem hiding this comment.
I wanted to run multiple parallel jobs, each one which writes to a shard, and manually run as many jobs as I had processors.
There was a problem hiding this comment.
Mmm, interesting, and this was causing consistency issues? I have some sharded dask writing but haven't noticed any issues. 2 jobs would try writing to the same shard?
There was a problem hiding this comment.
Ah, it wasn't causing any data issues, just performance issues. I was manually writing to one shard per process, and because my data was local on a fast SSD async concurrency wouldn't have bought me anything, and the internal multithreading that zarr-python does was causing more threads than I had processors to spin up without me wanting/needing them.
There was a problem hiding this comment.
Mmmm interesting, I think zarrs just has a global pool (at least I think that's rayon's default) so that might explain why I haven't hit this issue. Good to know!
4 participants
As requested in #3526, this PR adds some documentation about the change to and performance considerations of the async.concurrency config setting.
TODO:
docs/user-guide/*.mdchanges/