Simple parallel processing implementation #200
d-chambers
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
This post highlights an idea to better support simple parallel processing in DASCore. Ideally, we would implement a
map_to
method on theBaseSpool
. It works like this:It would support passing in a client which has a
map
method (to support parallelization with dask, ray, threading, processing, etc.) as well as a chunksize argument.This works because spools are lazy and can be serialized with pickle.
To make this work though, we need a few more spool methods:
spool.put
- puts a patch into the spool. Not all spools have to implement thisspool.split
- split the spool into sub-spools with certain number of patches in each (slicing already does this)Thoughts? Different ideas?
Beta Was this translation helpful? Give feedback.
All reactions