Parallel regridder with dask distributed #321
-
|
I tried to generate the weights for the regridder with parallel = True while using a Dask distributed client activated and I receive the following warning: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results. From my experience using Dask, this happen when you use Dask but not using the distributed client. Is there a way to use the client instead? Cheers, |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Hi @fedecutraro, Generating the weights in parallel is a recent feature and is still experimental. Due to the difficulties of running ESMF itself (the compiled fortran program) in a multi-thread setting. This means that we can't use the distributed client (which mixes both threads and processes)1 so the current implementation forces dask to use only processes, leading to the warning message you saw. Once the weights are computed, they are applied according to dask current settings, using your distributed client. Note that parallel generation is only useful if the output grid is very large (i.e. you are upscaling your data). Footnotes
|
Beta Was this translation helpful? Give feedback.
Hi @fedecutraro,
Generating the weights in parallel is a recent feature and is still experimental. Due to the difficulties of running ESMF itself (the compiled fortran program) in a multi-thread setting. This means that we can't use the distributed client (which mixes both threads and processes)1 so the current implementation forces dask to use only processes, leading to the warning message you saw.
Once the weights are computed, they are applied according to dask current settings, using your distributed client. Note that parallel generation is only useful if the output grid is very large (i.e. you are upscaling your data).
Footnotes
Or at least, nobody has yet found a clean way to use …