You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In my massively-parallel processing as part of MapLibre's Martin code I use for_each_concurrent to run hundreds of millions of tasks in parallel. I was trying to understand the point you were making in the docs about its inefficiencies, or if I am using it wrong, but I might have misunderstood. Is this the right approach, or should I go with some alternative that involves scoped async? Thanks!
The text was updated successfully, but these errors were encountered:
The signature of the "map" function in [for_each_concurrent] is F: FnMut<...>. This implies that the "map" function is never executed parallely. The futures in the stream are awaited concurrently, but not in parallel (the poll method is never run simultaneously in multiple CPU cores). Essentially, I/O, signals, etc. are awaited concurrently across the futures.
For parallelism, I've typically used rayon for CPU heavy algorithm parallelization. Another option is to use the spawn function of the executor (tokio/async-std/...), to spawn separate futures that are then run parallelly. Then, we simply await their join-results concurrently. In this case, the F is essentially no-op, so it's okay to not schedule it in parallel. Now, spawn expects 'static future, so this crate provides an ergo. API to spawn local futures (not 'static).
In my massively-parallel processing as part of MapLibre's Martin code I use
for_each_concurrent
to run hundreds of millions of tasks in parallel. I was trying to understand the point you were making in the docs about its inefficiencies, or if I am using it wrong, but I might have misunderstood. Is this the right approach, or should I go with some alternative that involves scoped async? Thanks!The text was updated successfully, but these errors were encountered: