-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Question] How to stop a parallel loop when an error has been raised #213
Comments
This will be a long reply with lots of details: To understand how to achieve "early stopping", it's important to understand that there is no way for a parallel R workers to "signal" back to the master process. The only way the master process can know whether a future has produces an error or not is for the master process to "ask" the worker. In other words, we need to poll the futures in order to detect errors. You can poll a future for errors by explicitly retrieving its value, i.e. There is another way to signal errors early. I've kept this feature a little bit under the radar - by specifying
What's important to understand is that, just explained above, errors from this future will only be signalled when that future is polled. When does that happen? It happens when there are no more workers available - then the future framework starts polling existing futures to check if they're done. As soon as one is done, it'll collect it's value _and if the future that is collected happens to produce and error and was created using A poor man's approach may then be to continuously loop over the existing futures and check if they're resolved (which is non-blocking) and only then query the value (blocking). Something like: signal_errors_early <- function(futures) {
for (future in futures) if (resolved(future)) value(future)
invisible()
}
for(i in 1:100)
{
signal_errors_early(x[seq_len(i - 1L)])
x[[i]] <- future::future({tryCatch(h(), error = function(e){ write(toString(e), log) })})
[...]
} Note that About > library("future.apply")
> plan(multiprocess, workers = 3L)
> system.time(y <- future_lapply(1:3, FUN = function(ii) { if (ii == 2) stop("boom"); Sys.sleep(10) }))
Error in ...future.FUN(...future.X_jj, ...) : boom
Timing stopped at: 0.58 0.076 10.19 It could be that you concluded this because chunking takes place by default, so if you I'll add "early signalling" to the list of features that could be improved, e.g. it might be that the above |
Thank you very much for this long and detailed answer. The good new is that my understanding of how I like the |
This a question I asked on stackoverflow. Nobody answered so I'm annoying you here 😉
I create parallel loops using
future
. Sometime the expression called raise an error. In that cases the whole process is ran (takes a long time) and the function fails only at the end when the futures are evaluated.Instead the function should fail when an error is raised. Especially if the error is raised at the beginning of the process. For example this code will fail only after 100 iterations but it is actually likely to fail before the 10th one.
How to manage error handling with
future
? I perfectly understand why it is difficult but in the meantimefuture_lapply
can do it so it is possible.Btw, I'm actually able to do it but to me it looks more like a kind of hack than a real solution. I'm using a temporary file to communicate the error to the main process.
Thanks
The text was updated successfully, but these errors were encountered: