making stop() immediate

[rickhelmus]

Hello,

Consider the following simple example:

library(future.apply)
plan(cluster, workers = 4)

future_lapply(1:4, function(x)
{
    if (x == 2)
        stop("x==2")
    Sys.sleep(10)
})

Ideally in my scenario this would stop immediately with an error message, but instead it takes ten seconds and then the error is thrown. Is there a way to change this? I saw a somewhat related discussion on throwing custom conditions, which I guess could also work, but to me the above situation seemed like a fairly common use case to have.

Thanks,
Rick

[HenrikBengtsson]

Yes, this "early stopping" feature is frequently requested. In order to be able to do this, we need to be able to terminate the other three futures that were launched at the same time as the future that produced the error. Being able to terminate futures is something that is on the roadmap and it needs to be resolved before even being able to add that option to future_lapply() and friends.

FYI, the first place where this has to be implemented is in the parallelly package (futureverse/parallelly#33), then it'll have to be added to the future package (futureverse/future#93). Only when that is in place, it can be added to future.apply (or furrr, or doFuture). So, this is something that will take time. It's not something easy to fix but hopefully doable.

[rickhelmus]

Hi Henrik,

Thanks for the quick response! And thanks for the clear explanation on the current situation. Looking forward to when you get around to implement all these changes!

[HenrikBengtsson]

This will be supported by default in the next version of future (> 1.42.0) and the next version of future.apply (> 1.11.3).

[HenrikBengtsson]

future.apply 1.20.0, which cancels outstanding futures and frees up their workers on error, is now on CRAN.