Trouble changing plans

[wlandau]

This issue is related to #263 (comment) and https://github.com/wlandau/targets/issues/198#issuecomment-712433554, and it interferes with the ability of targets to create heterogeneous transient workers (cc @mattwarkentin).

When I stick to a multisession plan before running any futures, things seem to work fine.

library(future)
plan_sequential <- plan(multisession, workers = 4)
plan_multisession <- plan()
out <- future(1)
resolved(out)
#> [1] TRUE

^{Created on 2020-10-19 by the reprex package (v0.3.0)}

However, when I try to change to a sequential plan before calling resolved(), the connection gets corrupted.

library(future)
plan_sequential <- plan(multisession, workers = 4)
plan_multisession <- plan()
out <- future(1)
plan(plan_sequential)
resolved(out)
#> Error: Cannot resolve MultisessionFuture (<none>), because the connection to the worker is corrupt: Connection (connection: index=4, description="NA", class="NA", mode="NA", text="NA", opened="NA", can read="NA", can write="NA", id=210, raw_id="<pointer: 0xd2>") is no longer valid. There is currently no registered R connection with that index 4

^{Created on 2020-10-19 by the reprex package (v0.3.0)}

Same thing if I try to restore the original multisession plan.

library(future)
plan_sequential <- plan(multisession, workers = 4)
plan_multisession <- plan()
out <- future(1)
plan(plan_sequential)
plan(plan_multisession)
resolved(out)
#> Error: Cannot resolve MultisessionFuture (<none>), because the connection to the worker is corrupt: Connection (connection: index=4, description="NA", class="NA", mode="NA", text="NA", opened="NA", can read="NA", can write="NA", id=210, raw_id="<pointer: 0xd2>") is no longer valid. There is currently no registered R connection with that index 4

^{Created on 2020-10-19 by the reprex package (v0.3.0)}

[wlandau]

This could affect cloud computing strategies such as https://gist.github.com/DavisVaughan/865d95cf0101c24df27b37f4047dd2e5. On the bright side, though, future.callr seems unaffected.

[HenrikBengtsson]

It's a hack, but you can avoid the shutdown of the cluster by using:

plan(sequential, .cleanup = FALSE)

[wlandau]

Thanks, Henrik! I just committed wlandau/targets@1ede0fb, which uses .cleanup to preserve clusters until the very end of the pipeline. Seems to work. But what exactly makes .cleanup = FALSE a hack?

[HenrikBengtsson]

Only that it exists in the first place for internal purposes and I don't want to commit to long term support for it at this time - despite it's been the for a long time already

[wlandau]

Will there eventually be a different officially supported way to preserve clusters to allow more heterogeneity among workers?

[HenrikBengtsson]

FWIW, though it does not address the original question of keeping parallel backends for non-resolved futures around until they've been resolved, the upcoming withPlan() and localPlan() avoids this problem in the case one only wants to change plan temporarily. Proof of concept:

library(future)
plan(multisession, workers = 4)

f <- future(1)

withPlan(sequential, {
  message("Temporarily using the 'sequential' plan")
})
#> Temporarily using the 'sequential' plan

r <- resolved(f)
print(r)
#> [1] FALSE

v <- value(f)
v
#> [1] 1

[HenrikBengtsson]

I'm actually going to close this. The idea on "keeping parallel backends for non-resolved futures around until they've been resolved" is now tracked in #309 (comment).