Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trouble changing plans #430

Open
wlandau opened this issue Oct 19, 2020 · 5 comments
Open

Trouble changing plans #430

wlandau opened this issue Oct 19, 2020 · 5 comments

Comments

@wlandau
Copy link

wlandau commented Oct 19, 2020

This issue is related to #263 (comment) and ropensci/targets#198 (comment), and it interferes with the ability of targets to create heterogeneous transient workers (cc @mattwarkentin).

When I stick to a multisession plan before running any futures, things seem to work fine.

library(future)
plan_sequential <- plan(multisession, workers = 4)
plan_multisession <- plan()
out <- future(1)
resolved(out)
#> [1] TRUE

Created on 2020-10-19 by the reprex package (v0.3.0)

However, when I try to change to a sequential plan before calling resolved(), the connection gets corrupted.

library(future)
plan_sequential <- plan(multisession, workers = 4)
plan_multisession <- plan()
out <- future(1)
plan(plan_sequential)
resolved(out)
#> Error: Cannot resolve MultisessionFuture (<none>), because the connection to the worker is corrupt: Connection (connection: index=4, description="NA", class="NA", mode="NA", text="NA", opened="NA", can read="NA", can write="NA", id=210, raw_id="<pointer: 0xd2>") is no longer valid. There is currently no registered R connection with that index 4

Created on 2020-10-19 by the reprex package (v0.3.0)

Same thing if I try to restore the original multisession plan.

library(future)
plan_sequential <- plan(multisession, workers = 4)
plan_multisession <- plan()
out <- future(1)
plan(plan_sequential)
plan(plan_multisession)
resolved(out)
#> Error: Cannot resolve MultisessionFuture (<none>), because the connection to the worker is corrupt: Connection (connection: index=4, description="NA", class="NA", mode="NA", text="NA", opened="NA", can read="NA", can write="NA", id=210, raw_id="<pointer: 0xd2>") is no longer valid. There is currently no registered R connection with that index 4

Created on 2020-10-19 by the reprex package (v0.3.0)

@wlandau
Copy link
Author

wlandau commented Oct 19, 2020

This could affect cloud computing strategies such as https://gist.github.com/DavisVaughan/865d95cf0101c24df27b37f4047dd2e5. On the bright side, though, future.callr seems unaffected.

@HenrikBengtsson
Copy link
Owner

It's a hack, but you can avoid the shutdown of the cluster by using:

plan(sequential, .cleanup = FALSE)

wlandau-lilly pushed a commit to ropensci/targets that referenced this issue Oct 20, 2020
@wlandau
Copy link
Author

wlandau commented Oct 20, 2020

Thanks, Henrik! I just committed ropensci/targets@1ede0fb, which uses .cleanup to preserve clusters until the very end of the pipeline. Seems to work. But what exactly makes .cleanup = FALSE a hack?

@HenrikBengtsson
Copy link
Owner

Only that it exists in the first place for internal purposes and I don't want to commit to long term support for it at this time - despite it's been the for a long time already

@wlandau
Copy link
Author

wlandau commented Oct 20, 2020

Will there eventually be a different officially supported way to preserve clusters to allow more heterogeneity among workers?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants