Staged parallelism for the drake R package

With staged parallelism, drake partitions the dependency graph into stages of conditionally independent targets and processes each stage with semi-transient parallel workers. This functionality is already deprecated, and it will be removed at some point later on.

Installation

library(remotes)
install_github("ropensci/drake")
install_github("wlandau/drake.future.lapply.staged")

Usage

We begin with a drake project.

library(drake.future.lapply.staged)
plan <- drake_plan(x = rnorm(100), y = mean(x), z = median(x))

plan
#> # A tibble: 3 x 2
#>   target command   
#>   <chr>  <chr>     
#> 1 x      rnorm(100)
#> 2 y      mean(x)   
#> 3 z      median(x)

First, create a future plan. See the future README and future.batchtools README for guidance, and consult tables here and here for options for your plan.

library(future)
plan(multiprocess)

Next, run your drake project.

library(drake.future.lapply.staged)
make(plan, parallelism = backend_future_lapply_staged, jobs = 2)
#> Warning: `drake` can indeed accept a custom scheduler function for the
#> `parallelism` argument of `make()` but this is only for the sake of
#> experimentation and graceful deprecation. Your own custom schedulers may
#> cause surprising errors. Use at your own risk.
#> Warning: Staged parallelism for drake is deprecated and will be removed
#> eventually.