A much friendlier API for creating large plans

[wlandau]

Summary

This PR combines concepts from the proposed DSL (#233) and target(). Both were amazingly helpful ideas from @krlmlr. Also cc @AlexAxthelm, @dapperjapper, @tiernanmartin, @tmastny, @lorenzwalthert, and @idavydov. Hopefully we can work this into the delayed expansion idea from #233.

The implementation is still experimental, but it is noninvasive. It is lightweight, and because it relies on a custom "transform" column in the plan, it does not interfere with any other functionality (internals or API).

library(drake)
plan <- drake_plan(
  small = simulate(48),
  large = simulate(64),
  reg = target(
    reg_fun(data),
    transform = cross(reg_fun = c(reg1, reg2), data = c(small, large))
  ),
  summ = target(
    sum_fun(data, reg),
    transform = cross(sum_fun = c(coef, residuals), reg)
  ),
  winners = target(
    min(summ),
    transform = summarize(data, sum_fun)
  )
)

plan
#> # A tibble: 18 x 2
#>    target                 command                                          
#>    <chr>                  <chr>                                            
#>  1 small                  simulate(48)                                     
#>  2 large                  simulate(64)                                     
#>  3 reg_reg1_small         reg1(small)                                      
#>  4 reg_reg2_small         reg2(small)                                      
#>  5 reg_reg1_large         reg1(large)                                      
#>  6 reg_reg2_large         reg2(large)                                      
#>  7 summ_coef_reg_reg1_la… coef(large, reg_reg1_large)                      
#>  8 summ_residuals_reg_re… residuals(large, reg_reg1_large)                 
#>  9 summ_coef_reg_reg1_sm… coef(small, reg_reg1_small)                      
#> 10 summ_residuals_reg_re… residuals(small, reg_reg1_small)                 
#> 11 summ_coef_reg_reg2_la… coef(large, reg_reg2_large)                      
#> 12 summ_residuals_reg_re… residuals(large, reg_reg2_large)                 
#> 13 summ_coef_reg_reg2_sm… coef(small, reg_reg2_small)                      
#> 14 summ_residuals_reg_re… residuals(small, reg_reg2_small)                 
#> 15 winners_large_coef     min(summ_coef_reg_reg1_large = summ_coef_reg_reg…
#> 16 winners_small_coef     min(summ_coef_reg_reg1_small = summ_coef_reg_reg…
#> 17 winners_large_residua… min(summ_residuals_reg_reg1_large = summ_residua…
#> 18 winners_small_residua… min(summ_residuals_reg_reg1_small = summ_residua…

drake_plan_source(plan)
#> drake_plan(
#>   small = simulate(48),
#>   large = simulate(64),
#>   reg_reg1_small = reg1(small),
#>   reg_reg2_small = reg2(small),
#>   reg_reg1_large = reg1(large),
#>   reg_reg2_large = reg2(large),
#>   summ_coef_reg_reg1_large = coef(large, reg_reg1_large),
#>   summ_residuals_reg_reg1_large = residuals(large, reg_reg1_large),
#>   summ_coef_reg_reg1_small = coef(small, reg_reg1_small),
#>   summ_residuals_reg_reg1_small = residuals(small, reg_reg1_small),
#>   summ_coef_reg_reg2_large = coef(large, reg_reg2_large),
#>   summ_residuals_reg_reg2_large = residuals(large, reg_reg2_large),
#>   summ_coef_reg_reg2_small = coef(small, reg_reg2_small),
#>   summ_residuals_reg_reg2_small = residuals(small, reg_reg2_small),
#>   winners_large_coef = min(summ_coef_reg_reg1_large = summ_coef_reg_reg1_large, summ_coef_reg_reg2_large = summ_coef_reg_reg2_large),
#>   winners_small_coef = min(summ_coef_reg_reg1_small = summ_coef_reg_reg1_small, summ_coef_reg_reg2_small = summ_coef_reg_reg2_small),
#>   winners_large_residuals = min(
#>     summ_residuals_reg_reg1_large = summ_residuals_reg_reg1_large,
#>     summ_residuals_reg_reg2_large = summ_residuals_reg_reg2_large
#>   ),
#>   winners_small_residuals = min(
#>     summ_residuals_reg_reg1_small = summ_residuals_reg_reg1_small,
#>     summ_residuals_reg_reg2_small = summ_residuals_reg_reg2_small
#>   )
#> )

config <- drake_config(plan)
vis_drake_graph(config)

^{Created on 2019-01-14 by the reprex package (v0.2.1)}

Related GitHub issues and pull requests

• Ref: DSL based on dplyr-like verbs? #233

Checklist

• I have read drake's code of conduct, and I agree to follow its rules.
• I have listed any substantial changes in the development news.
• I have added testthat unit tests to tests/testthat to confirm that any new features or functionality work correctly.
• I have tested this pull request locally with devtools::check()
• This pull request is ready for review.
• I think this pull request is ready to merge.

[codecov-io]

Codecov Report

Merging #674 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #674   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files          71     72    +1     
  Lines        5512   5588   +76     
=====================================
+ Hits         5512   5588   +76

Impacted Files	Coverage Δ
R/utils-utils.R	100% <100%> (ø)	⬆️
R/api-transform.R	100% <100%> (ø)
R/api-plan.R	100% <100%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09917f8...4de099f. Read the comment docs.