drake (R package)

[wlandau-lilly]

Summary

The drake package is an R-focused pipeline toolkit. It reproducibly brings results up to date and automatically arranges computations into successive parallelizable stages. It has a Tidyverse-friendly front-end, powerful interactive visuals, and a vast arsenal of multicore and distributed computing backends.

Package: drake
Title: Data Frames in R for Make
Version: 4.4.1.9000
Authors@R: c(
  person(
    family = "Landau",
    given = c("William", "Michael"),
    email = "will.landau@lilly.com",
    role = c("aut", "cre")),
  person(
    family = "Axthelm",
    given = "Alex",
    email = "aaxthelm@che.IN.gov",
    role = "ctb"),
  person(
    family = "Clarkberg",
    given = "Jasper",
    email = "jasper@clarkberg.org",
    role = "ctb"),
  person(
    family = "Eli Lilly and Company",
    role = "cph"))
Description: A solution for reproducible code and 
  high-performance computing.
License: GPL-3
Depends:
  R (>= 3.2.0)
Imports:
  codetools,
  crayon,
  eply,
  evaluate,
  digest,
  formatR,
  future,
  grDevices,
  igraph,
  knitr,
  lubridate,
  magrittr,
  parallel,
  plyr,
  R.utils,
  rprojroot,
  stats,
  storr (>= 1.1.0),
  stringi,
  stringr,
  testthat,
  utils,
  visNetwork,
  withr
Suggests: 
  abind,
  DBI,
  future.batchtools,
  MASS,
  methods,
  RSQLite,
  rmarkdown,
  tibble
VignetteBuilder: knitr
URL: https://github.com/wlandau-lilly/drake
BugReports: https://github.com/wlandau-lilly/drake/issues
RoxygenNote: 6.0.1

• URL: https://github.com/wlandau-lilly/drake

• Fit: drake falls easily within reproducibility and high-performance computing.

• Target audience: anyone who uses R for medium-to-long computations for which the results need to stay up to date with the dependencies.

Similar work

Remake

Drake overlaps with its direct predecessor, remake. In fact, drake owes its core ideas to remake and @richfitz, and explicit acknowledgements are in the documentation. However, drake surpasses remake in several important ways, including but not limited to the following.

1. High-performance computing. Remake has no native parallel computing support. Drake, on the other hand, has a vast arsenal of parallel computing options, from local multicore computing to serious distributed computing. Thanks to future, future.batchtools, and batchtools, it is straightforward to configure a drake project for most popular job schedulers, such as SLURM, TORQUE, and the Sun/Univa Grid Engine, as well as systems contained in Docker images.
2. A friendly interface. In remake, the user must manually write a YAML configuration file to arrange the steps of a workflow. In drake, this configuration is based on data frames that built-in wildcard templating functionality easily generates at scale.
3. Thorough documentation. Drake contains eight vignettes, a comprehensive README, examples in the help files of user-side functions, and accessible example code that users can write with drake::example_drake().
4. Active maintenance. Drake is actively developed and maintained, and issues are usually solved promptly.

Factual's drake

Factual's drake is similar in concept, but the development effort is completely unrelated to the R package of the same name.

Other pipeline toolkits

There are many other successful pipeline toolkits, and the drake package distinguishes itself with its R-focused approach, Tidyverse-friendly interface, and parallel computing flexibility.

Requirements

Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• has a CRAN and OSI accepted license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions.
• contains a vignette with examples of its essential functions and uses.
• has a test suite.
• has continuous integration, including reporting of test coverage, using services such as Travis CI, Coeveralls and/or CodeCov.
• I agree to abide by ROpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

Publication options

• Do you intend for this package to go on CRAN?
• Do you wish to automatically submit to the Journal of Open Source Software? If so:
  • The package has an obvious research application according to JOSS's definition.
  • The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
  • The package is deposited in a long-term repository with the DOI:
  • (Do not submit your package separately to JOSS)

I plan to submit to JOSS in the future, but the manuscript is not currently ready.

Detail

• Does R CMD check (or devtools::check()) succeed? Paste and describe any errors or warnings:
• Does the package conform to rOpenSci packaging guidelines? Please describe any exceptions:

[sckott]

Thanks for your submission @wlandau-lilly ! Editors are discussing now

[wlandau-lilly]

Thanks, @sckott.

Edit: I think @richfitz would be an excellent reviewer due to the similarity of remake, but I understand if a potential conflict of interest precludes his participation.

[wlandau-lilly]

I just ran goodpractice::gp() on ropensci/drake@ee475f1:

It is good practice to

  x avoid the attach() and detach() functions, they are
    fragile and code that uses them will probably break sooner than
    later.

    tests/testthat/test-Makefile.R:165:3
    tests/testthat/test-Makefile.R:167:3
    tests/testthat/test-Makefile.R:192:3
    tests/testthat/test-Makefile.R:194:3

  x avoid calling setwd(), it changes the global environment.
    If you need it, consider using on.exit() to restore the working
    directory.

    tests/testthat/test-cache.R:43:3
    tests/testthat/test-cache.R:153:3
    tests/testthat/test-cache.R:216:3
    tests/testthat/test-cache.R:225:3
    tests/testthat/test-cache.R:241:3

I can explain these idiosyncrasies.

Calls to detach() in tests/testthat/test-Makefile.R

Drake promises to load the user's packages, which is especially important for distributed computing across multiple nodes on a cluster. To test, I occasionally need to call detach() to remove packages from search(). Unfortunately, unloadNamespace() does not have the desired effect.

Calls to setwd() in tests/testthat/test-cache.R

By default, drake searches through parent directories to find the current drake project's storr cache. To test, I need to change directories. But rest assured: every test is wrapped in a call to test_with_dir(), which uses withr::with_dir() to ensure that the original working directory is restored. Nested calls to withr::with_dir() give me errors.

[maelle]

Editor checks:

• Fit: The package meets criteria for fit and overlap
• Automated tests: Package has a testing suite and is tested via Travis-CI or another CI service.
• License: The package has a CRAN or OSI accepted license
• Repository: The repository link resolves correctly
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

Editor comments

Thanks for your submission @wlandau-lilly. Running goodpractice::gp() is actually my role but now we have your comments (and I get the same flags) so all is good. 😉

devtool::spell_check identified:

• chioces make.Rd:193

I'm now looking for reviewers.

---

Reviewers: @jules32 @benmarwick @gothub
Due date: 2017-01-04

[maelle]

@wlandau-lilly I forgot to mention you can now add this review badge to the README

[![](https://badges.ropensci.org/156_status.svg)](https://github.com/ropensci/onboarding/issues/156)

[wlandau-lilly]

Thanks, @maelle! I appreciate your forgiveness regarding goodpractice::gp(), and I just fixed the spelling mistake in ropensci/drake@5c9388a. I will add the badge soon, and I am excited for the review process!