Submission: tic

[krlmlr]

Submitting Author: Kirill Müller (@krlmlr)
Repository: ropenscilabs/tic
Version submitted: 0.2.13.0916
Editor: @annakrystalli
Reviewer 1: @ldecicco-USGS
Reviewer 2: @maxheld83
Archive: TBD
Version accepted: 0.3.0.9004

---

DESCRIPTION file

Type: Package
Package: tic
Title: Tasks Integrating Continuously: CI-Agnostic Workflow Definitions
Version: 0.2.13.9016
Authors@R: c(
    person("Kirill", "M\u00fcller", email = "krlmlr+r@mailbox.org", role = c("aut", "cre"), comment = c(ORCID = "0000-0002-1416-3412")),
    person("Patrick", "Schratz", email = "patrick.schratz@gmail.com", role = "aut", comment = c(ORCID = '0000-0003-0748-6624')),
    person("Mika", "Braginsky", email = "mika.br@gmail.com", role = c("aut")),
    person("Karthik", "Ram", email = "karthik.ram@gmail.com", role = c("aut")),
    person("Jeroen", "Ooms", email = "jeroen.ooms@stat.ucla.edu", role = c("aut")),
    person("rOpenSci", role = c("fnd"))
  )
Description: 
    Provides a way to describe common build and deployment workflows for R-based
    projects: packages, websites (e.g. blogdown, pkgdown), or data processing
    (e.g. research compendia).  The recipe is described independent of the
    continuous integration tool used for processing the workflow (e.g.
    'Travis CI' or 'AppVeyor').
License: GPL (>= 2)
URL: https://github.com/ropenscilabs/tic
BugReports: https://github.com/ropenscilabs/tic/issues
Depends: 
    R (>= 3.0.0)
Imports: 
    backports,
    cli,
    crayon,
    magrittr,
    memoise,
    methods,
    remotes,
    R6,
    rlang,
    withr
Suggests: 
    base64enc,
    bookdown,
    callr,
    covr,
    devtools,
    drat,
    fansi,
    git2r,
    knitr,
    openssl,
    rcmdcheck,
    rmarkdown,
    testthat,
    travis (>= 0.2.11.9001),
    usethis
VignetteBuilder: knitr
Remotes: 
    r-lib/rcmdcheck,
    ropenscilabs/travis
ByteCompile: No
Encoding: UTF-8
LazyData: TRUE
Roxygen: list(markdown = TRUE)
RoxygenNote: 6.1.1
Collate: 
    'ci.R'
    'appveyor.R'
    'base64.R'
    'dsl-storage.R'
    'dsl.R'
    'git2r.R'
    'keys.R'
    'local.R'
    'macro.R'
    'macro-package-checks.R'
    'macro-pkgdown.R'
    'macro-bookdown.R'
    'mock.R'
    'print.R'
    'repo.R'
    'run.R'
    'stage.R'
    'steps-base.R'
    'steps-bookdown.R'
    'steps-code.R'
    'steps-drat.R'
    'steps-git.R'
    'steps-install.R'
    'steps-rcmdcheck.R'
    'steps-pkgdown.R'
    'steps-ssh.R'
    'steps-write-text-file.R'
    'tic-package.R'
    'travis.R'
    'use_tic.R'
    'utils.R'

Scope

• Please indicate which category or categories from our package fit policies this package falls under:

  • data retrieval
  • data extraction
  • database access
  • data munging
  • data deposition
  • reproducibility
  • geospatial data
  • text analysis

• Explain how and why the package falls under these categories (briefly, 1-2 sentences):

Provides a better interface to CI systems: define CI/CD using R code instead of YAML files. Work supported by rOpenSci.

• Who is the target audience and what are scientific applications of this package?

R package developers, R + Git users who would like to run R code automatically on code update and deploy artifacts.

• Are there other R packages that accomplish the same thing? If so, how does yours differ or meet our criteria for best-in-category?

language: r support in Travis CI, the r-appveyor project for AppVeyor. This project aims at integrating these and other services using a common DSL.

Technical checks

Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• has a CRAN and OSI accepted license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions.
• contains a vignette with examples of its essential functions and uses.
• has a test suite.
• has continuous integration, including reporting of test coverage using services such as Travis CI, Coveralls and/or CodeCov.

Publication options

• Do you intend for this package to go on CRAN?
• Do you wish to automatically submit to the Journal of Open Source Software? If so:

JOSS Options

• The package has an obvious research application according to JOSS's definition.
  • The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
  • The package is deposited in a long-term repository with the DOI:
  • (Do not submit your package separately to JOSS)

• Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution? If so:

MEE Options

• The package is novel and will be of interest to the broad readership of the journal.
• The manuscript describing the package is no longer than 3000 words.
• You intend to archive the code for the package in a long-term repository which meets the requirements of the journal (see MEE's Policy on Publishing Code)
• (Scope: Do consider MEE's Aims and Scope for your manuscript. We make no guarantee that your manuscript will be within MEE scope.)
• (Although not required, we strongly recommend having a full manuscript prepared when you submit here.)
• (Please do not submit your package separately to Methods in Ecology and Evolution)

Code of conduct

• I agree to abide by rOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

---

CC @pat-s @maelle @karthik

[annakrystalli]

Hello @krlmlr! 👋

I'll be your editor on this submission. Will be back in touch when I've completed the editor checks. 👍

[annakrystalli]

Editor checks:

• Fit: The package meets criteria for fit and overlap
• Automated tests: Package has a testing suite and is tested via Travis-CI or another CI service.
• License: The package has a CRAN or OSI accepted license
• Repository: The repository link resolves correctly
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

Editor comments

Hello again @krlmlr,

I've completed the editors checks, see below for issues raised. Feel free to address/respond to any issues raised.

Can I also please ask that you add the rOpenSci review badge to the README?

[![](https://badges.ropensci.org/305_status.svg)](https://github.com/ropensci/software-review/issues/305)

In the meantime, I'll start looking for reviewers!

---

checks

The checks throw up an example which isn't working:

#> ── R CMD check results ─────────────────────────────── tic 0.2.13.9016 ────
#> Duration: 54.8s
#> 
#> ❯ checking examples ... ERROR
#>   Running examples in ‘tic-Ex.R’ failed
#>   The error most likely occurred in:
#>   
#>   > base::assign(".ptime", proc.time(), pos = "CheckExEnv")
#>   > ### Name: step_do_push_deploy
#>   > ### Title: Step: Perform push deploy
#>   > ### Aliases: step_do_push_deploy
#>   > 
#>   > ### ** Examples
#>   > 
#>   > dsl_init()
#>   �[32m✔�[39m Creating a blank tic stage configuration
#>   > 
#>   > get_stage("deploy") %>%
#>   +   add_step(step_do_push_deploy(path = "docs"))
#>   Error: Error evaluating the step argument of add_step(), expected an object of class TicStep.
#>   Original error: The working directory is not in a git repository
#>   Execution halted
#> 
#> 1 error ✖ | 0 warnings ✔ | 0 notes ✔
#> Error: R CMD check found ERRORs

^{Created on 2019-05-29 by the reprex package (v0.3.0)}

tests

devtools:test() is throwing up some warnings for me (well the same warning in a couple of locations):

pkg_dir <- "/Users/Anna/Documents/workflows/rOpenSci/editorials/tic"
devtools::test(pkg_dir)
⠙ |   1   1   | test-integration-git.R
✔ |   1   1   | test-integration-git.R [0.8 s]
#> ───────────────────────────────────────────────────────────────────────────────────
#> test-integration-git.R:19: warning: integration test: git
#> `recursive` is deprecated, please use `recurse` instead
#> ───────────────────────────────────────────────────────────────────────────────────
#> 
⠏ |   0       | test-integration-package.R┌───────────────────────────────────────┐
#> │                                       │
#> │   integration test: package failure   │
#> │                                       │
#> └───────────────────────────────────────┘
#> 
#> Error : A step failed in stage "script": script.
#> 
⠹ |   1   2   | test-integration-package.R
✔ |   1   2   | test-integration-package.R [6.2 s]
#> ───────────────────────────────────────────────────────────────────────────────────
#> test-integration-package-failure.R:9: warning: integration test: package failure
#> `recursive` is deprecated, please use `recurse` instead
#> 
#> test-integration-package-failure.R:9: warning: integration test: package failure
#> `recursive` is deprecated, please use `recurse` instead
#> ───────────────────────────────────────────────────────────────────────────────────
#> 
⠏ |   0       | test-integration-package.R┌───────────────────────────────┐
#> │                               │
#> │   integration test: package   │
#> │                               │
#> └───────────────────────────────┘
⠹ |   1   2   | test-integration-package.R
✔ |   1   2   | test-integration-package.R [5.6 s]
#> ───────────────────────────────────────────────────────────────────────────────────
#> test-integration-package.R:9: warning: integration test: package
#> `recursive` is deprecated, please use `recurse` instead
#> 
#> test-integration-package.R:9: warning: integration test: package
#> `recursive` is deprecated, please use `recurse` instead
#> ───────────────────────────────────────────────────────────────────────────────────
#> 
⠏ |   0       | test-repo
⠋ |   1       | test-repo
⠙ |   2       | test-repo
⠹ |   3       | test-repo
⠸ |   4       | test-repo
✔ |   4       | test-repo [3.9 s]
#> 
⠏ |   0       | test-stages.R
✔ |   2       | test-stages.R
#> 
⠏ |   0       | tasks-base
✔ |   6       | tasks-base
#> 
⠏ |   0       | test-utils.R
✔ |   4       | test-utils.R
#> 
#> ══ Results ════════════════════════════════════════════════════════════════════════
#> Duration: 24.3 s
#> 
#> OK:       64
#> Failed:   0
#> Warnings: 5
#> Skipped:  0

^{Created on 2019-05-29 by the reprex package (v0.3.0)}

goodpractice

Output from goodpractice::gp(pkg_dir)

It is good practice to

  ✖ write unit tests for all functions, and all package code in general. 51% of code lines are covered by test
    cases.

    R/base64.R:15:NA
    R/base64.R:16:NA
    R/base64.R:17:NA
    R/base64.R:18:NA
    R/base64.R:31:NA
    ... and 368 more lines

  ✖ avoid long code lines, it is bad for readability. Also, many people prefer editor windows that are about 80
    characters wide. Try make your lines shorter than 80 characters

    R/appveyor.R:24:1
    R/ci.R:45:1
    R/dsl-storage.R:81:1
    R/dsl-storage.R:85:1
    R/dsl-storage.R:110:1
    ... and 88 more lines

  ✖ not import packages as a whole, as this can cause name clashes between the imported packages. Instead, import
    only the specific functions you need.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 

spelling

A few candidates for genuine typos identified through devtools::spell_check()

DESCRIPTION does not contain 'Language' field. Defaulting to 'en-US'.
  WORD              FOUND IN
beim              tic.Rmd:221
builded           NEWS.md:195
commited          deployment.Rmd:166
constitutent      dsl.Rd:37
interfer          tic.Rmd:129
undefinied        NEWS.md:227
ususally          build-lifecycle.Rmd:20

Finally, picking up from the coverage issue picked up by gp(), coverage() breaks down the problem areas in a bit more detail:

tic Coverage: 51.93%
R/base64.R: 0.00%
R/keys.R: 0.00%
R/macro-bookdown.R: 0.00%
R/macro-pkgdown.R: 0.00%
R/mock.R: 0.00%
R/print.R: 0.00%
R/steps-bookdown.R: 0.00%
R/steps-drat.R: 0.00%
R/steps-pkgdown.R: 0.00%
R/steps-ssh.R: 0.00%
R/steps-write-text-file.R: 0.00%
R/use_tic.R: 0.00%
R/ci.R: 36.84%
R/steps-install.R: 42.31%
R/local.R: 46.15%
R/steps-code.R: 46.15%
R/utils.R: 48.00%
R/macro-package-checks.R: 60.71%
R/git2r.R: 66.67%
R/dsl-storage.R: 72.41%
R/steps-rcmdcheck.R: 73.68%
R/stage.R: 85.92%
R/dsl.R: 87.18%
R/run.R: 88.46%
R/steps-git.R: 90.79%
R/repo.R: 90.91%
R/steps-base.R: 100.00%

In general, we require atleast 75% test coverage. Is there a reason coverage is at ~50% and that a number of R/ scripts aren't covered at all?

Comments for reviewers

• There is a NOTE that is output from an R CMD check which is run as part of test-integration-package.R on a fake package.

  #> �[36m── R CMD check results ─────────────────── ticpkg457a3a5e2a7apkg 0.0.0.9000 ────�[39m
  #> Duration: 3s
  #> 
  #> �[34m❯ checking top-level files ... NOTE�[39m
  #>   Non-standard file/directory found at top level:
  #>     ‘tic.R’
  #> 

  You can safely ignore it. It's just the result of the tic file in the fake package not having been added to the .Rbuildignore file. I'm just flagging it because I got confused as there's a lot of output from the tests. Thanks @maelle for help trouble shooting!

• Also can I draw your attention to the actual test at the end of test-integration-package.R which is a simple expect_true(TRUE) and invite you to consider whether a more specific test could be more appropriate?

[annakrystalli]

We know have reviewers! 🎉 Many thanks @ldecicco-USGS & @maxheld83 for agreeing to review.

Any questions, please feel free to reach out.

---

Reviewers: @ldecicco-USGS, @maxheld83
Due date: 2019-06-23

[krlmlr]

Thanks. I've addressed the editorial comments in v0.2.13.9017:

• Condition example on presence of Git repository.
• Fix typos.
• Update wordlist.
• Add tic.R to .Rbuildignore for internal tests to remove spurious warnings in test output.
• Add review badge.
• Reformat long source code lines.
• I don't think I can do much better than expect_true(TRUE), added a comment.
• Exclude code that can only run interactively or in a CI from coverage.
• Test utils and printing.

The comments regarding coverage helped identify a smell in the API, thanks! Working on improving it, will report here.

{rlang} and {cli} are sufficiently safe to be imported as a whole, {backports} really needs to be imported as a whole.

[krlmlr]

That API smell is removed in v0.2.13.9018. Looking forward to your reviews!

[ldecicco-USGS]

Can someone remind me where to find the blank reviewer templates?

[annakrystalli]

Hey Laura, You can find the template here: https://devguide.ropensci.org/reviewtemplate.html

…

Sent from my iPhone

On 19 Jun 2019, at 18:08, Laura DeCicco ***@***.***> wrote: Can someone remind me where to find the blank reviewer templates? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[maxheld83]

Package Review

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide

• As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

As discussed with editor @annakrystalli, I am the maintainer of the ghactions package for R workflow automation with GitHub Actions.

Documentation

The package includes all the following forms of documentation:

• A statement of need clearly stating problems the software is designed to solve and its target audience in README
• Installation instructions: for the development version of package and any non-standard dependencies in README
• Vignette(s) demonstrating major functionality that runs successfully locally
• Function Documentation: for all exported functions in R help
• Examples for all exported functions in R Help that run successfully locally
• Community guidelines including contribution guidelines in the README or CONTRIBUTING, and DESCRIPTION with URL, BugReports and Maintainer (which may be autogenerated via Authors@R).

For packages co-submitting to JOSS

• The package has an obvious research application according to JOSS's definition

The package contains a paper.md matching JOSS's requirements with:

• A short summary describing the high-level functionality of the software
• Authors: A list of authors with their affiliations
• A statement of need clearly stating problems the software is designed to solve and its target audience.
• References: with DOIs for all those that have one (e.g. papers, datasets, software).

Functionality

• Installation: Installation succeeds as documented.
• Functionality: Any functional claims of the software been confirmed.
• Performance: Any performance claims of the software been confirmed.
• Automated tests: Unit tests cover essential functions of the package
  and a reasonable range of inputs and conditions. All tests pass on the local machine.
• Packaging guidelines: The package conforms to the rOpenSci packaging guidelines

Final approval (post-review)

• The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing: 3

• Should the author(s) deem it appropriate, I agree to be acknowledged as a package reviewer ("rev" role) in the package DESCRIPTION file.

---

Review Comments

CI/CD is extremely important for open and reproducible research, and the tic package is an incredibly ambitious, generous and comprehensive initiative to make it easier for more people to adhere to these best practices.

The promotion of CI/CD for non-package projects is particular helpful for research purposes, and somewhat unaddressed so far.

The package is thoroughly documented, with great introductory vignettes for newcomers.
I checked only a relatively small subset of the feature set, but found the UI/UX pretty smooth and didn't run into any problems.

So: definetely accept this great contribution!

I do have some reservations towards the agnostic philosophy of tic, though this isn't something I would bet on, let alone to reject this.
Users should decide for themselves what suits their needs, and time will tell which approaches work, and which won't.

Perhaps my thoughts can still be valuable for the authors.

I'm a bit skeptical as to whether it's a good idea to wrap yet another layer on top of the (usually) *.ymls used to express workflows on different CI services.
These *.ymls, after all, are already abstractions over (bash?) scripts run by the services, and are meant as the UI for users.

There is an obvious appeal to abstracting over different providers, and for making this so smooth, but there are also some potential downsides:

• The complexity of managing CI/CD workflows grows considerably; now there's another abstraction layer which may have bugs or unexpected behavior.
• This will, among other things, require the authors to keep their package up to date with any upstream changes to the *.ymls by the CI/CD service providers.
  (A similar problem appears to be raised in #171)
• It's not obvious whether the abstraction (the "steps") actually do or will map elegantly to all providers now or in the future.
  For a counter-example, GitHub Actions doesn't know "steps."
  (GitHub Actions can be used for CI/CD, but it's still pretty new and weird, so this isn't a very relevant limitation right now -- and maybe it could be mapped).
• It's not obvious that the interface of tic is more obvious or user-friendly than writing a *.yml, which isn't really that hard.
  On the one hand, for easy CI/CD workflows, one can usually just copy/paste a *.yml (plus some GitHub PAT action, admittedly).
  For very custom/complicated CI/CD workflows, on the other hand, learning the R6 class system of tic may be even more complicated than adding the logic of some providers *.yml.
• If wrapping *.ymls in another abstraction / DSL, one might wonder whether R is, in fact, the right language and level at which to do this.
  Arguably, writing service-agnostic workflows could be such a big undertaking (which the tic team seems to have largely covered, incredibly!), that it might be better organised under a "bigger tent" with a larger (non-R) community, that then simply gets wrapped for R convenience.
  Absent such a bigger community, I hope that the authors will be able to cope with the maintenance load imposed by this (rapidly?) evolving/consolidating marketplace of tools.
• Alternatively, it would be interesting to see whether other communities (Python, Julia?) have also developed their own CI/CD abstraction/DSL.
• Lastly, one might wonder whether it's actually a good idea pedagogically, so to speak, to isolate R users from anything non-R (bash scripting, *.yml).
  I understand that many R users are not software developers and have limited patience and time for yet more tooling, but by wrapping ever more tools in R, we risk keeping (new) users relatively unprepared for the times when they will need to use outside (standard) tooling.

I've learned CI/CD "the hard way", with many wasted hours editing travis.yml.
Such experiences "in the old days" (well, 2 years go) can easily lead one to falsely that new users should also learn and use tooling this way (cf the base vs tidyverse controversies).
So I might very well be wrong.

But I wonder whether the tradeoff is real.
And if the authors agree (that the tradeoff exists), it might be worthwhile adding a bit of a disclaimer to the package for newcomers.
I think R users who just quickly want to set up CI/CD should understand that using tic is a choice, and this choice should better be an informed one.

[annakrystalli]

Thank you for your very thoughtful review @maxheld83 ! Stay tuned for @krlmlr's response once all reviews are in.