Submission: circle

[pat-s]

Submitting Author: Patrick Schratz (@pat-s)
Repository: https://github.com/ropenscilabs/circle
Version submitted: v0.5.0.9001
Editor: @noamross
Reviewers: @sharlagelfand, @mbjoseph

Due date for @sharlagelfand: 2020-12-07

Due date for @mbjoseph: 2020-12-07
Archive: TBD
Version accepted: TBD

---

• Paste the full DESCRIPTION file inside a code block below:

Package: circle
Title: R Client Package for Circle CI
Version: 0.7.0
Authors@R:
    person(given = "Patrick",
           family = "Schratz",
           role = c("aut", "cre"),
           email = "patrick.schratz@gmail.com",
           comment = c(ORCID = "0000-0003-0748-6624"))
Description: Tools for interacting with the 'Circle CI' API.
    Besides executing common tasks such as querying build logs and
    restarting builds, this package also helps setting up permissions to
    deploy from builds.
License: GPL-3
URL: https://docs.ropensci.org/circle/,
    https://github.com/ropenscilabs/circle
BugReports: https://github.com/ropenscilabs/circle/issues
Imports:
    cli (>= 2.0.0),
    httr,
    jsonlite
Suggests:
    clipr,
    covr,
    gert,
    gh,
    knitr,
    openssl,
    purrr,
    rmarkdown,
    testthat (>= 3.0.0),
    usethis (>= 2.0.0),
    utils,
    vcr,
    withr
VignetteBuilder:
    knitr
Config/testthat/edition: 3
Config/testthat/parallel: false
Encoding: UTF-8
LazyData: true
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.1.1

Scope

• Please indicate which category or categories from our package fit policies this package falls under: (Please check an appropriate box below. If you are unsure, we suggest you make a pre-submission inquiry.):

  • data retrieval
  • data extraction
  • database access
  • data munging
  • data deposition
  • workflow automation
  • version control
  • citation management and bibliometrics
  • scientific software wrappers
  • database software bindings
  • geospatial data
  • text analysis

• Explain how and why the package falls under these categories (briefly, 1-2 sentences):

It helps to automate CI workflows and goes in line with the {tic} and {travis} package.

• Who is the target audience and what are scientific applications of this package?

Users who prefer to interact with the command line to query information about CI build statuses.
No scientific applications.

• Are there other R packages that accomplish the same thing? If so, how does yours differ or meet our criteria for best-in-category?

Besides a discontinued package which is 4 years old, there is no similar package.

• If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted.

Technical checks

Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• has a CRAN and OSI accepted license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions.
• contains a vignette with examples of its essential functions and uses.
• has a test suite.
• has continuous integration, including reporting of test coverage using services such as Travis CI, Coveralls and/or CodeCov.

Publication options

• Do you intend for this package to go on CRAN?
• Do you wish to automatically submit to the Journal of Open Source Software? If so:

JOSS Options

• The package has an obvious research application according to JOSS's definition.
  • The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
  • The package is deposited in a long-term repository with the DOI:
  • (Do not submit your package separately to JOSS)

• Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution? If so:

MEE Options

• The package is novel and will be of interest to the broad readership of the journal.
• The manuscript describing the package is no longer than 3000 words.
• You intend to archive the code for the package in a long-term repository which meets the requirements of the journal (see MEE's Policy on Publishing Code)
• (Scope: Do consider MEE's Aims and Scope for your manuscript. We make no guarantee that your manuscript will be within MEE scope.)
• (Although not required, we strongly recommend having a full manuscript prepared when you submit here.)
• (Please do not submit your package separately to Methods in Ecology and Evolution)

Code of conduct

• I agree to abide by rOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

[maelle]

Many thanks @pat-s for your submission! It is in scope indeed. However we think it'd make more sense to finalize #305 first as you're quite involved in that one as well.

[pat-s]

Thanks Maelle. {circle} and {travis} are both deps of {tic} to some degree but also kind of independent.

Let's finish {tic} first then to not confuse things.

[pat-s]

{tic} is done - ready when you are :)

[noamross]

Editor checks:

• Fit: The package meets criteria for fit and overlap
• Automated tests: Package has a testing suite and is tested via Travis-CI or another CI service.
• License: The package has a CRAN or OSI accepted license
• Repository: The repository link resolves correctly
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

Editor comments

Thanks for your submission, @pat-s! My initial attempt to test this package failed (see below). Based on Failed to creating checkout keys for repo pat-s/travis-testthat, and I still got failures when I forked the repository to my own GitHub account and set that as the only remote (I have a CircleCI account). With packages like this it's common for some permission issues to arise in this way. Before I assign to reviewers could you please document the developer workflow so reviewers know what remotes, accounts, keys etc., they need set up in order to test the package locally? This is best placed in CONTRIBUTING.md and linked from the README.

Also, to preempt another issue, your README is somewhat sparse and both it and the vignette lack an introduction to what the package is for and, importantly in this case, what it is not for. Remember our principle of multiple points of entry*: any piece of documentation may be the first encounter the user has with the package and/or the tool/data it wraps. It will be helpful to start these out with an introduction that explains that the package is for managing a CI service and its relationship to tic and other tools.

Loading circle
Testing circle
✓ |  OK F W S | Context
✓ |   1       | authentication [1.0 s]
x |   2 1     | builds [10.0 s]
──────────────────────────────────────────────────────────────────────────────
test-builds.R:24: error: restarting a workflow works
Forbidden (HTTP 403). Failed to restarting workflow 'c006f0a5-5c05-4306-b66f-dc04f141e2ae' on Circle CI.
Backtrace:
 1. testthat::expect_s3_class(retry_workflow(workflow_id), "circle_api") tests/testthat/test-builds.R:24:2
 4. circle::retry_workflow(workflow_id)
 5. httr::stop_for_status(...) R/builds.R:145:2
──────────────────────────────────────────────────────────────────────────────
x |   0 2     | checkout-key [0.2 s]
──────────────────────────────────────────────────────────────────────────────
test-checkout-keys.R:8: error: setting checkout keys works
Forbidden (HTTP 403). Failed to creating checkout keys for repo pat-s/travis-testthat on Circle CI.
Backtrace:
 1. testthat::expect_s3_class(...) tests/testthat/test-checkout-keys.R:8:2
 4. circle::create_checkout_key(repo = repo, user = user, type = "user-key")
 5. httr::stop_for_status(...) R/checkout-key.R:38:2

test-checkout-keys.R:18: error: deleting checkout keys works
Forbidden (HTTP 403). Failed to get checkout keys for repo pat-s/travis-testthat on Circle CI.
Backtrace:
 1. circle::get_checkout_keys(repo = repo, user = user) tests/testthat/test-checkout-keys.R:18:2
 2. httr::stop_for_status(...) R/checkout-key.R:67:2
──────────────────────────────────────────────────────────────────────────────
x |   0 3     | env-vars [0.2 s]
──────────────────────────────────────────────────────────────────────────────
test-env-vars.R:6: error: setting env vars works
Forbidden (HTTP 403). Failed to set env vars for repo 'pat-s/travis-testthat' on Circle CI.
Backtrace:
  1. testthat::expect_silent(...) tests/testthat/test-env-vars.R:6:2
  9. circle::set_env_var(...)
 10. httr::stop_for_status(...) R/env-var.R:96:2

test-env-vars.R:16: error: getting env vars works
Forbidden (HTTP 403). Failed to get env vars for repo pat-s/travis-testthat on Circle CI..
Backtrace:
  1. testthat::expect_silent(get_env_vars(repo = repo, user = user)) tests/testthat/test-env-vars.R:16:2
  9. circle::get_env_vars(repo = repo, user = user)
 10. httr::stop_for_status(...) R/env-var.R:51:2

test-env-vars.R:24: error: deleting env vars works
Forbidden (HTTP 403). Failed to delete env vars for repo 'pat-s/travis-testthat' on Circle CI.
Backtrace:
  1. testthat::expect_silent(...) tests/testthat/test-env-vars.R:24:2
  9. circle::delete_env_var(...)
 10. httr::stop_for_status(...) R/env-var.R:125:2
──────────────────────────────────────────────────────────────────────────────
x |   2 2     | general [0.7 s]
──────────────────────────────────────────────────────────────────────────────
test-general.R:21: error: triggering a new build works
Forbidden (HTTP 403). Failed to start a new build for repo pat-s/travis-testthat on Circle CI.
Backtrace:
 1. testthat::expect_s3_class(...) tests/testthat/test-general.R:21:6
 4. circle::new_build(repo = repo, user = user)
 5. httr::stop_for_status(...) R/general.R:114:2

test-general.R:31: error: checking the existence of checkout keys works
Forbidden (HTTP 403). Failed to get checkout keys for repo pat-s/travis-testthat on Circle CI.
Backtrace:
 1. testthat::expect_true(has_checkout_key()) tests/testthat/test-general.R:31:6
 4. circle::has_checkout_key()
 5. circle::get_checkout_keys(repo = repo, user = user, vcs_type = vcs_type) R/checkout-key.R:138:2
 6. httr::stop_for_status(...) R/checkout-key.R:67:2
──────────────────────────────────────────────────────────────────────────────
✓ |   4       | print S3 methods [8.7 s]

══ Results ═══════════════════════════════════════════════════════════════════
Duration: 21.0 s

OK:       9
Failed:   8
Warnings: 0
Skipped:  0

---

Reviewers:
Due date:

[pat-s]

Hi Noam,

thanks for taking this on!

My initial attempt to test this package failed

Tests are running on a test repo which is cloned during testing.
Since the repo lives in my account, other users won't have permissions to start/cancel builds.
We will also have to skip all tests on CRAN since there is no way I can set the API keys there.
The API keys are stored on the CI providers securely, so tests will run there for everyone.
We could potentially move the test repo to ropenscilabs though this would also just give a handful people access to it if they have local keys set up.
I'd say having the tests running on the CI for everyone is the best solution here.
Let me know if I overlook something.

Also for some tests I need a GITHUB_TOKEN with extended permissions (to delete a private GH user key of mine automatically which will be created by use_circle_deploy()).

I've added this piece of information to CONTRIBUTING.md as suggested and put a link into the README.

Also, to preempt another issue, your README is somewhat sparse and both it and the vignette lack an introduction to what the package is for and, importantly in this case, what it is not for.

I've added more information to the README.

In addition, I also optimized the Getting Started vignette. (You might want to wait a bit until the vignette is deployed because it seems there are some SSL certificate issues with pkgdown today).

Let me know if there are still points to address before the review can start.

[noamross]

Hi @pat-s, I don't think this is a great solution, because this means that a contributor or reviewer can't test the package, and if they do only pushing and having the PR run for you then you need to expose your extended-permission GITHUB_TOKEN (very insecure!) to all. I get that you will need to skip these on CRAN, but I'd much prefer that there be a full developer workflow. This should hopefully save you effort in the long run, as contributors will be able to test prior to submitting PRs. It's fine if this takes some setting up by the contributor.

A solution could be that the user provides an env var R_CIRCLE_TEST_REPO (forked from your repo if its contents are important), as well as an extended permissions github token.

[pat-s]

if they do only pushing and having the PR run for you then you need to expose your extended-permission GITHUB_TOKEN (very insecure!) to all

I don't see why anything should be exposed. The token is stored as a secure env var in the build and nothing will be exposed when a PR is run.

This should hopefully save you effort in the long run, as contributors will be able to test prior to submitting PRs.

I see the advantage of being able to run the tests locally.
However, maybe your are overlooking the following: Even if an API key is set, the reviewers would need my API key to make API calls to pat-s/travis-testthat to start/cancel build.

Even when setting their personal API key, reviewers would need to

• adjust the test suite manually so tests run on a repo of their own (possibly create that first)
• repo needs to have XY builds because tests are not just running on the most recent build/job
• repo needs to have a working Circle CI workflow

Because testing an API client package is not trivial, I created the dummy repo to have a dedicated repo for testing API functionality.
Sometimes things go wrong and nobody wants to have such things happening on a production repo.

If I would be a reviewer of such a package, I would certainly dislike having to do all of this effort just to run tests locally whereas they just run fine on the CI.

I'd argue that a API client package is special when it comes to testing, especially when it comes to CI client packages for which it is not sufficient to just replace the API key.

From my perspective, it is important that the functions of this package work if users apply them to a repo of themselves (after having set the API key) rather than requiring them to adjust the whole test suite.

I am not sure if there was a previous case like this one - maybe a general decision how such packages are handled by {ropensci} is needed here.
The next one ({travis}) is already waiting.

Happy to make adjustments if I get a pointer how to do so.
I just do not see one right now.

[noamross]

A very large number of our packages are actually API clients; circle's case is only a little different from several others. At a minimum we need a functioning test suite so that tests can pass locally and then later with rOpenSci's automatic checks. But there are a number of ways to accomplish this.

• Split tests into those that run with and without certain tokens available, and selectively run tests based on availability. Fewer tests may run locally but testthat::skip* functions make it transparent what is skipped. You'll see this pattern in arkdb. In this case it's an especially good idea to use mocking with the webmockr or httptest packages so that you can separate out tests on HTTP content from the actual response:
• Provide enough direction in CONTRIBUTING for someone to run the full test suite. e.g., get a token, fork/clone a repository, etc. This is the type of effort we expect from reviewers of packages, though a smaller number of contributors will likely go through it. Here's an example of more complex contributing directions written in response to a review. A few things would make this easier, for instance, using an environment variable (CIRCLE_TEST_REPO=http://...) to define the the URI of the test repo, avoiding the need to adjust the test suite.
• Minimize the testing effort by including set up / tear down the example repository as part of the test suite, thus only requiring the tester to provide keys with adequate scopes. I asked Jenny Bryan, who maintains a lot of packages with complex API interactions and auth setups (e.g. googledrive), about this case, and she strongly recommended a setup/teardown approach for long-term maintainability by both yourself and contributors. I won't require this full level of automation but it's an option in your toolbox.

In any case I'll leave it to reviewers to look at the completeness of the test suite.

Regarding this:

I don't see why anything should be exposed. The token is stored as a secure env var in the build and nothing will be exposed when a PR is run.

If you allow PRs to use your variables, all one needs to do is slip a Sys.getenv("GITHUB_TOKEN") and then a line of code like a curl to ship that value elsewhere in the PR. This is why CI systems pretty much all prevent such variables from being exposed to PR builds by default and it's highly recommended to leave it that way.

[pat-s]

Thank Noam, lots of good resources and pointers in there.
It might take me a while to come up with a test suite that fulfills all these requirements.

for long-term maintainability by both yourself and contributors.

I see the point even though it means a lot of work. Knowing this before might have scared me away even but now I'm in it ;)

In any case I'll leave it to reviewers to look at the completeness of the test suite.

Happy to wait for reviews until I implemented a more proper testing suite.

[noamross]

To clarify, @pat-s, we do need a test suite that passes locally in order to assign to reviewers, though they may have feedback on completeness or setup.