Github action for atime
atime is an R package that provides functions for asymptotic complexity testing, which means running several pieces of R code for various input data sizes, and recording the time and memory requirements for each piece/size. Different pieces can be code in different packages, or different versions of the same code from one package.
-
atime_versions()can run the same piece of code on different versions of an R package in a github repo. -
atime_pkg()runs user code in pkg/inst/atime/tests.R on different versions of the R package (CRAN, branch, main/master).
Related work in R is limited to non-asymptotic testing
- Rperform and touchstone in R, and asv in python support some similar features, but more focus on metrics for a single data size (not asymptotic/variable data size).
- microbenchmark/bench packages are useful for measuring and comparing time/memory usage of R expressions (and bench is used by atime for that purpose).
The goal of this project would be to write a new GitHub action that can create/update a PR comment showing if there are any changes in performance (time/memory) between a PR and the main/release versions.
The idea is that the package author writes some code like this in their pkg/inst/atime/tests.R file,
- https://github.com/tdhock/aum/blob/73235b9dcf447361389f51d03bf1af7bbde06ca2/inst/atime/tests.R
- https://github.com/tdhock/binsegRcpp/blob/5e5e1349304238b6d20b55dff236989caafec239/inst/atime/tests.R
Then after every push to a github PR, the github action will run the
atime_pkg() function, which computes the benchmark and makes the
plots to inspect for any changes in performance. New code needs to be
written to create/update a PR comment based on these results.
- Creating example PRs where changes are detected.
- Writing blog/tutorial to explain how to setup the new GH action on a new repo.
This project will make atime much more easy to use, and therefore make it much easier for R package developers to see if their PRs introduce performance regressions/improvements.
Contributors, please contact mentors below after completing at least one of the tests below.
- EVALUATING MENTOR: Toby Hocking toby.hocking@r-project.org is the author of numerous R packages, and was the PI on the R consortium RcppDeepState grant.
- Fabrizio Sandri sandri.fabr@gmail.com is an expert in GitHub actions, and has previous experience developing RcppDeepState-action in R-GSOC'22.
Contributors, please do one or more of the following tests before contacting the mentors above. In addition to doing the coding, make sure to describe the solution to each test in English, along with hyperlinks to the relevant results/actions/code files.
- Easy: setup a GitHub repo with an R package, and use r-lib/actions to automatically check the package after every commit. Modify your repo's github action so it does something different / non-standard. (maybe print out something that is not usually printed?)
- Medium: create a new github repo for your Easy action, maybe call it custom-check, and use that new github action on your R package github repo.
- Hard: pick an R package which is on github, and use
atime_versions()to make a plot of the performance of several versions of the code.
Contributors, please post a link to your test results here.
- EXAMPLE CONTRIBUTOR 1 NAME, LINK TO GITHUB PROFILE, LINK TO TEST RESULTS.
- Siyao (Jane) He, SiyaoIsHiding, The repo of the GitHub Action, The repo using the GitHub Action