Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to lorenzwalthert/touchstone #698

merged 26 commits into from Nov 26, 2020


Copy link

@lorenzwalthert lorenzwalthert commented Nov 24, 2020

This PR replaces the continuous benchmarking infrastructure provided by r-lib/bench with the touchstone package. Key insights were:

  • Does not make sense to only benchmark the feature branch, but benchmark the base branch as well and look at relative difference because two CI/CD runs are not exactly comparable in terms of available compute power. This matters in particular when running one iteration of a benchmark takes long (>> a few seconds) and speed implications are small.
  • Maintaining a timeline such as the implementation of r-lib/bench is of limited use because of the first bullet. Speed implications are to be checked between two revisions.
  • R and package versions must be fixed via RSPM to allow as much continuation as possible anyways. Changing the timestamp of RSPM can happen in PRs that are only dedicated to dependency updates.


First results with 20 iterations are promising and take about 30 minutes. They show 0% difference in median for this PR, which was expected as no code that is benchmarked changes. Second run: 0.9%, third run 3%, fourth -0.1%. Results can be found under the GitHub Action -> Artifacts.

Here is the comparison of 20 iterations with 20 times style_pkg() for the here package for master and touchstone branch.


Copy link

codecov-io commented Nov 24, 2020

Codecov Report

Merging #698 (933e176) into master (a2a2e2e) will decrease coverage by 0.00%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #698      +/-   ##
- Coverage   90.42%   90.41%   -0.01%     
  Files          47       47              
  Lines        2225     2223       -2     
- Hits         2012     2010       -2     
  Misses        213      213              
Impacted Files Coverage Δ
R/set-assert-args.R 92.30% <0.00%> (-0.38%) ⬇️
R/rules-line-break.R 100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d6131b...933e176. Read the comment docs.

@lorenzwalthert lorenzwalthert changed the title Add touchstone artifacts Switch to lorenzwalthert/touchstone Nov 25, 2020
@lorenzwalthert lorenzwalthert merged commit f81ff7a into r-lib:master Nov 26, 2020
@lorenzwalthert lorenzwalthert deleted the touchstone branch November 26, 2020 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

2 participants