A GitHub Action for the standard continuous benchmarking

[lorenzwalthert]

I am working on a continuous benchmark with GitHub Actions for styler in r-lib/styler#674. I believe a generic GitHub action for continuous benchmarking for PRs would be very useful. It could work like this (pretty much what I implemented in r-lib/styler#674):

• run custom benchmark code in bench/ and then read all the benchmarks associated with the latest commit that match the regex ^actions_* to create boxplots from timing. Here is how one could create one that matches: bench::mark(action_with_cache = f(x)).
• fetch target branch of PR, create same boxplot.
• Put the two boxplots into one picture/file.
• replicate this behavior for a series of benchmarks (with_cache is an example).

That's how this would look:

With the actions prefix in name, we'd allow automatic benchmarking but give people the flexibility to at the same time write some custom benchmarking code and process the output independently of the GitHub Action. I believe 99% of the people would want to use continuous benchmarking would like to use such an out-of-the box solution.

I saw #87, but I felt it's slightly different so I opened a new issue. Also, I see that there is already a ggplot autoplot method but as far as I see, it does not compare two commits.

Edit: I created https://github.com/lorenzwalthert/touchstone for this.

[jimhester]

Hi @lorenzwalthert, I like what you are doing at touchstone, and unfortunately I don't currently have a lot of time to work on continuous benchmarking in bench myself.

If you were interested in contributing some or all of what you have developed in touchstone to bench I would be happy to incorporate it.

[lorenzwalthert]

Ok, I think we can close this. I implemented all functionality I wanted in {touchstone}. Standing on top of the shoulders of giant {bench} 😉.