Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove continuous benchmarking infrastructure #134

Merged
merged 1 commit into from May 4, 2023

Conversation

DavisVaughan
Copy link
Member

@DavisVaughan DavisVaughan commented May 3, 2023

We are planning on removing the continuous benchmarking infrastructure from bench. This decision is being made for a few reasons:

  • It is only partially finished, as COVID interrupted development and we haven't picked it back up.
  • touchstone exists, and I think the relative benchmark ideas that it uses are quite neat. Arrow also has arrowbench and their larger language agnostic conbench service, which is worth exploring.
  • I am not completely convinced that git notes are the best long term storage for results. It doesn't seem widely supported by git (they no longer even show notes in the commit message by default), and they are kind of a pain to work with if you have to make any manual adjustments to the notes since they are technically orphaned branches. It is also hard to actually see the files on github. You can locally checkout refs/notes/benchmarks and that shows them to you.
  • A big reason is because none of us are actively using this infrastructure yet, with the exception of tibble.
  • Because we aren't convinced that these functions are stable, we haven't been shipping them in CRAN releases, which makes doing a bench release a little complicated (you have to remove, send to CRAN, and then re-add the CB functions)

It has been in the dev version for awhile, so I am going to park the current implementation in a branch. tibble can use this if it needs some temporary support. That branch is frozen/continuous-benchmarking https://github.com/r-lib/bench/tree/frozen/continuous-benchmarking.

There is also some additional work in https://github.com/r-lib/bench/tree/CB where development was pushed forward a little further, but was never merged in.

It would be nice to figure out a way to pick this back up in the future, in particular for rlang, vctrs, and dplyr's vector functions like case_when() and first(). Having benchmarks in place for those lower level tools could have likely avoided some of the patches needed in dplyr 1.1.0 related to performance issues.

@DavisVaughan DavisVaughan merged commit c8c3f29 into r-lib:main May 4, 2023
12 checks passed
@DavisVaughan DavisVaughan deleted the feature/remove-cb-functions branch May 4, 2023 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant