Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a continuous benchmarking workflow #184

Closed
pratikunterwegs opened this issue Feb 21, 2024 · 3 comments · Fixed by #200 or #206
Closed

Add a continuous benchmarking workflow #184

pratikunterwegs opened this issue Feb 21, 2024 · 3 comments · Fixed by #200 or #206
Assignees
Labels
CI Related to continuous integration

Comments

@pratikunterwegs
Copy link
Collaborator

pratikunterwegs commented Feb 21, 2024

This PR requests the addition of a GHA workflow for continuous benchmarking. This is expected to help avoid changes that reduce model run speed, which is important as the expected use case is 100s-1000s of runs.

An example can be found in {EpiNow2}. @sbfnk and @jamesmbaazam any feedback on your experience with using {touchstone} would be great - would you recommend using it?

@pratikunterwegs pratikunterwegs added the CI Related to continuous integration label Feb 21, 2024
@pratikunterwegs pratikunterwegs self-assigned this Feb 21, 2024
@sbfnk
Copy link

sbfnk commented Feb 21, 2024

Yes, I'd recommend it. A problem with EpiNow2 is that the benchmarks take quite long to run (1-5 minutes) and so doing repeated testing with multiple tests inflates run times pretty quickly. Because of this we have small sample sizes for the repeated tests (n=5) and get quite lot of uncertainty in the benchmarks. I think it would work really well for less computation intensive code such as in styler.

Perhaps worth noting that the package is looking for a maintainer
lorenzwalthert/touchstone#31 (comment)
although code contributions from others are still supported.

@pratikunterwegs
Copy link
Collaborator Author

Thanks @sbfnk - good to know - I'll look into adding this to the open PR.

@pratikunterwegs
Copy link
Collaborator Author

Just looking into this and something I should have realised earlier is that it won't be useful to add this workflow in the current PR as the function names are changing as well. I could add the old names as aliases to the functions, but that would have to be reversed eventually. Possibly best to add this workflow after the current PR is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment