Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarks #38

Closed
NickCondron opened this issue Nov 30, 2022 · 2 comments
Closed

Add benchmarks #38

NickCondron opened this issue Nov 30, 2022 · 2 comments

Comments

@NickCondron
Copy link
Contributor

I've been looking into adding benchmarks. It would also help clarify if certain changes have a justifiable performance impact (#24, #25). Right now, I'm looking at Criterion (https://github.com/bheisler/cargo-criterion). We may also want to use Iai (https://github.com/bheisler/iai) for CI benchmarks.

If anyone has experience with either of these tools (or something else) I would appreciate some guidance.

Also, we need some set of replay files that cover different performance scenarios. Different matchups, Ice Climbers games, long games, short games, netplay games, local games, non-standard games (ie. not 4-stock 8 minutes), etc. I wonder if it would make sense to store the replay set compressed w/ peppi and inflate each time we run the benchmarks.

@hohav
Copy link
Owner

hohav commented Dec 13, 2022

This is a good idea, but I lack experience with Rust benchmarking so I can't offer much advice there.

The data set is another interesting question. A wide variety of games is definitely a good idea, but do we need to go "deep" too for robustness? I.e. do we need a largeish set of replays, or is a small number of handpicked samples sufficient as long as they're diverse enough?

@hohav
Copy link
Owner

hohav commented Dec 27, 2023

Implemented via #43 (b62b811 and 0c671c8).

@hohav hohav closed this as completed Dec 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants