Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance benchmarking improvements #504

Open
ajfriend opened this issue Aug 13, 2021 · 1 comment
Open

Performance benchmarking improvements #504

ajfriend opened this issue Aug 13, 2021 · 1 comment

Comments

@ajfriend
Copy link
Contributor

As discussed offline, we think there are a few improvements we could make in how we do performance benchmarks for H3:

  • use some standard benchmarking framework instead of rolling our own (which is what we currently do)
    • a framework could provide nice features like warm-ups, advanced stats (min vs avg), etc.
    • what options currently exist for C?
  • standardize the benchmark format to make it more machine-readable (like in Go), which would make it easier to automatically compare timings between diffs
  • avoid doing benchmarks on laptops and/or finding a dedicated benchmarking machine
  • random (but deterministic) inputs for improved input coverage
  • benchmark performance tracking/plotting over the timeline of diffs, like what is done in Python using asv for libraries like numpy

For context, this issue came up after seeing a lot of variance in the benchmarks I was running on my laptop for #496

@ajfriend
Copy link
Contributor Author

Some overlap with #530

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant