Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advanced benchmark interference detection #481

Open
sharkdp opened this issue Feb 19, 2022 · 0 comments
Open

Advanced benchmark interference detection #481

sharkdp opened this issue Feb 19, 2022 · 0 comments

Comments

@sharkdp
Copy link
Owner

sharkdp commented Feb 19, 2022

We already have

  • an outlier detection which is useful if a background process has interferred with a single (or a few) benchmark runs
  • a "first run was significantly longer than the rest" detection which can detect caching effects

But there are other things that can go wrong during benchmarking. For example:

  1. a sudden jump in runtime which would result in a bimodal (or multi-modal) runtime distribution. This could be caused by a long-running background process that spun up
  2. a continuous trend in the runtimes. Like a slowly growing/decreasing runtime which might be caused by things like thermal throttling.

It would be nice to add more heuristics to hyperfine to detect these scenarios as well. The second case could maybe be handled by fitting a linear model to the runtimes and to check whether its slope is close (enough) to zero.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant