bench.js: easy/fast way to answer the question 'did I regress?' #3

Open
nornagon opened this Issue Feb 25, 2012 · 1 comment

Projects

None yet

1 participant

Collaborator

Currently the benchmarks take upwards of 15 minutes to run. It's really hard to determine, for a given change, whether a performance regression was introduced.

Possible solutions:

  1. only run each benchmark 3 times, instead of 9. This might lead to less stable numbers, but if it doesn't then it's a good solution.
  2. define a 'fast' subset of benchmarks that's reasonably representative. You've been tracking the benchmarks over time, so it shouldn't be too hard to pick the few that have reliably indicated overall performance (least-squares analysis perhaps? :))
  3. introduce a single large 'complete' benchmark. Even if it took 20s to run each time, that would only be 9*20s = 3 minutes total.

It would also be nice to be able to easily compute baseline numbers, e.g. by 'node bench.js --did-i-regress' performing a git clone into a temporary directory and running the benchmarks at origin/master, then running with your changes.

Collaborator

(also, something that might make the numbers more stable: node --expose-gc bench.js, and call gc() between each run to invoke the collector deterministically.)

Edit: for reference, I just tried this, and I was unable to produce more predictable results by calling gc() manually. (method: I measured the standard deviation across 9 runs with and without manual gc() calls between run_benches. With the calls to gc() in, the standard deviation was not noticeably different to without them.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment