Adding A Benchmark

ziegs edited this page Nov 7, 2012 · 5 revisions

Adding a Benchmark

This page documents how to add a benchmark on a practical level. If you're not an active developer on the benchmark, you'll probably be more interested in the Community Participation page.

  1. Design your benchmark to ensure it follows the high-level best practices.
  2. Create an html file for your benchmark in tests/
    • Follow the recommended benchmark code style.
    • Your file must include the test.js script: <script src="../test.js"></script>. This script enables the test to run stand-alone or as part of the larger harness, as well as shadowing Math.random with a deterministic alternative.
    • Your file can do whatever it likes. It will be run as an iframe in the Test Frame. It should have a setUp, tearDown, reset, and test function defined in the global scope. All functions are passed the config argument defined in runs in benchmarks.js.
      • setUp : Called to setup a run of the benchmark (a run consists of a number of calls to test).
      • tearDown : Called after a run of the benchmark has completed.
      • reset : Called after each time test is called to reset for the next test.
      • test : The actual benchmark. Called repeatedly until enough runs have completed to get a statistically significant result.
    • It's important that you call Math.random the same number of times in each invocation of your test function for the same config. The harness will run test as many times as it deems necessary to minimize error and zero in the "true" performance of your test. If your test takes different amounts of time because of different return values from Math.random, it will make it harder to zero in on the true value. The mocked Math.random's seed is reset as often as necessary to ensure that--as long as you use Math.random the same number of times for each call to test--the behavior will be deterministic.
  3. Add an entry to benchmarks.js for your benchmark.
  4. Send a pull request to add in your benchmark that clearly defines which performance issue it targets. Remember that only issues with the minimal number of votes are eligible for inclusion.