Concepts

Measuring

This package is designed for micro-benchmarks — comparing implementations of the same functionality to find which is faster. The goal is to measure the code, not the benchmark overhead.

Some benchmark suites wrap code in a function, but the call itself can cost more than the code being measured. nano-benchmark avoids this:

• Measured functions live in a separate module, which can include initialization code that should not be measured.
• Each function takes an iteration count as a parameter, so the call overhead and any setup are amortized over many iterations.

A measured function looks like this:

const fn = n => {
  // some additional initialization code
  for (let i = 0; i < n; ++i) {
    // the measured fragment
  }
};

A measured module looks like this:

// imports and an optional initialization code

export default {
  fn1: n => {
    /* the code similar to above */
  },
  fn2: n => {
    /* the code similar to above */
  },
  fn3: n => {
    /* the code similar to above */
  }
};

Statistics

Nonparametric statistics

Many benchmark suites assume the data is normally distributed. This assumption is usually wrong, especially for timing data.

This package uses nonparametric statistics. Bootstrapping estimates the confidence interval and median.

Median vs. mean

Timing distributions are often asymmetric, making the mean and standard deviation misleading.

This package uses a quantile-based approach, reporting median and confidence interval instead.

Statistical significance

When comparing measurements, you need to know whether the difference is statistically significant. Most benchmark suites leave this to the user.

This package uses the Mann-Whitney U test to compare two samples and the Kruskal-Wallis significance test to compare multiple samples.

If significant, a matrix shows which functions differ.

Continuous measurements

Continuous benchmarking reveals memory leaks and performance regressions in long-running servers, e.g., from caching or garbage collection.

nano-watch does this using online algorithms that update statistics incrementally without storing all measurements.