Concepts

Measuring

The package is designed for micro-benchmarks — comparing different implementations of the same functionality to see which one is faster. The goal is to measure the code itself, not the benchmark overhead.

Some benchmark suites wrap the code in a function, but the function call itself can be more expensive than the code being measured. To avoid this, nano-benchmark uses a different approach:

• Measured functions live in a separate module, which can include initialization code that should not be measured.
• Each function takes an iteration count as a parameter, so the call overhead and any setup are amortized over many iterations.

So the measured function looks like this:

const fn = n => {
  // some additional initialization code
  for (let i = 0; i < n; ++i) {
    // the measured fragment
  }
};

And the measured module looks like this:

// imports and an optional initialization code

export default {
  fn1: n => {
    /* the code similar to above */
  },
  fn2: n => {
    /* the code similar to above */
  },
  fn3: n => {
    /* the code similar to above */
  }
};

Statistics

Nonparametric statistics

Many benchmark suites assume the data is normally distributed. This assumption is usually wrong, especially for timing data.

This package uses nonparametric statistics to process the data. For example, bootstrapping is used to estimate the confidence interval and the median.

Median vs. mean

Timing distributions are often asymmetric, making the mean and standard deviation misleading.

This package uses a quantile-based approach, reporting the median and the confidence interval instead.

Statistical significance

When comparing measurements, it is important to know whether the observed difference is statistically significant. Most benchmark suites leave this to the user.

This package uses the Mann-Whitney U test to compare two samples and the Kruskal-Wallis significance test to compare multiple samples.

If the difference is statistically significant, a significance matrix is shown highlighting which functions differ.

Continuous measurements

Sometimes it is useful to benchmark code continuously, watching memory consumption and performance over time. This helps detect memory leaks and performance regressions in long-running servers, e.g., due to caching or garbage collection.

The nano-watch utility does exactly this. It uses online algorithms that update statistics incrementally without storing all measurements.