Concepts

Measuring

The package was created mostly for micro benchmarks and testing. One important use case was to compare different implementations of the same functionality and see which one is faster. Frequently we want to benchmark a short fast code running in a tight loop, so it was important to measure the code and not the benchmark itself.

Some benchmark suites package the code in a function, but calling a function can be more expensive than the code itself. To avoid measuring various wrappers nano-benchmark uses the following approach:

• Measured functions are located in a separate module so it can include an independent code for initialization, which shouldn't be measured.
• The measured function takes a number of iterations as a parameter. This way calling it and any additional initialization will be amortized over the number of iterations.

So the measured function looks like this:

const fn = n => {
  // some additional initialization code
  for (let i = 0; i < n; ++i) {
    // the measured fragment
  }
};

And the measured module looks like this:

// imports and an optional initialization code

export default {
  fn1: n => { /* the code similar to above */ },
  fn2: n => { /* the code similar to above */ },
  fn3: n => { /* the code similar to above */ }
};

Statistics

Nonparametric statistics

Many benchmark suites make unwarranted assumptions about the statistical properties of the data. For example, frequently it is assumed that the data is normally distributed. Usually this assumption is not true especially for short time-based data.

This package uses nonparametric statistics to process the data. For example, bootstrapping is used to estimate the confidence interval and the median.

Median vs. mean

With asymmetric distributions, which are common, the mean value and the standard deviation are not always the best way to deal with data. Many benchmark suites make this mistake.

This package uses quantiles-based approach to process the data calculating the median and the confidence interval.

Statistical significance

When getting measurements it is important to know whether the difference between measured values is statistically significant. Most benchmark suites leave it to the user to decide.

This package uses the Mann-Whitney U test to compare two samples and the Kruskal-Wallis significance test to compare multiple samples.

If the difference is statistically significant, the significance matrix is shown highlighting the difference.

Continuous measurements

Sometimes it is necessary to run a benchmark suite in a continuous manner watching memory consumption and the changes in the performance of the code. It can be very important for code used in long running servers. It is a way to detect subtle memory leaks and possible performance regressions over time, e.g., due to caching issues or the garbage collection.

This package provides the nano-watch utility to continuously run and monitor a benchmarked code. The utility uses online algorithms, which continuously update the common statistics over time.