Concepts

Measuring

This package is designed for micro-benchmarks — comparing implementations of the same functionality to find which is faster. The goal is to measure the code, not the benchmark overhead.

Some benchmark suites wrap code in a function, but the call itself can cost more than the code being measured. nano-benchmark avoids this:

• Measured functions live in a separate module, which can include initialization code that should not be measured.
• Each function takes an iteration count as a parameter, so the call overhead and any setup are amortized over many iterations.

A measured function looks like this:

const fn = n => {
  // some additional initialization code
  for (let i = 0; i < n; ++i) {
    // the measured fragment
  }
};

A measured module looks like this:

// imports and an optional initialization code

export default {
  fn1: n => {
    /* the code similar to above */
  },
  fn2: n => {
    /* the code similar to above */
  },
  fn3: n => {
    /* the code similar to above */
  }
};

Statistics

The guiding principle: timing data is not normal, so don't treat it as if it were. Execution times have a hard lower bound (the code cannot run in negative time), a long right tail (GC pauses, JIT warmup, scheduler preemption, cache misses), and are often multimodal (a fast path and a slow path). Mean and standard deviation — which assume a symmetric, light-tailed distribution — are misleading on data like this. nano-bench therefore uses nonparametric methods throughout: it assumes nothing about the shape of the distribution.

Nonparametric statistics

Nonparametric methods make no distributional assumption. Instead of fitting a normal curve, nano-bench works from the samples directly — ranks for significance, resampling for the confidence interval, quantiles for the point estimate.

Median vs. mean

Because the distribution is asymmetric, the median describes the typical run far better than the mean: a single GC spike drags the mean but barely moves the median. nano-bench reports the median and a quantile-based confidence interval. (nano-watch still shows mean, standard deviation, skewness, and kurtosis — useful as shape diagnostics, not as the headline number.)

Confidence interval — bootstrap percentile of the median

The confidence interval comes from bootstrapping: the samples are resampled with replacement many times (--bootstrap, default 1000), a weighted quantile is taken from each resample, and the reported median and its low/high bounds are the means of those resampled quantiles. This needs no normal assumption and no closed-form standard error. The resampling RNG is seeded (--seed, or an auto-recorded seed), so the interval is reproducible — the same samples always yield the same interval.

Statistical significance

A visible difference between two medians may be real or may be noise. nano-bench tests it rather than leaving the judgement to the eye:

• Two functions — the Mann-Whitney U test (two-sided, tie-corrected): a rank-based test of whether one distribution tends to produce larger values than the other.
• Three or more functions — the Kruskal-Wallis H test as an omnibus ("do any of these differ?"); if it is significant, a Conover-Iman pairwise post-hoc identifies which pairs differ. The result is the N×N matrix.

The threshold is --alpha (default 0.05). --verbose prints the test statistic and its critical value.

Multiple comparisons

Each pairwise test has its own chance of a false positive, so testing many pairs inflates the overall ("family-wise") error rate: at α = 0.05, ten independent comparisons have a ~40% chance of at least one spurious "significant". nano-bench corrects the Kruskal-Wallis post-hoc for this with --correction:

• holm (default) — the Holm step-down procedure. It controls the family-wise error rate while being uniformly more powerful than Bonferroni (it never rejects fewer true differences), which is why it is the default.
• bonferroni — the classic single-step correction (test each pair at α/m). More conservative; kept because it is the most recognizable name to cite.
• none — no correction.

The correction adjusts only the significance threshold, never the confidence interval's α (a CI is one interval, not a family). The two-function Mann-Whitney case is a single comparison and needs no correction.

Plot the shape first

A median and a confidence interval cannot reveal multimodality, skew, or the size of the tail — and those are exactly what you want to see when tuning hot code. --histogram draws each function's sample distribution inline, on a shared scale so the shapes are directly comparable, and flags when the mean lands where few samples did (a sign the distribution may be multimodal). Look at the shape before trusting a single number.

Comparing across runs

Significance is a pure function of the samples, so a saved run can be re-tested later without re-measuring. nano-bench-compare recomputes the same tests from the stored samples to answer "did this get faster between versions?" — with one caveat it surfaces loudly: a difference measured on a different CPU, runtime, or OS may be the environment, not the code, so it warns whenever the compared runs' environments disagree.

Other tests

A Kolmogorov-Smirnov two-sample test is available in the library API (src/significance/kstest.js) for comparing whole distributions, but is not wired into the CLI.

Continuous measurements

Continuous benchmarking reveals memory leaks and performance regressions in long-running servers, e.g., from caching or garbage collection.

nano-watch does this using online algorithms that update statistics incrementally without storing all measurements.