nano‐bench‐compare

nano-bench-compare views and compares saved nano-bench results JSON files. It recomputes significance and confidence intervals from the raw samples stored in each file — it never runs a benchmark. Use it to re-render a saved run, or to compare a baseline against a new run (before/after) without re-measuring.

Usage

$ npx nano-bench-compare --help
Usage: nano-bench-compare [options] <files...>

View and compare nano-bench results JSON files.

Arguments:
  files                  one or more results JSON files

Options:
  -V, --version          output the version number
  -a, --alpha <alpha>    significance level for the recompute (default: the
                         first file’s α)
  --correction <method>  post-hoc correction (default: the first file’s method)
                         (choices: "none", "holm", "bonferroni")
  -v, --verbose          show significance test statistics and critical values
  --pooled               compare all series as one k-sample omnibus instead of
                         pairing by name
  --clusters             split multimodal distributions into clusters
                         (dip-test gated)
  --no-emoji             use ASCII fastest/slowest markers (F/S) instead of
                         emoji
  --histogram            show a distribution histogram per series
  --chart <type>         histogram orientation (choices: "columns", "bars",
                         default: "columns")
  --bins <bins>          histogram bin count (default: auto)
  -h, --help             display help for command

• --alpha — significance level for the recompute. Default: the first file's recorded α.
• --correction — post-hoc correction for the recompute (none / holm / bonferroni). Default: the first file's recorded method. See nano-bench → Significance and correction.
• --pooled — compare all series as one omnibus instead of pairing by name (see below).
• --clusters — split multimodal series at KDE density minima and report each cluster apart; a dip-test note flags multimodal series even without the flag (see nano-bench-io → Clusters).
• --verbose (-v) — show the test statistic, critical value, and per-comparison α.
• --histogram / --chart / --bins — distribution charts, one per series on a shared axis (see nano-bench → Distribution histograms).
• --no-emoji — ASCII fastest/slowest markers (F/S).

Producing files to compare

Save runs with nano-bench --json:

$ npx nano-bench bench.js --json before.json --label before
# ...change the code...
$ npx nano-bench bench.js --json after.json --label after

Each file stores the raw per-sample arrays (the source of truth), the derived summaries, the run parameters (including the bootstrap seed), and a snapshot of the environment. The --label becomes the run's display name when comparing.

Comparison modes

Given a baseline and a new run, nano-bench-compare joins same-named series across the files:

• Paired by name (default) — one before/after test per function name that appears in two or more files (two series → Mann-Whitney; three or more versions of one name → Kruskal-Wallis within that group). A name that appears only once is listed in the summary table but not tested. This is the headline "save a baseline, run a new version, compare" case.
• Pooled (--pooled) — throw every series into one Kruskal-Wallis omnibus with a Conover-Iman post-hoc matrix across all of them. It answers "which of these series differ from which", but for a before/after it buries the meaningful same-name pairs among cross-name ones and lets unrelated series perturb the ranks — so it is opt-in.
• Degrade — if no name is shared (a single file, or files with disjoint names), the default falls back to one omnibus so a single-file render still reproduces the original in-run table.

$ npx nano-bench-compare after.json                      # re-render one saved run
$ npx nano-bench-compare before.json after.json          # before/after, paired by name
$ npx nano-bench-compare before.json after.json --pooled # one omnibus over all series
$ npx nano-bench-compare a.json b.json c.json            # k-way comparison

Comparability banner

Recomputing from saved samples is exact, but it does not make a cross-machine comparison valid. Above the results, nano-bench-compare warns on anything that could confound the verdict:

• Environment differences — runtime/engine version, OS/arch, CPU model and count, total memory. (host and the instantaneous cpu.speedMHz are shown as provenance but never trigger a warning.)
• Parameter differences — alpha, samples, bootstrap, correction.
• Body changes — for a same-named pair, a differing bodyHash (sha256 of the function source) means the code changed, so a measured delta may be real; an identical hash means any delta is environment or noise.

Read a warned comparison with care — the difference may be the machine, not the code.

Reproducibility

Every saved run records its bootstrap seed, and each series uses its own seeded mulberry32 stream, so a recompute reproduces the original run's median, CI, and verdict exactly. The rank tests are deterministic regardless of seed. Pass --alpha to deliberately re-decide at a different threshold.

Output

The summary table (median, CI, op/s, batch) is shown for every series, followed by a significance block per comparison — the same renderer nano-bench uses, with the fastest series marked 🐇 and the slowest 🐢. With --histogram, a distribution chart per series follows.

When the loaded files carry per-run system metrics (recorded by nano-bench-io --metrics), their medians are rendered side by side in a metrics table after the summary — memory and I/O diff across runs the same way timing does. Series without metrics are simply absent from that table.

Running

Run it the same way as the other tools: with a global install (npm install -g nano-benchmark) nano-bench-compare is on the PATH; otherwise prefix with npx. Comparison only reads JSON and uses no high-precision timers, so any of Node, Bun, or Deno works:

$ npx nano-bench-compare before.json after.json
$ bun node_modules/nano-benchmark/bin/nano-bench-compare.js before.json after.json
$ deno run -A node_modules/nano-benchmark/bin/nano-bench-compare.js before.json after.json

Unlike nano-bench and nano-watch, this tool has no --self option — it is not benchmarking, so there is no calibration step to keep out of the timed path.