Reprobench is a reproducible benchmark runner powered by Nix flakes.
It helps you run, compare, guard, and publish benchmark results from pinned development environments.
Benchmark results are only meaningful when run in a consistent environment. Reprobench ties together:
- Pinned environments via Nix flakes — no more "it was faster on my machine"
- Structured results — benchmarks are stored as JSON for easy diffing and archiving
- Comparison — compare current results against a baseline with percentage deltas
- Markdown reports — generate tables ready to paste into GitHub READMEs or PRs
- CI guards — detect performance or size regressions automatically
- Extensible — designed to support multiple runtimes, languages, and environments
pnpm add -D reprobenchOr install globally:
npm install -g reprobenchpnpm add -D reprobench
pnpm reprobench init
pnpm reprobench run
pnpm reprobench compare
pnpm reprobench reportInitialize a reprobench.config.json in the current directory. If flake.nix is found, defaults to manager: "nix".
reprobench init
reprobench init --force # overwrite existing config
reprobench init --manager nix # force Nix manager
reprobench init --manager local # force local managerAlso creates the bench/results/ directory.
Run all benchmark tasks defined in the config.
reprobench runEach task's stdout is saved as JSON (or raw text) to the configured output path.
Compare two benchmark result files.
reprobench compare # uses config paths
reprobench compare bench/results/baseline.json bench/results/latest.json
reprobench compare --json # JSON outputExample output:
size
✓ single-small: 166 -> 139 bytes (-16.27%)
speed
✓ encode batch: 7700 -> 16000 ops/s (+107.79%)
Generate a Markdown report from benchmark results.
reprobench report
reprobench report --input bench/results/latest.json --output bench/results/report.mdExample output:
## Benchmark Results
| group | benchmark | value | unit |
| ----- | ------------ | -----: | ----- |
| size | single-small | 139 | bytes |
| speed | encode batch | 16,000 | ops/s |Check guard conditions. Exits with code 1 if any guard fails.
reprobench guardExample output:
✓ single-small: 139 bytes <= 200 bytes
✗ encode batch: 12000 ops/s < 14000 ops/s
Check the project configuration and environment.
reprobench doctorVerifies config existence, schema validity, file paths, package.json scripts, and more.
Set environment.manager to "nix" to run benchmark tasks inside the Nix dev shell. This ensures the benchmark runs in the exact pinned environment defined by your flake.nix.
{
"environment": {
"manager": "nix",
"flake": ".",
"shell": "default"
},
"tasks": {
"bench": {
"command": "pnpm bench",
"output": "bench/results/latest.json"
}
}
}reprobench run will execute:
nix develop .#default --command bash -lc "pnpm bench"{
"$schema": "https://reprobench.dev/schema.json",
"project": "my-library",
"environment": {
"manager": "nix",
"flake": ".",
"shell": "default"
},
"tasks": {
"bench": {
"command": "pnpm bench",
"output": "bench/results/latest.json"
}
},
"compare": {
"baseline": "bench/results/baseline.json",
"current": "bench/results/latest.json"
},
"report": {
"input": "bench/results/latest.json",
"output": "bench/results/report.md"
},
"guards": {
"benchmarks": {
"encode batch": {
"min": 14000,
"unit": "ops/s"
},
"single-small": {
"max": 200,
"unit": "bytes"
}
}
}
}Benchmark tasks should output JSON to stdout in this format:
{
"project": "my-library",
"timestamp": "2024-01-01T00:00:00Z",
"benchmarks": [
{
"name": "encode single-small",
"group": "speed",
"value": 16038,
"unit": "ops/s"
},
{
"name": "batch-homogeneous-256",
"group": "size",
"value": 5316,
"unit": "bytes"
}
]
}| Unit | Direction |
|---|---|
| bytes | lower-is-better |
| ms | lower-is-better |
| s | lower-is-better |
| ops/s | higher-is-better |
| count | higher-is-better |
| (other) | higher-is-better |
You can override the direction per entry using the direction field:
{
"name": "my-bench",
"value": 42,
"unit": "custom",
"direction": "lower-is-better"
}name: Check
on:
pull_request:
push:
branches:
- main
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: DeterminateSystems/nix-installer-action@main
- run: nix develop --command pnpm install --frozen-lockfile
- run: nix develop --command pnpm checkname: Benchmark
on:
pull_request:
push:
branches:
- main
jobs:
benchmark:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: DeterminateSystems/nix-installer-action@main
- run: nix develop --command pnpm install --frozen-lockfile
- run: nix develop --command pnpm reprobench run
- run: nix develop --command pnpm reprobench guardIf environment.manager is set to "nix", Reprobench itself will enter the configured Nix dev shell before running each benchmark task — no need to wrap the run step manually.
This project provides a Nix flake for reproducible development:
nix develop
pnpm install
pnpm checkThe dev shell includes Node.js 22, pnpm, and git.
You can run reprobench without installing it via npm:
nix run github:minagishl/reprobench -- --help
nix run github:minagishl/reprobench -- init
nix run github:minagishl/reprobench -- compare baseline.json latest.jsonThis project uses Oxc tools for formatting and linting.
pnpm fmt
pnpm fmt:check
pnpm lint
pnpm lint:fixNo Biome, ESLint, or Prettier configuration files are present in this project.
size
✓ single-small: 166 -> 139 bytes (-16.27%)
speed
✓ encode batch: 7700 -> 16000 ops/s (+107.79%)
## Benchmark Results
| group | benchmark | value | unit |
| ----- | ------------ | -----: | ----- |
| size | single-small | 139 | bytes |
| speed | encode batch | 16,000 | ops/s |{
"comparisons": [
{
"name": "single-small",
"group": "size",
"unit": "bytes",
"baseline": 166,
"current": 139,
"delta": -27,
"deltaPercent": -16.27,
"direction": "lower-is-better",
"improved": true
}
]
}This project is licensed under the MIT License - see the LICENSE file for details.