The Baseline

The Baseline

The baseline is the set of reference measurements every raw value is compared against. It's what makes a score mean something on its own rather than only in a head-to-head.

reference-v1

The built-in baseline is embedded in the binary at compile time; its source is baseline/reference-v1.toml. Each value is the geometric mean of that metric across seven real machines benchmarked at --duration thorough from the released (default, non-native) build:

host	CPU	cores	OS
SGS-D47TDY3	Intel Core i7-1370P (2023)	20	Windows 11
dell-ubuntu	Intel Core i7-10510U (2019)	8	Linux
ThinkPad-X280	Intel Core i5-8350U (2018)	8	Linux
DESKTOP-QRLKF2J	Intel N95 (2023)	4	Windows 11
docker-nuc	Intel Core i5-5250U (2015)	4	Linux
LYNN-LAPTOP	Intel Core i5-6300U (2015)	4	Windows 10
openvms	Intel Celeron J4005 (2017)	2	Linux

Component	Subtest	Baseline value
cpu	int_single	6432 Mops/s
cpu	int_multi	17660 Mops/s
cpu	float_single	5684 MFLOP/s
cpu	float_multi	23030 MFLOP/s
cpu	hash	1873 MiB/s
cpu	compress	44.37 MiB/s
cpu	aes_gcm	760.2 MiB/s
cpu	sha256	355.7 MiB/s
memory	bw_read	10.64 GiB/s
memory	bw_write	6.849 GiB/s
memory	bw_copy	6.498 GiB/s
memory	bw_read_mt	16.32 GiB/s
memory	latency	147.6 ns
disk	seq_write	342.3 MiB/s
disk	seq_read	571.6 MiB/s
disk	rand_read	8195 IOPS
disk	rand_write	10820 IOPS
network	tcp_stream	1.956 GiB/s
network	tcp_parallel	2.939 GiB/s
network	tcp_rtt	39.80 µs
network	udp_pps	93.13 Kpps
gpu	compute_fp32	282.0 GFLOP/s
gpu	bandwidth	20.52 GiB/s

A run that hits every one of these scores 1000 on every component and grades straight B. (The gpu row is the geomean over the four sample machines that had a usable GPU; GPU is not folded into the overall grade, so a discrete card scoring far above 1000 there is expected.)

It is a small sample

Seven machines, all Intel, skewed toward older low-power laptops. Consequences:

• The anchors sit low: against them a current mainstream machine grades A/S, and the 2015-2018 chips land B/C.
• Five of the seven lack the SHA instruction extension, so sha256 anchors at ~356 MiB/s — a machine with the extension scores several times that on the subtest (the component geomean dampens it).
• No AMD, no ARM, no desktop/workstation-class hardware.

So:

• Treat a single machine's absolute score and letter as a rough position, not a verdict.
• For anything that has to be defensible, use compare, which doesn't touch the baseline.
• The baseline is calibration data, not part of the stability contract (VERSIONING.md); it will be re-anchored as more machines are measured, and a recalibration is a normal minor release even though it shifts everyone's absolute numbers. If you have representative hardware, build your own.

If a benchmark subtest ever lacks a baseline entry, scoring fails with a hard error rather than silently skipping it — the baseline can't quietly fall out of sync with the code.

Recalibrating

loadbearer baseline with no arguments prints the embedded baseline. Given result files, it emits a new baseline whose every value is the geometric mean of that metric across the inputs:

# collect runs from machines you consider "the standard"
loadbearer run --output ref-1.json
loadbearer run --output ref-2.json
loadbearer run --output ref-3.json

# average them into a baseline and replace the shipped one
loadbearer baseline ref-1.json ref-2.json ref-3.json \
  --name reference-v2 \
  --description "our 2026 standard-issue laptops" \
  > baseline/reference-v1.toml

cargo build --release

Subtests missing from some inputs (e.g. one run used --only) are still averaged over whatever files have them, with a warning on stderr.

Guidance for a good baseline: use runs at the same --duration (ideally thorough), all built the same way (all target-cpu=native or all portable), on machines that genuinely represent your "par" hardware — the median of your fleet, not the best or worst. Three to five machines is plenty; the geometric mean is stable.

Re-scoring without re-running

A generated baseline only becomes the embedded one after a cargo build --release. But you don't have to rebuild — or re-run any benchmark — to see a machine's grade against a different baseline:

# how does this machine look against our fleet baseline?
loadbearer score result.json --baseline our-fleet.toml

# ...and under the server profile, with a steeper curve?
loadbearer score result.json --baseline our-fleet.toml \
  --profile server --curve-k 0.7 --output result-fleet.json

score reads the result file's raw metrics and recomputes components / overall only — nothing is measured again. --profile and --curve-k default to whatever the file was scored with; a short banner shows the before → after for each knob (including the loadbearer version, which explains score drift on an old file even when the baseline is unchanged). Subtests the chosen baseline has no entry for are left out of the score with a note rather than being an error — a fleet baseline that omits the OS-dependent network component is normal. The file's full raw is copied into the --output file untouched, so it can be re-scored again later.

Result files

loadbearer run --output result.json writes a versioned document. schema is "loadbearer.result/1"; a future breaking change bumps the number.

Field	Contents
schema	"loadbearer.result/1"
tool_version	the loadbearer version that produced it
timestamp	RFC 3339, UTC
machine	full inventory — identical to loadbearer info --json (host, OS, CPU model/vendor/cores, RAM, disks, and the GPU and battery when the machine has them)
config	profile, duration_preset, curve_k, seed, threads, baseline name, only
raw	every benchmark, every subtest: direction, unit, the median value, full stats (every timed run, median, mean, min, max, stddev, cv), and the confidence flag
components	scored: per component a score, grade, confidence, a graded flag (whether it counts toward the overall — false for network), and per subtest the value, baseline, ratio, score, confidence
overall	score, grade, profile, and the why lines — computed from the graded components only
link	present only when --net-target was used: target, tcp_upload_gibps, tcp_rtt_us, udp_send_kpps — ungraded, not part of raw/components
soak	present only when --soak was used: every per-second sample (t_secs, rate, mhz) plus the derived peak_rate, steady_rate, retained_pct, onset_secs, steady_cv_pct, mhz_peak/mhz_steady — ungraded, not part of raw/components

Because raw is preserved in full, a result file is a permanent record: loadbearer score can re-grade it against a different baseline, profile or curve (the scored components are just one view of it), and compare and loadbearer baseline both work straight off raw without trusting the scores.

# what did this machine actually do, ignoring the grade?
jq '.raw[] | {id, subtests: [.subtests[] | {id, value, unit, confidence}]}' result.json

# just the headline
jq '{cpu: .components[0].grade, overall: .overall.score, why: .overall.why}' result.json