Comparing Machines

Comparing Machines

loadbearer compare is the point of the tool. Run loadbearer run --output on each machine, collect the JSON files, and:

loadbearer compare thinkpad.json precision.json
loadbearer compare a.json b.json c.json d.json        # any number ≥ 2
loadbearer compare a.json b.json --json               # structured output

It uses the raw metrics, not the scores

compare reads the raw section of each result file and works from the actual measurements. It does not trust the components / overall scores those files were written with. This matters because:

• The two files might have been scored against different baselines or with a different --curve-k, which makes their stored scores incomparable.
• The raw metrics are curve- and baseline-independent — a MiB/s is a MiB/s — so a verdict built from them is stable no matter how either file was scored.

It still shows each machine's stored overall score in the header, as context, but the verdict below is computed fresh.

The network component is compared — it's part of raw like the others — but only meaningfully between machines running the same OS. Loopback performance is an OS property as much as a hardware one, so compare prints a warning when the two files report different operating systems. A --net-target link result is not compared at all: it measures the path between two hosts.

How the verdict is built

The first file on the command line is the reference. For every subtest present in all files, each machine gets a direction-adjusted ratio to the reference:

goodness = value / reference_value           (higher-is-better)
goodness = reference_value / value            (lower-is-better: latency)

So goodness > 1 always means "better than the reference machine on this metric", and the displayed percentage (+28%, -12%) reads the same way for throughput and for latency.

• Per component: the geometric mean of that component's subtest goodness values.
• Overall: the geometric mean of the per-component figures — each component weighted equally, so a component with more subtests doesn't count for more.
• Ranking: machines sorted by overall goodness, best first.

compare does not apply a profile — it's an equal-weight, raw-metric comparison. If you want a profile-weighted verdict, compare the stored overall scores from two runs made with the same --profile and baseline.

Reading the table

  A  thinkpad-x280      Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
     normal preset · baseline reference-v1 · k=0.5 · stored overall 731 [C]  (thinkpad.json)
  B  precision-5560     Intel(R) Core(TM) i7-11800H
     normal preset · baseline reference-v1 · k=0.5 · stored overall 1180 [A]  (precision.json)

  metric                           A: thinkpad-x280 B: precision-5560
  CPU
  Integer, single-core (Mops/s)               9238       13396 +45%  B
  Integer, all cores (Mops/s)                37346       54152 +45%  B
    → component                                ref            +45%  B
  MEMORY
  Sequential read (GiB/s)                      12.8        15.1 +18%  B
  Random access latency (ns)                  145.3       123.1 +18%  B
    → component                                ref            +18%  B

  OVERALL                                      ref            +31%  B

  Verdict: precision-5560 leads by 31% overall (ahead on cpu +45%, memory +18%).

• Each machine gets a tag (A, B, C, …). The reference column shows raw values; every other column shows value ±percent relative to the reference.
• The → component row is that component's geometric-mean goodness.
• The trailing tag on each row (B, or =) is the winner. It shows = when the top two machines are within 2% of each other on that row — a difference that small isn't a real result.
• For two machines the Verdict is a sentence: who leads overall, by how much, and which components each side wins. Under 3% overall it says the machines are "effectively equal". For three or more it's a ranked list with each machine's gap to the leader.

Warnings

compare prints a ! line for anything that undermines the comparison, then proceeds anyway:

Warning	Meaning
files used different duration presets	One run was short and is noisier; prefer comparing runs at the same preset, ideally thorough.
files were scored against different baselines	Only affects the stored scores shown in the header; the verdict is unaffected.
files used different curve k	Same — stored scores only.
files are from different operating systems	The network component in particular is not comparable across OSes (loopback is an OS property). CPU / memory / disk are fine.
component / subtest X is not present in every result file	It was excluded from the comparison. Happens when one run used --only.

If the files share no common component at all, compare errors instead of producing an empty verdict.

JSON output

--json emits the full structure — machines, warnings, components (with per-subtest values and rel arrays, one entry per machine), and overall (rel, ranking, summary). Useful for feeding a dashboard or asserting on a threshold in CI:

loadbearer compare baseline.json candidate.json --json \
  | jq -e '.overall.rel[1] >= 0.95'      # candidate within 5% of baseline