Accuracy Notes

Accuracy Notes

loadbearer is a relative instrument. Every number it produces is only as meaningful as the conditions it was measured under, and a comparison is only valid between runs made the same way. This page is the list of things that change a result and the things loadbearer deliberately doesn't try to measure.

Build flags

The CPU integer and float kernels compile to whatever vector width the target CPU allows. A default cargo build --release targets the portable SSE2 baseline (2-wide f64). Building with:

RUSTFLAGS="-C target-cpu=native" cargo build --release

lets them use AVX2 or AVX-512 where the machine has them, which raises the CPU numbers — sometimes substantially on the float and hash subtests.

This is fine as long as both machines you're comparing use the same build. The released binaries are portable builds; if you build native on one machine, build native on the other too, or compare a released binary against a released binary. compare can't detect this mismatch — it's on you.

Filesystem choice for the disk benchmark

• Point --target-dir at the actual disk you care about. The scratch file goes there.
• Not a RAM disk. On Linux, tmpfs accepts O_DIRECT as a no-op, so a scratch file on /tmp (if /tmp is tmpfs, which is common) measures memory bandwidth and labels it "disk". loadbearer detects a tmpfs/ramfs target on Linux and adds a note; on macOS/Windows it can't.
• Unbuffered I/O support. NTFS, ReFS, ext4, xfs and btrfs all honour O_DIRECT / FILE_FLAG_NO_BUFFERING. Some network, overlay, and encrypted filesystems reject it; loadbearer falls back to buffered I/O and records a note that the read figures may be cache-influenced. A buffered result is still useful for same-filesystem comparisons.
• Free space. A normal run needs ~1 GiB free in --target-dir (2 GiB at thorough); the run fails cleanly if there isn't enough.

Thermals and background load

A thin laptop that throttles under sustained all-core load is slower under sustained all-core load — that's a real property worth grading. But if you want a clean, repeatable number rather than a whatever-state-it-was-in number:

• Run on mains power, not battery (power profiles cap clocks).
• Let the machine cool between runs.
• Close other work — a browser doing background tabs will show up as variance.
• Use --duration thorough; the longer budget and larger iteration count average over short spikes.

The low / medium confidence flags exist precisely to tell you when a result was jittery. A low-confidence component in the "Why" line means "don't lean on this number" — re-run under better conditions.

Queue depth

The disk random-I/O subtests run at queue depth 1 — one I/O outstanding at a time. This is a latency-bound measurement and produces smaller numbers than a QD32 tool (fio, CrystalDiskMark, ATTO), which measure how much parallel I/O the drive and its controller can absorb.

QD1 was chosen because it's simple, single-threaded, portable, and still cleanly separates storage classes (NVMe ≫ SATA SSD ≫ HDD for random 4K). It is not the right tool if you specifically care about high-queue-depth throughput for a database or a busy server — benchmark that workload directly.

Security tooling (EDR)

An endpoint-detection agent — CrowdStrike, SentinelOne, Defender for Endpoint, Carbon Black — hooks the kernel and inspects process, file and network activity. Measured on a CrowdStrike-managed Windows 11 laptop:

• Network. Every loopback send/recv goes through the agent's Windows Filtering Platform driver. UDP packet-rate dropped ~15× versus Linux, TCP round-trip sat at a ~55 µs floor, and within-run variance was huge and erratic (the agent cutting in on some iterations). This is the main reason the network component is not in the overall grade.
• Everything else. A benchmark that pegs every core, does direct unbuffered disk I/O and opens raw sockets draws behavioural scrutiny, so CPU and memory subtests showed elevated variance too (±40–60% on single-threaded memory bandwidth on that machine).
• Disk was fine. The unbuffered-I/O path punched straight through to real device speed — the agent doesn't sit in that path the way it sits in the network path.

On a managed machine, prefer --duration thorough, run it a few times, and read the confidence flags — they will be medium/low and that's correct.

Reproducibility

• --seed N fixes the pseudo-random data the workloads use (buffer contents, random offsets). Two runs with the same seed, machine and build differ only by measurement noise.
• Seeding does not freeze timings. Wall-clock rates always vary run to run with thermal and scheduling state; that's what the repeated iterations, the median, and the confidence flag handle.
• The default seed is fixed, so runs are already seed-stable unless you change it.

What loadbearer does not measure

• GPU. No compute, graphics, or video-decode benchmark.
• The physical network link. The network component is loopback only — the machine's network stack, not a NIC, cable or wifi radio. loadbearer makes no network calls unless you ask it to. The opt-in loadbearer run --net-target measures a real link between two machines, reported separately (see Benchmark Methodology).
• Anything in the overall grade about network. Even the loopback network score is kept out of the overall — it's dominated by the OS and any EDR, not the hardware (see below).
• Sustained thermal behaviour as its own metric. A thorough run is a few minutes per component, long enough to catch throttling as lower scores and higher variance, but there's no dedicated "how does it do after 30 minutes flat out" test.
• Multi-threaded memory bandwidth. The memory subtests are single-threaded (see Benchmark Methodology).
• Real-world application performance. The kernels are representative microbenchmarks, not a compile, an encode, or a game. They rank hardware; they don't predict how long your specific workload takes.

When two runs disagree

If the same machine grades differently on two runs:

1. Check the confidence flags on both — a low component explains most of it.
2. Check they used the same --duration and the same build.
3. Check nothing else was running, and it wasn't on battery / warm from the last run.
4. If it still moves more than a few percent at thorough under good conditions, that spread is itself a finding about the machine.