Skip to content

Benchmark host baselines

Jerry Holland edited this page Sep 29, 2026 · 3 revisions

Benchmark host baselines

Benchmark latency is specific to a machine. Select an explicit, stable host label when checking or recording a baseline; linux or win32 alone does not establish which machine produced a measurement.

The repository testing guide is the authoritative contract. This page explains the everyday workflow.

Check an existing baseline

From the repository checkout, use the label assigned to this machine:

npm run test:benchmark -- --host lab-linux-a
npm run test:benchmark -- --host lab-linux-a --json

Alternatively, set VLAB_BENCHMARK_HOST to the label in your shell environment. An explicit --host takes precedence. Keep a label stable across toolchain upgrades and assign different labels to different machines. Labels contain 1–64 lowercase ASCII letters, digits, dots, underscores, or hyphens, starting with a letter or digit. Choose a public-safe label; the recorder does not derive one from the hostname or a private machine identifier.

The check builds disposable scale, three-mode forecast, and six-change publication fixtures. Process counts must not grow; latency uses the unchanged limit max(2 × baseline, baseline + 5 ms). Forecast semantic equality checks always run. Unsupported forecast modes are explicitly skipped.

Interpret the result

Selection What is compared Qualification meaning
Identified host with matching hardware and benchmark settings Available latency and deterministic measurements A passing check supplies latency evidence for that identified host.
Missing, unknown, or mismatched identity with a compatible historical OS reference and no engine/session overrides Deterministic counts only; latency skipped A deterministic pass does not qualify host latency.
No compatible reference Baseline comparisons skipped; semantic checks still run No baseline regression qualification.

JSON identifies the selected reference and reports latencySkipped. passed: true together with latencySkipped: true does not satisfy release latency qualification. Fully skipped baseline comparisons report skipped: true. Read the notes and per-metric skips as well as the exit status.

Establish or deliberately refresh a baseline

On the actual quiet qualification machine, from a clean checkout:

npm run benchmark:record -- --host lab-linux-a
npm run test:benchmark -- --host lab-linux-a

Recording refuses before measurement if the label is missing. Review the diff and provenance, then commit the baseline with the reason for recording. Do not re-record merely to clear a regression; investigate the cause first.

Each identified entry records its label, platform, architecture, CPU models and logical CPU count, memory capacity, OS release, Git and Node versions, and VLAB_ENGINE, VLAB_GIT_SESSION, and VLAB_FORECAST_ENGINE overrides. Hardware and overrides must match before comparing latency. OS/toolchain versions remain provenance for interpreting upgrades; they do not automatically create a new identity. Changed hardware or settings require deliberate re-recording.

Schema vcs-lab.benchmark-baseline/v3 stores identified machines under hosts. The former Linux/Windows entries remain unchanged in legacyHosts: their original machines cannot be identified from the old records, so their latency limits are never borrowed. Recording preserves other entries and historical measurements. Unsupported schemas or changed profiles/tolerances require an explicit migration instead of silently dropping data or relaxing limits.

Qualification and further reading

Maintainers must establish identified baselines on the actual qualification machines before using latency as release evidence. The Windows re-record in #22 established lab-windows-a. Current releases qualify latency on that matching Windows machine only, as selected in #68. The Linux commands above illustrate a future recording; no identified Linux baseline exists. Linux latency is unqualified, while Windows/POSIX functional release testing remains required. See the qualification procedure for current checks and the evidence needed to add Linux coverage. Real-repository, multi-host evidence and budget ratification remain in #42. These issues carry progress and acceptance criteria. A successful synthetic benchmark does not ratify production budgets.

Clone this wiki locally