Repository navigation
Benchmark host baselines
Benchmark latency is specific to a machine. Select an explicit, stable host
label when checking or recording a baseline; linux or win32 alone does not
establish which machine produced a measurement.
The repository testing guide is the authoritative contract. This page explains the everyday workflow.
From the repository checkout, use the label assigned to this machine:
npm run test:benchmark -- --host lab-linux-a
npm run test:benchmark -- --host lab-linux-a --jsonAlternatively, set VLAB_BENCHMARK_HOST to the label in your shell environment.
An explicit --host takes precedence. Keep a label stable across toolchain
upgrades and assign different labels to different machines. Labels contain
1–64 lowercase ASCII letters, digits, dots, underscores, or hyphens, starting
with a letter or digit. Choose a public-safe label; the recorder does not derive
one from the hostname or a private machine identifier.
The check builds disposable scale, three-mode forecast, and six-change
publication fixtures. Process counts must not grow; latency uses the unchanged
limit max(2 × baseline, baseline + 5 ms). Forecast semantic equality checks
always run. Unsupported forecast modes are explicitly skipped.
| Selection | What is compared | Qualification meaning |
|---|---|---|
| Identified host with matching hardware and benchmark settings | Available latency and deterministic measurements | A passing check supplies latency evidence for that identified host. |
| Missing, unknown, or mismatched identity with a compatible historical OS reference and no engine/session overrides | Deterministic counts only; latency skipped | A deterministic pass does not qualify host latency. |
| No compatible reference | Baseline comparisons skipped; semantic checks still run | No baseline regression qualification. |
JSON identifies the selected reference and reports latencySkipped.
passed: true together with latencySkipped: true does not satisfy release
latency qualification. Fully skipped baseline comparisons report skipped: true.
Read the notes and per-metric skips as well as the exit status.
On the actual quiet qualification machine, from a clean checkout:
npm run benchmark:record -- --host lab-linux-a
npm run test:benchmark -- --host lab-linux-aRecording refuses before measurement if the label is missing. Review the diff and provenance, then commit the baseline with the reason for recording. Do not re-record merely to clear a regression; investigate the cause first.
Each identified entry records its label, platform, architecture, CPU models and
logical CPU count, memory capacity, OS release, Git and Node versions, and
VLAB_ENGINE, VLAB_GIT_SESSION, and VLAB_FORECAST_ENGINE overrides. Hardware
and overrides must match before comparing latency. OS/toolchain versions remain
provenance for interpreting upgrades; they do not automatically create a new
identity. Changed hardware or settings require deliberate re-recording.
Schema vcs-lab.benchmark-baseline/v3 stores identified machines under hosts.
The former Linux/Windows entries remain unchanged in legacyHosts: their
original machines cannot be identified from the old records, so their latency
limits are never borrowed. Recording preserves other entries and historical
measurements. Unsupported schemas or changed profiles/tolerances require an
explicit migration instead of silently dropping data or relaxing limits.
Maintainers must establish identified baselines on the actual qualification
machines before using latency as release evidence. The Windows re-record in
#22 established lab-windows-a.
Current releases qualify latency on that matching Windows machine only, as
selected in #68. The Linux
commands above illustrate a future recording; no identified Linux baseline
exists. Linux latency is unqualified, while Windows/POSIX functional release
testing remains required. See the qualification procedure
for current checks and the evidence needed to add Linux coverage. Real-repository,
multi-host evidence and budget ratification remain in
#42. These issues carry progress
and acceptance criteria. A successful synthetic benchmark does not ratify
production budgets.