Performance Research Test Protocol

Performance research — common test protocol

Common implementation and test protocol

Unproven performance research · Performance research — reproducible checks

Use this protocol for the 26 research addenda. Topic pages contain the specific mechanism, hazards, first patch boundary, counters and focused cases. This page defines the shared evidence contract so that each page need not repeat it.

1. Identify what exists before implementing

The research source pin is 322986f4c5502b3fa887a9605ac8cd88e9b2c579; it is not automatically the current baseline. Resolve the source being tested from a clean checkout and record its full commit and tree IDs. Read the current implementation and linked issue/PR history before applying a historical proposal: some mechanisms already exist, some repairs are historical, and some discussions retain obsolete status text.

git status --porcelain=v1
git rev-parse HEAD
git rev-parse HEAD^{tree}

Use the build instructions and pinned toolchain for the current project. Do not compare Release candidate timings against a Debug control. The source pin supports this review's claims; it is not a mandate to undo newer accepted changes.

2. Write the smallest falsifiable claim

Name the operation to remove, the complete eligibility predicate, the outputs/state that must not change, and the metric expected to improve. Split independent changes: HRTF target correctness versus indexing, DMA allocation ownership versus transfer batching, TLB inlining versus mode filtering, and SPIR-V persistence versus driver pipeline precreation.

For example, A11 claims that a table populated using the same attenuation expression and environment returns identical float bits for all masked inputs. It does not claim a universal FPS improvement. G03 claims that some report waiting may overlap useful work before a required observation; it does not claim all recorded waiting is removable.

3. Run focused correctness controls first

Implement the topic's proposed test IDs against the real production path. The standalone reductions on the checks page are supporting evidence, not substitutes for production tests.

Use a control that would expose the specific bug: one-byte overlap for a range guard, direct guest RAM mutation for a cache, a delayed reader for lifetime, or left-sum-zero/one for the HRTF target update. Demonstrate that the control can fail the defective or deliberately invalid variant. Do not silently relax tolerances after observing candidate failures.

Compare all relevant observables: samples and future DSP state; RAM/register/IRQ transitions; cursor/notifier ordering; UBO bytes, descriptors and rendered output; query totals/destinations; or TLB/callback state. A screenshot alone is insufficient for state/lifetime changes. A final-state hash alone may miss an intermediate ordering defect.

4. Attribute opportunity in a diagnostic build

Use exact operation counts and bounded histograms to establish eligible work. Label sampled timers as sampled and record sample coverage. Aggregate outside the timing window; do not add synchronous logging, allocation or locks that manufacture the hotspot.

Keep local CPU execution, GPU active work, host waiting, guest progress and displayed presentation distinct. Sum mutually exclusive wait regions or use a timeline; do not add overlapping measurements as though they were independent elapsed time.

5. Compare uninstrumented controls

Use three identities: previous main, isolated candidate, and the project's fixed cycle baseline. Use same-session, order-balanced repetitions such as A/B/B/A for previous-main versus candidate; interleave the fixed baseline rather than comparing only to an unrelated old run. Repetitions characterize host scheduling/cache/driver variability even when guest input starts from the same snapshot.

Keep workload, guest-time settings, cache policy, renderer, GPU, resolution/scale, audio configuration, host power state and measurement interval fixed. For persistent-cache work, cold/warm cache is an explicit experimental factor, not an uncontrolled side effect. Record cache identity and whether the driver cache is controlled or merely observed.

A 30-FPS cap does not make CPU savings irrelevant; use fixed guest work and CPU time to expose headroom without changing guest clocks. Report guest display-write cadence as guest progress, not displayed FPS. Do not unlock clocks or caps in just one arm of a same-work performance comparison.

6. Keep a small, machine-readable result set

Use one JSON manifest per experiment and one row per metric per run. The following fields are the result contract; null means not measured, never an invented zero:

{
  "workstream": "A11",
  "case_id": "A11.01",
  "evidence_class": "PROPOSED_TEST",
  "status": "NOT_RUN",
  "source_sha": null,
  "tree_sha": null,
  "executable_sha256": null,
  "build_profile": null,
  "toolchain_identity": null,
  "settings_sha256": null,
  "workload_sha256": null,
  "cache_policy": null,
  "oracle": "float-bit equality for all 65536 uint16 inputs",
  "oracle_result": null,
  "raw_results": [],
  "limitations": ["No product run recorded in this template"]
}

Recommended normalized CSV header:

workstream,case_id,run_id,build_role,source_sha,executable_sha256,workload_id,cache_state,metric,value,unit,direction,status

Store per-test rows and logs in the existing evidence repository/layout, not as a giant wiki table. For a PR, use its dedicated folder under the retained evidence/wiki-xiso-per-test convention; a research-only run needs a unique experiment folder until a PR number exists. The wiki should link the evidence, show the few deciding numbers and retain the exact source/build/settings identities.

Report raw values beside percentage changes. For lower-is-better durations, use 100 * (reference - candidate) / reference; for higher-is-better throughput, use 100 * (candidate / reference - 1). Do not invent a percentage for a zero denominator. Derive confidence from paired runs or suitable per-run statistics; do not treat consecutive correlated frames as independent full experiments.

7. Progressively widen qualification

After the focused candidate passes and shows a plausible mechanism, run the complete relevant native/XISO suite in the supported renderer matrix, then fresh-start and snapshot retail routes, reset/restore/renderer lifecycle, and sustained resource checks. Determine the current suite's expected record count from its actual manifest; do not freeze the historical 147/157/159 counts into new acceptance.

Retain inherited failures as failures with references. No new validation errors, output mismatches, unresolved lifetime problems or repeatable adverse tails may be erased by a favorable average. Store rejected or neutral findings; they prevent repeating the same experiment later.

Decision table

Decision	Required basis
Implement a focused experiment	Defined mechanism, complete eligibility predicate and a discriminating first test
Correctness-qualified	Required production oracles/lifecycle gates passed for the exact candidate; coverage and limitations stated
Performance-supported	Uninstrumented same-work comparisons support a useful result beyond noise; relevant total-work/tail metrics included
Hold	Missing native gate, unclear state/ownership contract, unresolved regression or insufficient attribution
Reject this candidate	Demonstrated semantic defect, unsupported gain, or repeatable cost exceeding its benefit; preserve evidence

The research addenda do not change accepted-main policy or grant any candidate automatic merge approval.