Performance Methodology

Performance Methodology

Reviewed: 2026-11-01 · Source: b7140848 · Availability: current main; stable/main boundaries are stated below.

Current acceptance policy

Judge overall game cadence, tails, CPU efficiency, correct rendering and functionality together. Disclose changes over1%; a synthetic2% slowdown is diagnostic, not an automatic veto. Higher CPU can support a worthwhile throughput gain; lower CPU at a cap can be valuable. Establish run variation rather than declaring losses noise. Keep unresolved material regressions or missing required evidence HOLD; a passing build is not performance proof. Use balanced ABBA/BAAB without changing inputs, waits, scenes or analysis windows to obtain a favorable result. Policy#337.

Required XISO reporting includes each affected timed leaf, an unaffected control, before/after time, absolute difference, positive-is-better improvement, backend and correctness; retrieve full per-test reports rather than pass counts. Incomparable timing contracts must say NOT COMPARABLE. Dated gates in old studies remain historical. Measured examples.

Performance work in xemu must separate measurement, causal attribution, correctness, and acceptance. A patch is not a performance improvement merely because one named function becomes cheaper or one counter decreases.

The required question is:

Did the exact candidate reduce the total dependency that limits useful guest or presented progress, without changing correctness or moving the same work somewhere else?

Measurement model

flowchart TD
    A[Choose exact workload and progress metric] --> B[Pin reference and candidate source/build identities]
    B --> C[Validate functional equivalence]
    C --> D[Run matched performance cells]
    D --> E[Collect CPU / GPU / guest-progress evidence]
    E --> F[Attribute the changed path]
    F --> G{Net improvement established?}
    G -->|No| H[Hold / reject / investigate]
    G -->|Yes| I[Broaden workload and lifecycle validation]
    I --> J[Merge-ready evidence]

Loading

Instrumentation can identify a cause, but acceptance numbers should normally come from builds without intrusive trace overhead unless the instrumentation itself is proven negligible and identical on both sides.

1. Pin source and executable identity

Every result must be tied to an exact source state.

Record at minimum:

Identity	Why it matters
Repository + commit SHA	Defines source code under test
Base/reference commit SHA	Prevents comparing against a moving main
Build profile	Debug/Release, optimization, LTO, architecture flags and assertions affect timing
Executable hash	Proves the binary deployed to the test system is the intended build
Relevant runtime settings	Renderer, resolution scale, cache toggles, VSync, experimental options
GPU/adapter identity	Especially required for Vulkan device-selection work
Presentation transport	Shared external memory versus host-copy can change cost materially

A PR updated after a benchmark does not inherit the old result automatically. Historical measurements must say which earlier head they qualify.

2. Use two reference concepts

When possible, distinguish:

Previous main

The mainline source immediately relevant to the candidate. This answers:

Does this patch improve what would otherwise ship now?

For the default xemu performance reference, use the latest published Mainkill1 main executable whose source tree matches the selected main head. Reuse that exact release asset across all repeated runs; do not rebuild main for each PR or run. If no asset matches the selected tree, build and publish it once, record its source and executable hashes, then use that same executable for the full reference series.

Fixed/stable baseline

A retained source/build whose behavior is already characterized. This answers:

Did main or the candidate drift relative to a known reference?

The two references answer different questions. If current main has already regressed or improved, a candidate-versus-main comparison alone can hide that history.

3. Pin the workload

A performance cell is not just a game name.

Record:

• Title/build.
• Snapshot/save/XISO seed identity where applicable.
• Camera/view/location or deterministic scripted progression.
• Duration or progress window.
• Input sequence.
• Renderer and cache condition.
• Any private-disk/EEPROM/settings state that changes behavior.

A fixed Morrowind camera view and a full traversal are different workloads. A PGR2 snapshot and a fresh boot are different workloads. Keep them separate in tables and conclusions.

The canonical PGR2 full-start timings, controller binding, runner translation and evidence gates are documented in PGR2 Performance Test Routes.

4. Define the progress metric before testing

Metrics must state what event they actually measure.

Metric	What it can establish	What it does not automatically establish
Guest frame interval	Guest-side frame progress/cadence	Presented host FPS
NV2A display writes	Guest GPU/display progress marker	GPU-completed or presented frames
Report/query cadence	Guest-visible report completion	Whole-frame render speed
PresentMon / presentation FPS	Host-presented output when capture is valid	Guest CPU or GPU bottleneck cause
CPU samples	Where host CPU time is spent	Whether reducing that symbol improves frame time
GPU timestamps	GPU execution spans between timestamp points	CPU wait or presentation cost outside the timestamp region
XISO timed leaf and suite duration	Synthetic test throughput and the measured cost of named workload phases under the recorded settings	Retail gameplay FPS, or a candidate improvement without a matched reference

Never rename a proxy metric into a stronger claim. If the measurement is “guest display writes/s,” publish it as that, not “FPS.”

5. Normalize direction before showing percentages

Reader-facing Improvement % should use positive values for favorable changes.

Higher is better

For FPS, writes/s, operations/s, throughput:

Improvement % = 100 × (candidate / reference - 1)

Lower is better

For frame interval, latency, CPU time, GPU time, allocations, stalls:

Improvement % = 100 × (reference - candidate) / reference

If the direction is ambiguous, the reference is zero, or the metric is not meaningfully ratio-scaled, show the raw values and use N/A rather than forcing a percentage.

6. Control run order

Thermals, OS state, shader/driver caches, filesystem caches and process history can bias sequential tests.

Prefer paired and counterbalanced designs such as:

A → B
B → A

or, when appropriate:

A → B → B → A

where A and B are exact reference/candidate configurations.

If order reversal changes the sign of a small result, treat the result as inconclusive rather than averaging away the instability.

7. Separate cold, warm and same-process reuse

Cache state is part of the workload.

At minimum distinguish:

Condition	Meaning
Cold process	No same-process renderer cache history
Cold persistent cache	No previously persisted artifact relevant to this run
Warm persistent cache	Persisted artifacts intentionally reused across launches
Same-process revisit	In-memory caches can reuse objects from an earlier encounter

For shader work, a same-process module hit, a persisted SPIR-V hit and a Vulkan driver pipeline-cache hit are different events. Do not combine them into one “shader cache” metric.

8. Report distributions, not only averages

Emulator performance frequently fails in the tail.

For frame/progress intervals, retain at least:

• Count / duration.
• Mean or median as appropriate.
• p95.
• p99.
• Maximum.
• Explicit stall threshold counts when a threshold is defined.

A patch that improves the mean while introducing rare 80–100 ms cold stalls may be unacceptable even if average FPS increases.

9. Profile by thread and subsystem

Separate at least:

• vCPU thread.
• Main/event/UI thread where relevant.
• PFIFO/PGRAPH/renderer work.
• Worker threads.
• Host graphics driver threads when attributable.

For CPU work, distinguish generated guest code from emulator helpers/dispatch. For GPU work, distinguish useful draw spans from transfers, pass breaks and waits.

A useful classification tree is:

flowchart TD
    A[Slow interval] --> B{CPU-limited?}
    A --> C{GPU-limited?}
    A --> D{Synchronization/wait-limited?}

    B --> E[vCPU generated guest code]
    B --> F[TB dispatch / helpers / device work]
    B --> G[Renderer CPU preparation]

    C --> H[Useful draw work]
    C --> I[Copies / transitions / clears]

    D --> J[GPU fence / glFinish]
    D --> K[Report/query dependency]
    D --> L[Host scheduling/timer wait]

Loading

The categories can overlap; the point is to locate the dependency rather than optimize whichever symbol is easiest to edit.

10. Instrument the decision, not just the duration

Timers are useful only when accompanied by enough identity to explain what happened.

Examples:

Shader investigation

Record separately:

• Binding-cache hit/miss.
• Stage-module hit/miss.
• Exact generated-source identity.
• GLSL generation time.
• glslang compile time.
• Vulkan module-creation time.
• Reflection time.
• Graphics pipeline hit/miss/create time.

Vertex/surface investigation

Record separately:

• Dirty range.
• Surface-overlap/freshness decision.
• Direct versus staged upload.
• Bytes/pages affected.
• Render-pass break reason.
• Submission/fence reason.

Timer investigation

Record separately:

• Guest target tick/deadline.
• Converted QEMU-clock deadline.
• timer_mod / timer_del decisions.
• Actual callback count.
• IRQ state transitions.
• Host wake/oversleep data if host scheduling is under test.

A single aggregate counter cannot prove a mechanism if multiple code paths can increment it.

11. Validate correctness before accepting speed

Performance results are invalid if output changed incorrectly.

Relevant validation layers include:

• Unit/focused tests for changed logic.
• Full XISO coverage where the subsystem is exercised.
• OpenGL/Vulkan differential controls where appropriate.
• Framebuffer/output hashes where determinism allows them.
• Guest memory/report results.
• Vulkan validation/VUID checks.
• Sanitizers for changed ownership/lifetime code.
• Snapshot/save/restore when the patch touches derived or persistent state.

Inherited failures must stay visible. Do not turn a pre-existing nonpass into “PASS” merely because candidate and reference match.

12. XISO result discipline

When XISO is used as a correctness gate, retain the individual test records rather than only a suite-level summary.

For each record, capture enough to identify:

• Test name.
• Renderer/configuration.
• Outcome.
• Functional hash/result where applicable.
• Validation errors.
• Source/build identity.

Large per-test tables should normally live in evidence files with a compact summary linked from the PR/wiki. The Wiki should explain what the test proves, not reproduce hundreds of rows.

A full XISO run also measures performance. Retain timed leaf rows (iterations and total/average/median/p95/min/max duration), the suite's host wall time, renderer, validation and live-marker state. These numbers are descriptive even when a run is primarily used as a correctness gate. Compare performance only against a matched reference with equivalent conditions; active validation and missing live markers limit phase attribution and PR-grade speedup claims, but do not make recorded durations disappear.

PR #147 exact-head XISO reference (2026-09-19)

The a6a1b9d677 build (stripped EXE SHA-256 f743dfc73ea749017009e6e53ac1af7f44271bd8558341d423e77e449ce0f34e) completed the full Vulkan perf suite: 157/157 records (152 timed leaves, five groups), 1,789 measured iterations, 80.616 s summed leaf duration, 102.685 s host wall, functional-hash oracle PASS, and zero VUIDs. Validation was active, live markers were unavailable, and this XISO cell has no matched main control, so these are exact-head performance reference values rather than an established suite speedup. Cold Prewarm scheduled no jobs; the run also saved a 3,170,698-byte driver cache. The separate matched-cache PGR2 pair measured pipeline-worker construction at 44.941 ms without versus 4.740 ms with restored cache, and startup p99 at 40.232 versus 35.479 ms; one pair is descriptive, not repeatability proof. See PR #147 and its exact-head release.

Main 1b582a35 three-run XISO reference (2026-09-19)

The optimized merged-main release (tree 7e17df76, stripped EXE SHA-256 6a0459ccd3d14b664994fdb70e211ca87715cb59b18f0c76520058bdc483ee1a) is the dated default Mainkill1 main reference. Three consecutive GUI-session runs used identical ISO, catalog, canonical oracle, and settings: Vulkan on an RTX 3070 Ti at 1× scale, Prewarm, shader cache on, validation on, telemetry off, and zero warmup. Each passed 157/157 records (152 timed leaves, 1,789 iterations), the functional oracle, and zero VUIDs, then exited normally.

Run	Driver cache restored	Summed timed-leaf duration	Host suite wall time	Process CPU time
01, cold	0 B	81.522 s	102.453 s	213.063 s
02, warm	3,170,698 B	80.965 s	101.050 s	211.344 s
03, warm	3,184,880 B	80.578 s	100.980 s	210.750 s

The median timed-leaf total is 80.965 s (range 80.578–81.522 s). These are validation-enabled suite-performance values, not retail FPS; unavailable live markers limit phase attribution. The earlier single PR #147 exact-head total of 80.616 s falls within this main spread, so this set does not establish a PR speedup. The release includes a sanitized three-run summary asset with exact input/config hashes and per-run totals.

September 19 XISO timing correction and matched rerun

The first official-upstream/fork/candidate per-leaf timing table was withdrawn as a performance comparison. Its XISO launcher configured the display but omitted the fork Advanced controls. The candidate used its Prewarm and shader-fastpath defaults On, while the published fork-main executable used both defaults Off. The post-exit config file was a saved delta that omitted default-valued settings, so it did not expose the mismatch. The original 157/157 fork pass counts and raw guest timings remain valid functional records; the old speed percentages do not qualify a PR.

For the corrected fork comparison, the GUI-session runner requested Prewarm, shader fastpath, and shader cache On on both existing release executables and saved the requested config before launch. The four requested config files were byte-identical. Main–candidate–candidate–main order produced 157/157 passing records each time, with 150 of 152 leaf framebuffer hashes matching across roles. The matched report and per-leaf CSV include both paired timings and within-role drift. The descriptive median candidate speedup was −0.59%; a separate 80-iteration sampler-only texture check found −4.62% candidate speedup, equivalent to 4.84% longer candidate latency. A CPU-only control changed direction relative to the original grouped run. This did not meet PR#135's historical across-the-board gate; current policy evaluates the overall game outcome rather than imposing that dated gate on new work.

All four runs logged 32 ready prewarm pipelines but zero later demand hits, so this XISO path does not demonstrate a prewarm benefit. Live guest markers remained unavailable and the runner used its waiver. Guest leaf durations are useful diagnostic work measurements, but they are not host-correlated frame-time percentiles or displayed FPS.

For future XISO comparisons: set every relevant fork control explicitly; save and hash the pre-launch config; verify the effective feature path in the emulator log; keep source, executable, guest ISO/catalog, cache role, and functional hashes paired; reverse run order when practical. Treat a missing live-marker stream as a limitation, not as a silent success. Keep raw leaf values and host wall/CPU time, and label which metric each percentile represents.

13. Distinguish diagnostics from acceptance runs

Intrusive tracing changes timing.

Use diagnostic builds to answer questions such as:

• Why did this frame stall?
• Which cache missed?
• Why did a render pass end?
• Which thread waited?

Then use normal product builds to answer:

• Is the candidate actually faster?
• Did tails improve?
• Did total work decrease?

Do not mix diagnostic timing directly into product A/B tables unless both sides carry identical instrumentation and the remaining perturbation is explicitly accepted.

14. Enforce the net-total-work rule

A local reduction is not enough.

Examples of false wins:

Remove descriptor-capacity finish
    ↓
next report/fence waits longer
    ↓
total guest progress unchanged

Move shader compile to worker
    ↓
renderer immediately waits for worker
    ↓
first draw still blocks

Avoid one surface readback
    ↓
consume stale guest bytes
    ↓
faster but incorrect

Reduce xemu CPU work
    ↓
driver performs equivalent validation synchronously
    ↓
wall-clock frame unchanged

A candidate should show where the removed work went and why it is no longer on the critical dependency path.

15. Use before/after processing flows

Every nontrivial optimization PR should show how data/control flow changes.

Example:

Before
Guest draw
  ↓
shader miss
  ↓
synchronous compile
  ↓
draw

After
Guest draw
  ↓
shader ready?
  ├─ yes → specialized draw
  └─ no  → fallback draw + background specialization

The diagram should expose changed ownership, new caches, worker boundaries, synchronization and fallback behavior. A code diff alone rarely communicates those properties clearly.

16. Resource and lifecycle qualification

Performance patches often add hidden long-term costs.

Check:

• Bounded cache/queue sizes.
• Allocation growth.
• Object destruction/eviction.
• Worker stop/join.
• Renderer teardown.
• Persistent-file cleanup and atomic publication.
• Snapshot save/restore.
• Process cleanup after tests.

A faster 60-second benchmark is not acceptable if the design leaks state indefinitely or fails shutdown after several hours.

17. GPU and presentation identity

For Vulkan especially, record:

• Requested adapter identity.
• Actual initialized adapter identity.
• Driver/API version when relevant.
• Presentation path: shared external memory or host copy.
• Resolution scale and present/VSync policy.

A candidate running on another GPU is not an A/B comparison even if the executable and game state match.

18. Record the emulation fidelity boundary

Performance evidence should identify whether the measured path is architectural emulation, a host-service substitute, a compatibility workaround, a fast path, or an incomplete/stubbed device model.

A useful evidence row is:

Field	Example
Guest operation	SMBus byte-data read from SMC
Physical behavior	Timed serial bus transaction into PIC device
xemu behavior	Synchronous QEMU SMBus callback
Omitted work	SCL/SDA bit timing and PIC execution latency
Host-added work	Device callback / QEMU object dispatch
Measurement conclusion	Valid for xemu callback cost; not a physical bus-latency estimate

flowchart LR
    A[Physical hardware path] --> B[Guest-observable contract]
    C[xemu implementation path] --> B
    C --> D[Host-only costs]
    A --> E[Physical-only costs omitted by xemu]

Loading

This prevents two common mistakes: optimizing away a host-only cost that is actually removable, and “correcting” a cheap approximation by adding arbitrary wall-clock delay that does not model the guest-visible hardware state machine.

Source FIXMEs, assertions and no-op register paths should be recorded in benchmark notes when the workload exercises them. An incomplete model can make a candidate look fast simply because the missing hardware behavior is not being executed.

19. Acceptance table pattern

A compact final table should answer the decision without hiding evidence.

Workload	Metric	Reference	Candidate	Improvement	State
PGR2 snapshot	p99 interval	44.2 ms	42.8 ms	+3.17%	PASS/HOLD
Morrowind fixed scene	display writes/s	24.3	25.1	+3.29%	Diagnostic proxy
Full XISO Vulkan	functional records	160/161	160/161	N/A	Same inherited nonpass

The numbers above are an example layout, not current xemu results.

20. Decision vocabulary

Use explicit states:

State	Meaning
PASS	Required correctness/performance gates for the stated scope passed
HOLD	Evidence is incomplete, mixed, or contains a blocker
FAIL	Candidate violated a required gate or established regression
DIAGNOSTIC	Result identifies behavior but is not an acceptance benchmark
HISTORICAL	Result belongs to an earlier source/build and is retained for context

Avoid wording such as “looks faster” when the qualification state is HOLD.

21. Recommended investigation sequence

Reproduce exact problem
    ↓
Profile broad subsystem ownership
    ↓
Instrument hottest causal branch
    ↓
Form one patch hypothesis
    ↓
Add focused correctness oracle
    ↓
Implement minimal behavioral change
    ↓
Run exact build + focused tests
    ↓
Run matched normal A/B cells
    ↓
Check p95/p99/max and total work
    ↓
Run broader XISO / retail / lifecycle gates

This keeps evidence causal and prevents several speculative optimizations from being bundled into one patch whose result cannot be explained.

Detailed lag-debugging workflow

Use Debugging Lag & Hot Paths when the problem is still in the attribution stage. It explains why Release + matching DWARF is required for timing/race work, how diagnostic instrumentation is separated from acceptance timing, and how broad traces are reduced to one causal hypothesis.

Shortcut qualification

Use Hot-Path Shortcuts to identify the mechanism and protected invariants, then Using Hot-Path Shortcuts to select the correct state-hygiene and evidence protocol. A shortcut comparison is invalid when its effective route, reachability, reference path or initial cache/lifetime state is unproven.

Classify the mechanism before choosing the test

Use Hot-Path Shortcuts and its same-binary bypass/reference protocol only for a real guarded bypass. Use Using Performance Controls for caches, resource policies, restart-latched setup and alternate execution strategies.

Related pages

• CPU & vCPU Execution
• Clocks & Timers
• NV2A Graphics Architecture
• OpenGL vs Vulkan
• Vulkan Performance
• Development

Recent performance-method lessons

Shared correctness prerequisites below both roles — PR #138

If a bug affects bytes, state, or control used by both reference and candidate, repair it identically in both roles before measuring the optimization. Record the exact prerequisite commit in both source identities.

Measure downstream work — PR #135

Do not infer fewer uploads from fewer reconciliation calls. Prefer actual uniform writes, staged bytes, descriptor writes, submissions, and waits by reason.

Order sensitivity is not acceptance evidence — PR #134

If A/B order materially changes the result, keep the numbers as diagnostic history but rerun exact-head balanced order before acceptance.

Optional integrations need a true inactive-path control — PR #120

An optional host path can regress performance even when it eventually falls back. Gate initialization on effective use, back off after failed setup, compare against a build where the integration is absent, and test runtime setting transitions.

Attribution is not throughput — PR #133

A measured multi-millisecond wait identifies where time is spent. It does not prove that the wait can be removed or overlapped. Acceptance requires a dependency-preserving candidate that improves fixed guest progress in an uninstrumented build.