Skip to content

Performance Methodology

github-actions[bot] edited this page Sep 13, 2026 · 12 revisions

Performance Methodology

Performance work in xemu must separate measurement, causal attribution, correctness, and acceptance. A patch is not a performance improvement merely because one named function becomes cheaper or one counter decreases.

The required question is:

Did the exact candidate reduce the total dependency that limits useful guest or presented progress, without changing correctness or moving the same work somewhere else?

Measurement model

flowchart TD
    A[Choose exact workload and progress metric] --> B[Pin reference and candidate source/build identities]
    B --> C[Validate functional equivalence]
    C --> D[Run matched performance cells]
    D --> E[Collect CPU / GPU / guest-progress evidence]
    E --> F[Attribute the changed path]
    F --> G{Net improvement established?}
    G -->|No| H[Hold / reject / investigate]
    G -->|Yes| I[Broaden workload and lifecycle validation]
    I --> J[Merge-ready evidence]
Loading

Instrumentation can identify a cause, but acceptance numbers should normally come from builds without intrusive trace overhead unless the instrumentation itself is proven negligible and identical on both sides.

1. Pin source and executable identity

Every result must be tied to an exact source state.

Record at minimum:

Identity Why it matters
Repository + commit SHA Defines source code under test
Base/reference commit SHA Prevents comparing against a moving main
Build profile Debug/Release, optimization, LTO, architecture flags and assertions affect timing
Executable hash Proves the binary deployed to the test system is the intended build
Relevant runtime settings Renderer, resolution scale, cache toggles, VSync, experimental options
GPU/adapter identity Especially required for Vulkan device-selection work
Presentation transport Shared external memory versus host-copy can change cost materially

A PR updated after a benchmark does not inherit the old result automatically. Historical measurements must say which earlier head they qualify.

2. Use two reference concepts

When possible, distinguish:

Previous main

The mainline source immediately relevant to the candidate. This answers:

Does this patch improve what would otherwise ship now?

Fixed/stable baseline

A retained source/build whose behavior is already characterized. This answers:

Did main or the candidate drift relative to a known reference?

The two references answer different questions. If current main has already regressed or improved, a candidate-versus-main comparison alone can hide that history.

3. Pin the workload

A performance cell is not just a game name.

Record:

  • Title/build.
  • Snapshot/save/XISO seed identity where applicable.
  • Camera/view/location or deterministic scripted progression.
  • Duration or progress window.
  • Input sequence.
  • Renderer and cache condition.
  • Any private-disk/EEPROM/settings state that changes behavior.

A fixed Morrowind camera view and a full traversal are different workloads. A PGR2 snapshot and a fresh boot are different workloads. Keep them separate in tables and conclusions.

4. Define the progress metric before testing

Metrics must state what event they actually measure.

Metric What it can establish What it does not automatically establish
Guest frame interval Guest-side frame progress/cadence Presented host FPS
NV2A display writes Guest GPU/display progress marker GPU-completed or presented frames
Report/query cadence Guest-visible report completion Whole-frame render speed
PresentMon / presentation FPS Host-presented output when capture is valid Guest CPU or GPU bottleneck cause
CPU samples Where host CPU time is spent Whether reducing that symbol improves frame time
GPU timestamps GPU execution spans between timestamp points CPU wait or presentation cost outside the timestamp region
XISO duration Suite/test throughput Retail gameplay performance by itself

Never rename a proxy metric into a stronger claim. If the measurement is “guest display writes/s,” publish it as that, not “FPS.”

5. Normalize direction before showing percentages

Reader-facing Improvement % should use positive values for favorable changes.

Higher is better

For FPS, writes/s, operations/s, throughput:

Improvement % = 100 × (candidate / reference - 1)

Lower is better

For frame interval, latency, CPU time, GPU time, allocations, stalls:

Improvement % = 100 × (reference - candidate) / reference

If the direction is ambiguous, the reference is zero, or the metric is not meaningfully ratio-scaled, show the raw values and use N/A rather than forcing a percentage.

6. Control run order

Thermals, OS state, shader/driver caches, filesystem caches and process history can bias sequential tests.

Prefer paired and counterbalanced designs such as:

A → B
B → A

or, when appropriate:

A → B → B → A

where A and B are exact reference/candidate configurations.

If order reversal changes the sign of a small result, treat the result as inconclusive rather than averaging away the instability.

7. Separate cold, warm and same-process reuse

Cache state is part of the workload.

At minimum distinguish:

Condition Meaning
Cold process No same-process renderer cache history
Cold persistent cache No previously persisted artifact relevant to this run
Warm persistent cache Persisted artifacts intentionally reused across launches
Same-process revisit In-memory caches can reuse objects from an earlier encounter

For shader work, a same-process module hit, a persisted SPIR-V hit and a Vulkan driver pipeline-cache hit are different events. Do not combine them into one “shader cache” metric.

8. Report distributions, not only averages

Emulator performance frequently fails in the tail.

For frame/progress intervals, retain at least:

  • Count / duration.
  • Mean or median as appropriate.
  • p95.
  • p99.
  • Maximum.
  • Explicit stall threshold counts when a threshold is defined.

A patch that improves the mean while introducing rare 80–100 ms cold stalls may be unacceptable even if average FPS increases.

9. Profile by thread and subsystem

Separate at least:

  • vCPU thread.
  • Main/event/UI thread where relevant.
  • PFIFO/PGRAPH/renderer work.
  • Worker threads.
  • Host graphics driver threads when attributable.

For CPU work, distinguish generated guest code from emulator helpers/dispatch. For GPU work, distinguish useful draw spans from transfers, pass breaks and waits.

A useful classification tree is:

flowchart TD
    A[Slow interval] --> B{CPU-limited?}
    A --> C{GPU-limited?}
    A --> D{Synchronization/wait-limited?}

    B --> E[vCPU generated guest code]
    B --> F[TB dispatch / helpers / device work]
    B --> G[Renderer CPU preparation]

    C --> H[Useful draw work]
    C --> I[Copies / transitions / clears]

    D --> J[GPU fence / glFinish]
    D --> K[Report/query dependency]
    D --> L[Host scheduling/timer wait]
Loading

The categories can overlap; the point is to locate the dependency rather than optimize whichever symbol is easiest to edit.

10. Instrument the decision, not just the duration

Timers are useful only when accompanied by enough identity to explain what happened.

Examples:

Shader investigation

Record separately:

  • Binding-cache hit/miss.
  • Stage-module hit/miss.
  • Exact generated-source identity.
  • GLSL generation time.
  • glslang compile time.
  • Vulkan module-creation time.
  • Reflection time.
  • Graphics pipeline hit/miss/create time.

Vertex/surface investigation

Record separately:

  • Dirty range.
  • Surface-overlap/freshness decision.
  • Direct versus staged upload.
  • Bytes/pages affected.
  • Render-pass break reason.
  • Submission/fence reason.

Timer investigation

Record separately:

  • Guest target tick/deadline.
  • Converted QEMU-clock deadline.
  • timer_mod / timer_del decisions.
  • Actual callback count.
  • IRQ state transitions.
  • Host wake/oversleep data if host scheduling is under test.

A single aggregate counter cannot prove a mechanism if multiple code paths can increment it.

11. Validate correctness before accepting speed

Performance results are invalid if output changed incorrectly.

Relevant validation layers include:

  • Unit/focused tests for changed logic.
  • Full XISO coverage where the subsystem is exercised.
  • OpenGL/Vulkan differential controls where appropriate.
  • Framebuffer/output hashes where determinism allows them.
  • Guest memory/report results.
  • Vulkan validation/VUID checks.
  • Sanitizers for changed ownership/lifetime code.
  • Snapshot/save/restore when the patch touches derived or persistent state.

Inherited failures must stay visible. Do not turn a pre-existing nonpass into “PASS” merely because candidate and reference match.

12. XISO result discipline

When XISO is used as a correctness gate, retain the individual test records rather than only a suite-level summary.

For each record, capture enough to identify:

  • Test name.
  • Renderer/configuration.
  • Outcome.
  • Functional hash/result where applicable.
  • Validation errors.
  • Source/build identity.

Large per-test tables should normally live in evidence files with a compact summary linked from the PR/wiki. The Wiki should explain what the test proves, not reproduce hundreds of rows.

13. Distinguish diagnostics from acceptance runs

Intrusive tracing changes timing.

Use diagnostic builds to answer questions such as:

  • Why did this frame stall?
  • Which cache missed?
  • Why did a render pass end?
  • Which thread waited?

Then use normal product builds to answer:

  • Is the candidate actually faster?
  • Did tails improve?
  • Did total work decrease?

Do not mix diagnostic timing directly into product A/B tables unless both sides carry identical instrumentation and the remaining perturbation is explicitly accepted.

14. Enforce the net-total-work rule

A local reduction is not enough.

Examples of false wins:

Remove descriptor-capacity finish
    ↓
next report/fence waits longer
    ↓
total guest progress unchanged
Move shader compile to worker
    ↓
renderer immediately waits for worker
    ↓
first draw still blocks
Avoid one surface readback
    ↓
consume stale guest bytes
    ↓
faster but incorrect
Reduce xemu CPU work
    ↓
driver performs equivalent validation synchronously
    ↓
wall-clock frame unchanged

A candidate should show where the removed work went and why it is no longer on the critical dependency path.

15. Use before/after processing flows

Every nontrivial optimization PR should show how data/control flow changes.

Example:

Before
Guest draw
  ↓
shader miss
  ↓
synchronous compile
  ↓
draw

After
Guest draw
  ↓
shader ready?
  ├─ yes → specialized draw
  └─ no  → fallback draw + background specialization

The diagram should expose changed ownership, new caches, worker boundaries, synchronization and fallback behavior. A code diff alone rarely communicates those properties clearly.

16. Resource and lifecycle qualification

Performance patches often add hidden long-term costs.

Check:

  • Bounded cache/queue sizes.
  • Allocation growth.
  • Object destruction/eviction.
  • Worker stop/join.
  • Renderer teardown.
  • Persistent-file cleanup and atomic publication.
  • Snapshot save/restore.
  • Process cleanup after tests.

A faster 60-second benchmark is not acceptable if the design leaks state indefinitely or fails shutdown after several hours.

17. GPU and presentation identity

For Vulkan especially, record:

  • Requested adapter identity.
  • Actual initialized adapter identity.
  • Driver/API version when relevant.
  • Presentation path: shared external memory or host copy.
  • Resolution scale and present/VSync policy.

A candidate running on another GPU is not an A/B comparison even if the executable and game state match.

18. Acceptance table pattern

A compact final table should answer the decision without hiding evidence.

Workload Metric Reference Candidate Improvement State
PGR2 snapshot p99 interval 44.2 ms 42.8 ms +3.17% PASS/HOLD
Morrowind fixed scene display writes/s 24.3 25.1 +3.29% Diagnostic proxy
Full XISO Vulkan functional records 160/161 160/161 N/A Same inherited nonpass

The numbers above are an example layout, not current xemu results.

19. Decision vocabulary

Use explicit states:

State Meaning
PASS Required correctness/performance gates for the stated scope passed
HOLD Evidence is incomplete, mixed, or contains a blocker
FAIL Candidate violated a required gate or established regression
DIAGNOSTIC Result identifies behavior but is not an acceptance benchmark
HISTORICAL Result belongs to an earlier source/build and is retained for context

Avoid wording such as “looks faster” when the qualification state is HOLD.

20. Recommended investigation sequence

Reproduce exact problem
    ↓
Profile broad subsystem ownership
    ↓
Instrument hottest causal branch
    ↓
Form one patch hypothesis
    ↓
Add focused correctness oracle
    ↓
Implement minimal behavioral change
    ↓
Run exact build + focused tests
    ↓
Run matched normal A/B cells
    ↓
Check p95/p99/max and total work
    ↓
Run broader XISO / retail / lifecycle gates

This keeps evidence causal and prevents several speculative optimizations from being bundled into one patch whose result cannot be explained.

Related pages

Clone this wiki locally