Repository navigation
Debugging Lag and Hot Path Analysis
This page documents the performance-debugging workflow that has proven useful in the xemu fork when a game is visibly stuttering, guest progress is irregular, a thread appears saturated, or a profiler shows an unexpectedly hot emulator path.
The central lesson is that an emulator hot path is not automatically wasted work. A high sample count can represent useful guest execution, a symptom of an upstream state transition, a host-driver dependency, synchronization that moved from somewhere else, or instrumentation overhead. The job is to move from symptom → ownership → cause → minimal experiment → normal Release validation without changing the timing behavior so much that the problem disappears.
This is why most serious timing/race investigations in this fork use an optimized Release build with matching DWARF information, not a conventional Debug build.
A performance investigation should answer four separate questions:
-
What does the guest actually experience?
- a long guest frame interval,
- lost display-write cadence,
- a stalled report/query,
- a transition freeze,
- a timer/poll cadence problem,
- or a live lifecycle failure.
- Which thread or subsystem owns the dependency when the symptom occurs?
- Why did that subsystem do the expensive or blocking work?
- Does changing it improve normal Release execution without breaking correctness?
The workflow is deliberately staged:
flowchart TD
A[Reproduce lag in optimized Release] --> B[Capture broad profile with matching DWARF]
B --> C[Find hot or blocked thread]
C --> D[Resolve stack and source ownership]
D --> E[Correlate with guest progress or slow frame]
E --> F[Add narrow counters or bounded trace]
F --> G[Classify the causal branch]
G --> H[Build one minimal hypothesis patch]
H --> I[Disable intrusive tracing]
I --> J[Matched normal Release A/B]
J --> K[Correctness / XISO / retail / lifecycle gates]
Do not collapse those stages into one instrumented benchmark. Diagnostic builds answer why. Normal product-style Release builds answer whether the candidate is actually better.
The investigation flow on this page is intentionally tool-agnostic at the decision level. The practical command reference lives in Debugging Tools & Commands.
Use the tools by question, not by habit:
| Investigation stage | Primary tools |
|---|---|
| Capture optimized runtime behavior | Windows: WPR / ETW; Linux: perf record
|
| Inspect CPU, scheduling, waits and GPU intervals | Windows: WPA/WPAExporter/xperf; Linux: perf sched, BCC/eBPF, Perfetto |
| Check syscall/driver waits on Linux |
strace against the selected TID |
| Resolve optimized addresses to source | matching DWARF + addr2line
|
| Inspect exact generated machine code | objdump |
| Recover source paths/call sites |
rg + source inspection |
| Separate remaining causal hypotheses | bounded counters + Python analyzers |
| Prove artifact identity/correctness | Git/SHA-256 + hash/oracle tools |
The tools page includes the common commands, expected output, and interpretation traps. Keep this page focused on when and why each stage is used.
For Linux and Steam Deck, use the production-oriented Steam Deck and Linux Thread Diagnosis sequence. It separates running, blocked, and runnable-delay time before escalating to optional tools.
Debug and heavily instrumented builds are useful for assertions, state inspection, and some deterministic correctness bugs. They are often the wrong environment for emulator timing, races, queue pressure, and scheduling problems.
A Debug/unoptimized build can change all of the following at once:
- instruction count and code layout;
- inlining and call boundaries;
- lock hold duration;
- relative speed of the vCPU, renderer, UI, audio, and worker threads;
- queue fill/drain rates;
- timer servicing cadence;
- command submission cadence to the graphics driver;
- compiler worker completion timing;
- host scheduler decisions;
- memory layout and cache behavior.
A race that occurs in the optimized product can therefore disappear, move, or turn into a different failure under Debug. The inverse is also true: a Debug-only timing failure can be created by the diagnostic environment itself.
For a timing-sensitive investigation, the useful model is:
normal optimized product behavior
+ enough assertions to catch invalid state where practical
+ matching DWARF/debug information for source attribution
+ only the minimum diagnostic instrumentation needed for the current question
Historical fork campaigns have used Windows O2, often full LTO and x86-64-v3, while retaining symbols/assertions for investigation builds. PR #55, for example, explicitly qualified an uninstrumented candidate as Windows O2/full-LTO/x86-64-v3 with symbols/assertions retained. Current PR #71 similarly records optimized Windows product builds with assertions/DWARF for cold-path investigation.
This distinction is important:
DWARF does not make a build a Debug build.
DWARF is metadata that lets tools map machine addresses back to functions, inlined call chains, files, and source lines. Optimization remains enabled. The machine code still has Release-style inlining, register allocation, code motion, LTO behavior, and timing characteristics.
That gives us the combination we need:
flowchart LR
A[Release optimization / LTO] --> C[Representative runtime behavior]
B[Matching DWARF] --> D[Source attribution]
C --> E[Useful timing investigation]
D --> E
Do not replace this with a Debug binary merely because its symbols are easier to inspect.
A sampled profile usually begins with instruction addresses and stack addresses. Source-level conclusions require a symbol/debug artifact produced from the same build.
Keep these together as one identity:
source commit SHA
+
exact Release executable
+
executable SHA-256
+
matching DWARF-bearing/debug artifact
+
compiler / optimization / LTO / ISA settings
+
profile trace
If the executable changed, the addresses can move even when the source change looks harmless. LTO and inlining make this especially important. A stale debug artifact may resolve an address to a plausible but completely wrong function.
Never symbolize executable A with debug information from executable B.
If matching symbols are unavailable, record the profile as address/module-level evidence only. Do not make source-line claims from guessed or mismatched symbols.
For MinGW/DWARF investigations, an address-resolution pass can use the matching DWARF-bearing executable/artifact with addr2line. A representative pattern used in this work is:
x86_64-w64-mingw32-addr2line -a -f -i -C \
-e <matching-dwarf-artifact> \
<sample-address-1> <sample-address-2> ...
The exact artifact path and addresses come from the captured build/profile. The important part is not the command spelling; it is build identity.
Optimized code may inline several logical functions into one sampled address. Use symbolization that preserves inlined call information when available. Otherwise a hot helper can be attributed to the wrong logical owner.
A trace without enough identity to reproduce it becomes historical noise very quickly.
For every serious lag capture, record at least:
| Category | Record |
|---|---|
| Source | exact source/head SHA |
| Executable | SHA-256 or other immutable build identity |
| Symbols | matching DWARF/debug artifact identity |
| Build | Release/O2, LTO state, ISA target, assertions, diagnostic flags |
| Renderer | OpenGL or Vulkan |
| GPU | requested adapter and actual initialized adapter |
| Driver/API | relevant driver/API version |
| Presentation | shared external-memory or host-copy path where relevant |
| Display | resolution scale, VSync/present policy |
| Workload | game/XISO/snapshot and exact scene/seed |
| Lifecycle | fresh boot, startup snapshot load, or live restore |
| Cache state | cold, warm, same-process, restart |
| Storage | private test disk/seed identity when relevant |
| Duration | capture/test window |
| Instrumentation | exactly what counters/rings/traces are enabled |
This prevents several false comparisons that have already appeared during xemu investigations: different GPUs, different snapshot paths, cold versus warm shader state, old snapshot sources, and diagnostic builds being compared directly to normal builds.
The profiler should not define the bug. The guest-visible symptom defines the bug.
Examples of useful progress measurements include:
- guest display writes per second;
- distribution of guest display-write intervals;
- a particular report/query completing;
- a game checkpoint or deterministic transition completing;
- a fixed number of guest frames or operations;
- a known XISO result/hash;
- a snapshot restoring and resuming guest execution.
Host-presented FPS is useful, but it is not automatically equivalent to guest GPU or VM progress. Presentation, driver buffering, VSync, and backend differences can hide where the guest actually stalled.
These are not interchangeable:
fresh boot
startup -loadvm
live snapshot restore
same-process warm replay
cold process launch
XISO test
retail fixed scene
PR #92 demonstrated this directly. An earlier startup -loadvm comparison was excluded because startup loading bypassed the live RUN_STATE_RESTORE_VM callback that the experiment needed to exercise. The live PGR2 restore instead exposed a wait for a UI-held framebuffer.
flowchart LR
A[Startup -loadvm] --> B[Startup restore path]
C[Running VM] --> D[Live restore callback]
D --> E[Renderer/UI ownership interaction]
B -. not equivalent .-> E
If the bug is lifecycle-specific, a faster reproduction path that bypasses that lifecycle is not a valid test.
The first capture should answer where time or blocking lives with as little perturbation as possible.
Useful ownership domains in xemu include:
| Domain | Typical questions |
|---|---|
| vCPU / TCG | Is the guest executing useful code, returning to dispatch, translating, invalidating, or taking helpers? |
| PFIFO / PGRAPH | Is guest command processing or state translation the owner? |
| renderer CPU | Shader, pipeline, descriptor, surface, texture, vertex, report, or command recording work? |
| GPU/driver | Is xemu waiting on submitted GPU work or driver compilation? |
| UI/presenter | Is a framebuffer/window/presentation resource held? |
| timers/main loop | Who requested the next deadline and why is the loop waking? |
| MCPX APU | Is audio DSP/worker pacing or host audio backpressure involved? |
| network/storage | Is a device/backend I/O path blocking progress? |
A thread using nearly a full core can be important without being the critical bottleneck. A thread with low CPU usage can own the blocking dependency that determines the frame.
Suppose a profile says:
vCPU: ~100% of one core
renderer: moderate CPU
GPU: active
That does not prove “optimize the vCPU.” The guest might genuinely require that CPU work while the visible hitch is caused by a 40 ms renderer dependency once every few seconds.
Likewise, a renderer function with 20 ms of CPU work may be off the critical path if a worker already overlaps it successfully.
The next question is always:
What dependency prevented guest progress during the lag event?
Whole-run totals are good for finding broad ownership. They can hide short, severe stalls.
A useful workflow is:
- identify the guest-visible slow interval;
- locate the same time window in CPU/GPU/scheduler traces;
- inspect stacks and waits in that window;
- compare it with adjacent normal frames;
- then add focused instrumentation to distinguish the remaining branches.
For cold hitches, retain p95/p99/max and explicit stall thresholds. A mean can improve while the cold tail gets worse.
For example, a 30-second run with one 85 ms stall can look nearly unchanged in the average while still feeling worse to the player.
One earlier profile showed the vCPU consuming nearly a complete host core. Rough attribution put a large portion in generated guest execution and another large region in translation-block dispatch/lookup. tlb_reset_dirty_range_all also appeared prominently in graphics-thread investigation context.
The wrong conclusion would have been:
“TB dispatch is hot, so make the dispatcher return less.”
The correct question was:
“Why are translation blocks returning to the dispatcher?”
flowchart LR
A[Profile: dispatcher/lookup hot] --> B{Why did the TB return?}
B --> C[Normal chain boundary]
B --> D[Interrupt / exception]
B --> E[MMIO / helper]
B --> F[TB invalidation / dirty-TLB work]
B --> G[Debug / halt / external kick]
B --> H[Architecture-required boundary]
C --> I[Measure exit classes]
D --> I
E --> I
F --> I
G --> I
H --> I
PR #54 added diagnostic-only named counters and periodic return sampling. It explicitly rejected using the instrumented build for performance comparison because the counters/sampling changed execution cost.
Its result was highly useful but narrow:
- 39,262 sampled NULL returns in the Vulkan capture;
- 14,289 in the OpenGL capture;
- more than 99% of the sampled NULL-return leaves were attributed to the STI/shadow pair.
That established frequency attribution, not exclusive CPU time and not a speedup.
The lesson:
A hot dispatcher is a symptom until its return classes are measured.
PR #55 then removed the diagnostic counters and tested a small production change: chain STI into the translated block carrying the interrupt shadow while preserving the mandatory post-shadow return.
The candidate was tested as an optimized Windows O2/full-LTO/x86-64-v3 build with symbols/assertions retained.
The result stayed HOLD:
- one dispatcher transition was removed on the safe path;
- one focused OpenGL observation moved favorably;
- Vulkan was mixed/near neutral;
- native interrupt/debug/fault correctness remained unqualified;
- repeatable fixed-work host CPU/GPU evidence was absent.
This is the correct progression:
flowchart TD
A[Broad Release profile] --> B[Dispatcher appears hot]
B --> C[Diagnostic Release counters]
C --> D[STI/shadow dominates sampled returns]
D --> E[Archive instrumentation]
E --> F[Uninstrumented minimal STI chaining candidate]
F --> G[Correctness + matched Release performance]
G --> H[HOLD: benefit/correctness not established]
Do not carry diagnostic counters into the benchmark just because they gave us the hypothesis.
PR #68 is a strong example of why “the profiler says shaders” is not enough.
A Vulkan draw can encounter several different cache/compile events:
flowchart TD
A[Guest draw] --> B[Shader binding lookup]
B --> C[Stage module lookup]
C --> D[Exact generated source identity]
D --> E[glslang compile]
E --> F[VkShaderModule creation]
F --> G[Reflection / layout work]
G --> H[Graphics pipeline lookup/create]
H --> I[Draw can proceed]
A graphics-pipeline miss is not the same event as a shader compile. A binding miss is not automatically a stage-module miss. Two different keys can even map to the same generated source.
Historical PR #68 measurements made those layers visible:
-
pgraph_vk_bind_shaders: roughly 23.906–25.034 ms in the measured path; -
vkCreateGraphicsPipelines: roughly 12.208–13.538 ms; - pipeline key/hash/lookup/layout: below roughly 1.1 ms combined;
- 141 stage-module misses;
- 121 first-seen stage/source identities;
- 20 different-key/same-source repeats;
- about 463.419 ms aggregate glslang work;
- about 68.200 ms of duplicate-source glslang work.
For the reproduced primary hitch:
- guest interval: 59.493 ms;
- pipeline preparation: 43.738 ms;
- two vertex + two fragment module misses;
- all four were first-seen;
- glslang: 18.217 ms;
- module creation: 0.043 ms;
- reflection: 1.269 ms.
That changed the design direction.
PR #70 persisted validated SPIR-V artifacts across runs. That attacks known source already compiled previously, while recreating device-owned Vulkan objects for the new renderer lifetime.
It does not solve a genuinely unseen shader source.
PR #71 explores a fail-closed ubershader/hybrid route for newly encountered admitted fragment-combiner states. The current reviewed design keeps:
- immutable/deep-owned worker input;
- bounded speculative and required work;
- renderer-thread publication/cache mutation;
- synchronous specialized fallback for unsupported/unready state;
- complete pipeline readiness before switching route.
Diagnostic work found another important lesson. A route-selection path was called roughly 3,078 times per guest frame while only about 504 bindings changed. An effective-state fast return reduced the probes to roughly 503/frame and reduced measured route CPU substantially. Yet total Hybrid-On shader binding remained around 4.54 ms/frame in the compared diagnostic capture.
So both statements can be true:
local hot path became much cheaper
AND
end-to-end frame behavior did not materially improve
The current PR remains HOLD because cold-tail attribution and exact-head retail qualification are not complete.
Lesson: after optimizing a measured leaf, re-profile the parent and re-measure guest progress. The bottleneck can simply move or reveal the next layer.
A GPU trace from the vertex-upload investigation reported approximately:
| Region | Time / count |
|---|---|
| Guest display writes | 243 |
| Main GPU time | 2317.986 ms |
| Inside render passes | 1711.548 ms / 73.84% |
| Ordered copies | 375.501 ms / 16.20% |
| Other | 9.96% |
| Render passes | 61,718 |
| Copies | 57,702 |
The tempting conclusion is:
“73.84% is in render passes, so render passes contain 73.84% optimization opportunity.”
That is wrong.
Most time inside render passes is useful rendering. The diagnostic value was that it told us where GPU time was located and motivated questions about why passes were ending around ordered vertex updates.
Useful questions became:
- Did this vertex update force a render-pass break?
- Was the affected page already consumed by an earlier draw recorded in the active command buffer?
- Is guest RAM still authoritative, or can a GPU-authored surface overlap the fetched range?
- Can the CPU write the mapped vertex mirror directly without violating command-buffer ordering?
- Does page-read tracking cost more than the copies it avoids for this workload?
The safety rule that emerged is:
Direct mapped vertex write during an active batch
is allowed only when
no earlier recorded draw can observe the overwritten bytes
AND
guest/source freshness is valid for the fetched range.
Workload behavior was dramatically different:
- PGR2 marked roughly 3.81 million pages but produced only about 62 direct copies in one historical observation;
- Morrowind marked roughly 429,090 pages and produced about 27,649 direct copies.
That means the bookkeeping itself can be a net loss in one game and useful in another. PGR2 p99 results on the experimental branch were adverse enough to keep the work on HOLD.
Lesson: profiler category size is not the same as removable work, and fast-path hit rate must be measured on representative workloads.
Timer work requires another distinction:
guest-visible timer semantics
versus
host scheduling / polling implementation
For NV2A PTIMER the conceptual path is:
flowchart LR
A[PTIMER counter/alarm state] --> B[Convert to virtual deadline]
B --> C[QEMU timer / main-loop deadline]
C --> D[Host wait / wakeup]
D --> E[Timer callback]
E --> F[Pending state / IRQ]
F --> G[Guest observes timer event]
A host wait optimization must not silently change the counter/alarm phase, masking semantics, PLL/clock transition behavior, interrupt ordering, or guest-visible deadline.
PR #59 and later PR #81 reinforced this rule: explicit deadline ownership can make the implementation easier to reason about, but fewer timer operations or more precise host waiting are not performance results by themselves.
When a trace shows frequent short wakeups, ask:
- who requested the deadline?
- was the deadline guest-required or emulator-created?
- did the wakeup perform useful guest progress?
- is the main loop spinning because deadlines are repeatedly recomputed?
- does eliminating wakeups merely move the wait to another synchronization point?
Test both renderers and retail workloads before treating a timer change as general improvement.
Logging is code. In a concurrent emulator it changes timing.
A printf or formatted trace inside a hot path can:
- serialize threads on I/O locks;
- lengthen a critical section;
- change worker/main-thread ordering;
- make a queue empty earlier or later;
- alter GPU submission cadence;
- hide a race by adding delay;
- create a new race window that product code does not have.
This is why the preferred diagnostic escalation is:
flowchart TD
A[Normal Release + DWARF sampling] --> B[Cheap counters]
B --> C[Conditional / sampled events]
C --> D[Bounded in-memory ring]
D --> E[Slow-frame-only detailed record]
E --> F[Offline dump after event]
Prefer:
- integer counters over formatted text;
- per-thread or low-contention counters where possible;
- bounded rings instead of unbounded logging;
- only recording the slow frame or suspicious branch;
- environment/config gates defaulting Off;
- deeply owned/immutable trace data where asynchronous workers are involved;
- dumping after the critical interval, not during it.
| Build | Purpose | Timing claim? |
|---|---|---|
| Normal optimized Release + DWARF | broad sampling / stack attribution | Yes, if no intrusive instrumentation |
| Diagnostic optimized Release | classify branch/cause | No, unless both sides have identical accepted instrumentation and perturbation is explicitly tolerated |
| Minimal candidate Release | prove hypothesis and focused correctness | Preliminary |
| Normal candidate Release | final matched A/B | Yes |
PR #54 is the model: diagnostic counters answered the causal question, then the branch was archived instead of being benchmarked as production code.
The exact capture tool may change, but the workflow does not. ETW/WPA/xperf-style sampling is useful because it can expose CPU stacks, scheduling, waits, process/thread ownership, and timing without compiling printf-style logging into every path.
Use the optimized Release configuration that reproduces the lag/race. Retain or produce matching DWARF information. Record the source SHA and executable hash.
Do not switch to Debug merely to get source names.
Store or record:
xemu executable
matching DWARF/debug artifact
source SHA
executable SHA-256
build flags/profile
renderer/config
Choose a workload that reproduces the problem reliably and define the guest-progress metric before starting the capture.
Examples:
- 60-second fixed snapshot scene;
- fresh PGR2 boot window;
- Morrowind fixed snapshot;
- exact XISO case;
- live snapshot restore sequence.
Start with enough information to answer:
- which thread is running?
- which thread is blocked?
- what stack owns the CPU samples?
- where does a wait begin/end?
- does the hot interval align with the guest-visible stall?
Avoid enabling every tracing provider simply because it exists.
Unresolved addresses are not a reason to guess. Fix symbol coverage first.
If the profiler gives raw PE addresses, normalize them according to the profiler/module representation and resolve them against the matching artifact. Preserve the raw addresses in evidence so resolution can be repeated.
A global function ranking loses dependency context. Look at both:
- whole-run ownership;
- the specific lag window.
Ask what appears only in the slow interval:
- first-seen compile?
- cache miss?
- large readback?
- fence wait?
- timer spin?
- command-buffer rollover?
- snapshot lifecycle ownership?
At this point instrumentation should answer a specific question such as:
Which cache missed?
Why did this render pass end?
Why did this TB return?
Who requested this deadline?
Was this surface actually newer than guest RAM?
Did the worker finish before the renderer needed the result?
If the instrumentation cannot answer a named question, it is probably too broad.
Rebuild the normal candidate and rerun matched Release A/B cells.
| Symptom | Broad profile clue | Next question | Likely layer |
|---|---|---|---|
| vCPU nearly one core | generated code + TB dispatch | Why are TBs returning? | TCG/vCPU |
| renderer CPU spike | shader/pipeline/surface/vertex function | First-seen work, cache miss, coherency, or driver call? | renderer |
| low xemu CPU + long guest interval | fence/wait/present | Which dependency is blocking the guest? | GPU/driver/presenter |
| periodic cadence problem | short timer/poll wakeups | Who requested the deadline? | PTIMER/main loop/device timer |
| Vulkan-only hitch | OpenGL control clean | Vulkan translation, pipeline, descriptor, barrier, or driver? | Vulkan backend |
| GPU-specific regression | changes with adapter | Did requested and initialized GPU/presenter path match? | host graphics configuration |
| snapshot-only failure | startup load works | Is this live restore ownership/quiesce? | VMState/UI/renderer/APU |
| same-process gets faster | warm cache effect | Does a fresh process reproduce the benefit? | cache/initialization |
| profiler leaf got cheaper but frame did not | parent unchanged | Where did the work/dependency move? | next parent/layer |
Call-count reduction is useful evidence only if the removed calls were on the critical dependency path.
These are common false wins:
fewer finish() calls
↓
later fence wait grows
↓
total guest progress unchanged
move compile to worker
↓
renderer immediately waits for worker
↓
first draw still blocks
skip readback
↓
CPU consumes stale bytes
↓
faster but incorrect
reduce emulator CPU work
↓
host driver performs equivalent synchronous validation
↓
wall-clock frame unchanged
Always re-measure the parent dependency and guest progress after changing a leaf.
A strong investigation accumulates evidence in this order.
Release reproduction
↓
Release + matching DWARF sampling
↓
thread/subsystem attribution
↓
narrow counters/ring
↓
causal branch classification
focused unit/regression tests
↓
assertions / sanitizers where applicable
↓
XISO / renderer validation
↓
output/hash/guest-memory checks
↓
lifecycle/save/restore where affected
normal Release reference
↕ counterbalanced/interleaved order
normal Release candidate
↓
mean + p95 + p99 + max + stall counts
↓
fixed-work host CPU/GPU/resources
↓
retail / both renderers / required lifecycle gates
A result can be excellent diagnostic evidence and still be invalid as acceptance evidence.
If the production symptom depends on thread timing, queue pressure, or host scheduling, Debug can remove the reproduction. Use optimized Release plus matching symbols.
A plausible symbol name from the wrong build is worse than an unresolved address because it creates false confidence.
“73.84% of GPU time is inside render passes” does not mean 73.84% is overhead.
Single cells can identify a bug or show feasibility. They rarely prove a small performance delta.
Warm shader/pipeline/resource caches answer a different question.
Measure total guest progress and the next synchronization boundary.
PR #54 explicitly avoided this. Follow that model.
Record requested and actual device identity and presentation mode.
PR #92 showed why lifecycle path identity matters.
Calls are a mechanism metric. Guest progress/wall time is the result metric.
A cache, direct write, chaining rule, or skipped synchronization that changes guest-visible behavior is not an optimization.
If reference and candidate both fail a test, record it as an inherited/shared nonpass. Do not rename it PASS.
A debugging handoff should let another engineer continue without rerunning the entire investigation just to understand it.
Record:
| Field | Required content |
|---|---|
| Symptom | exact guest-visible lag/freeze/cadence issue |
| Reproduction | workload, lifecycle, renderer, cache state |
| Build identity | source SHA, executable hash, Release settings, matching DWARF |
| Progress metric | what defines guest progress |
| Broad owner | vCPU, renderer, GPU, timer, UI, APU, etc. |
| Hot/blocking stack | resolved functions/source with time window |
| Why the hot path executes | cache miss, return class, coherency, wait owner, etc. |
| Instrumentation | exact counters/ring and why it was needed |
| Causal result | what the diagnostic proved and what it did not prove |
| Candidate flow | before/after dependency flow |
| Correctness gates | tests required before timing matters |
| Acceptance | normal Release A/B results and tails |
| Status | PASS / HOLD / FAIL / DIAGNOSTIC / HISTORICAL |
The phrase “function X is hot” is not a complete diagnosis.
Use this sequence when handing a new lag issue to an agent:
1. Reproduce the player-visible symptom on normal optimized Release.
2. Freeze source/build/workload identity.
3. Produce/retain matching DWARF without disabling Release optimization.
4. Capture broad CPU/scheduler/GPU evidence with minimal perturbation.
5. Resolve stacks and identify the owning thread/subsystem.
6. Correlate samples/waits with the actual slow guest interval.
7. Write down 2–3 possible causes for that owner.
8. Add the smallest counter/ring needed to separate those causes.
9. Run diagnostic Release and classify the branch/event.
10. Archive diagnostic instrumentation if its job is complete.
11. Implement one minimal production hypothesis.
12. Run focused correctness/oracle tests first.
13. Profile again to confirm the targeted work/dependency changed.
14. Disable intrusive tracing and build normal Release reference/candidate.
15. Run matched/counterbalanced A/B cells.
16. Compare guest progress, p95/p99/max/stalls, and fixed-work resources.
17. Run XISO/retail/backend/lifecycle gates required by the touched subsystem.
18. Record the durable lesson even if the candidate is rejected.
This keeps profiling causal. It also turns negative experiments into useful future-work blockers instead of discarded effort.
After Release + DWARF attribution identifies a candidate path, consult Hot-Path Shortcuts before building another experiment. When a maintained control exists, use Using Hot-Path Shortcuts and prove both routes were reached.
After finding a hot path, classify the proposed change with Performance Optimization Catalog. Transient-buffer growth, prepared layouts and exact lookup tables can be good optimizations without being shortcuts.
Issue #106 demonstrated a common attribution trap: the host OpenGL presentation loop can repeatedly present one completed guest image and report thousands of host presents per second. That value is not guest FPS, guest flips, or useful Vulkan renderer throughput.
For correctness or performance experiments, use a guest-visible clock:
guest flip or frame sequence
fixed guest instruction/progress marker
exact transition admission point
in-process renderer event associated with guest work
For graphics corruption, capture both the renderer-produced guest/display image and the final host window. See Graphics Transition Debugging Tools and PGR2 Race-to-Menu Corruption — Lessons Learned.
- Performance
- Performance Methodology
- CPU & vCPU Execution
- Clocks & Timers
- Vulkan Performance
- Shader Binding & Compilation
- Vertex Uploads & Render-Pass Lifetime
- Pull Request Lessons
- PR #54 — diagnostic attribution of TCG return classes; instrumentation intentionally not a performance candidate.
- PR #55 — uninstrumented STI/shadow production hypothesis; held because correctness and repeatable benefit were not established.
- PR #68 — separated shader binding, module/source compilation, and graphics-pipeline misses.
- PR #70 — persistent validated SPIR-V for warm known shaders.
- PR #71 — hybrid/ubershader research for first-seen admitted states; current exact-head performance remains HOLD.
- PR #81 — explicit PTIMER deadline research; fewer host timer operations are not sufficient evidence.
- PR #87 — vertex copy/render-pass investigation; workload-specific hit rate and ordering constraints dominate the decision.
- PR #92 — negative live snapshot renderer-reinit experiment that exposed UI/framebuffer ownership instead of proving a renderer reset fix.
After a Release trace identifies a vCPU-heavy block, use vCPU Shortcut Candidates to map guest instructions to helper calls, st80f/ld80f, LDMXCSR, allocator spills, or per-element memory work. A candidate is worth implementing only when the eligible dynamic expansion is visible.
DSP tooling guide and navigation updated November 2, 2026 for experimental PR #384, source 8f51d237. Capture/replay usage and limits.
The earlier wiki-wide review was published November 1, 2026. Emulator source: b7140848. Reviewed wiki content: 1d2b671. Current status · Release availability · Publication process.
- Home
- Current status
- Stable versus main
- Implemented mechanisms
- Controls and defaults
- Measured results and limits
- Complete change register
- System
- CPU and vCPU
- Memory and DMA
- Graphics / NV2A
- APU and audio
- Timing and timers
- Platform and I/O
- Hardware observations
- Fidelity boundaries
- Methodology
- PGR2 routes
- Debugging lag / hot paths
- Tools and commands
- DSP capture, replay and fuzzing
- HLE readiness and public XISO proof
- Concurrent DSP shadow and OpenAL proof
- DSP HLE operation inventory
- DSP HLE fix examples and testing
- Development
- Upstream shortlist
- Wiki maintenance
- Research / retired work
- Optimization catalog
- Hot-path shortcuts
- Researched experiments
- Unproven proposal directory
- PR lessons
- September26 snapshot
Repository · PRs · Issues