Skip to content

Debugging Tools and Commands

github-actions[bot] edited this page Sep 14, 2026 · 10 revisions

Debugging Tools & Commands

This page is the practical tool reference for xemu performance, stutter, timing, race, and hot-path investigations.

Use Debugging Lag & Hot Paths for the investigation process. Use this page when you need to know which tool to run, what command to use, and how to interpret its output.

The tools here are intentionally separated by what question they answer. A profiler can tell us where samples landed; it cannot by itself prove why a thread returned, why a fence completed late, or whether an optimization helped the guest.


Tool-selection cheat sheet

Question Primary tool Typical output What it can establish What it cannot establish alone
Which thread/subsystem owns a lag interval? WPR / ETW + WPA sampled CPU, scheduling, GPU intervals thread activity and timing partition source-level cause
Is the process CPU-heavy or waiting? WPA / xperf samples, context switches, wait/ready intervals scheduled vs off-CPU behavior exact wait dependency
What source line owns a sampled address? addr2line + matching DWARF function, file, line, inline chain source attribution for the exact build runtime causality
What function contains an address when DWARF is unavailable? objdump + exact PE symbol map sections, symbols, disassembly bounded function ownership inline call chain or source line
Which source contains the resolved symbol or state transition? rg / source inspection file and line matches static implementation location proof that path executed
Why are TCG lookups/returns occurring? diagnostic counters + analyze_cause_counters.py causal counts/rates frequency attribution performance improvement
Are baseline/candidate outputs still equivalent? hash_compare.py / oracle tools PASS/FAIL hashes deterministic result equivalence for eligible records visual correctness outside the hashed contract
Is an exported WPA dataset complete enough to trust? invoke-wpa-export.ps1 + manifests CSV hashes, row counts, error markers bounded export integrity absence of every possible exporter omission
Can a saved ETL reproduce the accepted analysis? derive-wpa-profile.py, sanitize-attribution.py, analyze-sanitized.py compact CSV/JSON summaries repeatable bounded attribution new runtime evidence
Does the binary really match the reviewed source/build? Git + SHA-256 commit/tree/executable hashes artifact identity correctness/performance
flowchart LR
    A[Release xemu + matching DWARF] --> B[WPR / ETW capture]
    B --> C[WPA / xperf inspection]
    C --> D[addr2line / objdump]
    D --> E[rg + source inspection]
    E --> F[Minimal diagnostic counters if needed]
    F --> G[Python attribution tools]
    G --> H[Uninstrumented Release candidate]
    H --> I[Hash/oracle + performance validation]
Loading

1. Artifact identity tools

Before interpreting a trace, establish exactly which executable produced it and which symbol file belongs to that executable.

A source commit alone is insufficient. Compiler version, optimization, LTO, target ISA, link order, and stripping can all change the executable while the source SHA stays the same.

Git

Useful identity commands:

git rev-parse HEAD
git rev-parse HEAD^{tree}
git status --short
git diff --stat <baseline>..<candidate>
git diff --name-only <baseline>..<candidate>

Read the output

  • HEAD identifies the exact commit checked out.
  • HEAD^{tree} identifies the source tree independent of commit metadata.
  • A clean git status --short means the build did not silently include uncommitted source changes.
  • git diff --name-only is useful when a supposedly narrow experiment unexpectedly touches unrelated implementation files.

Do not write simply main or release build into evidence. Record the exact commit.

SHA-256

On Linux/MSYS:

sha256sum xemu.exe xemu.debug

On Windows PowerShell:

Get-FileHash .\xemu.exe -Algorithm SHA256
Get-FileHash .\xemu.debug -Algorithm SHA256

Record the hashes with the trace/report.

Read the output

A SHA-256 match establishes byte-for-byte identity. A mismatch means the binary or symbol artifact is different even when its filename is identical.

A trace should therefore carry at least:

source commit
source tree
xemu.exe SHA-256
matching DWARF/debug artifact SHA-256
compiler/toolchain identity
optimization/LTO/ISA settings

2. Release build + separate DWARF

For timing-sensitive xemu investigations, use an optimized Release-equivalent build and retain debug information separately.

DWARF does not make the executable a Debug build. It allows optimized machine addresses to be mapped back to source.

If the build pipeline already creates the stripped executable and matching debug file, use that path. A manual GNU-binutils split looks like this:

x86_64-w64-mingw32-objcopy --only-keep-debug xemu.exe xemu.debug
x86_64-w64-mingw32-strip --strip-debug xemu.exe
x86_64-w64-mingw32-objcopy --add-gnu-debuglink=xemu.debug xemu.exe

Hash the final stripped executable and the retained debug artifact after the split.

Why this matters

Debug/unoptimized binaries can materially change:

  • thread timing;
  • lock hold time;
  • worker completion order;
  • shader/pipeline compile overlap;
  • queue pressure;
  • TCG dispatch behavior;
  • renderer/vCPU races;
  • snapshot/lifecycle ordering.

If the bug only reproduces under optimized timing, a Debug build may make the investigation disappear or move somewhere else.


3. Windows Performance Recorder — WPR

WPR is our primary capture tool on Windows. It records ETW events into an .etl trace for later inspection.

Official reference: https://learn.microsoft.com/windows-hardware/test/wpt/wpr-command-line-options

Discover available profiles

wpr -profiles

For our broad performance captures, the important classes are general CPU/scheduling data and GPU data.

Start a file-backed capture

A typical built-in-profile form is:

wpr -start GeneralProfile -start GPU -filemode

When a dedicated temporary trace directory is needed:

wpr -start GeneralProfile -start GPU -filemode -recordtempto C:\xemu-trace-temp

The exact retained Vulkan investigation used the equivalent verbose general/GPU file profiles from WPR 10.0.26100.

Check recorder state

wpr -status
wpr -status profiles
wpr -status collectors -details

Read the output

Look for:

  • the expected profiles being active;
  • recording duration;
  • logging mode;
  • dropped/lost events = 0;
  • expected kernel collectors such as sampled profile/context switch data.

If buffers/events were lost, timing partitions and sample totals may be incomplete. Preserve that failure instead of silently treating the trace as clean.

Stop and save

wpr -stop C:\traces\xemu-lag.etl

Abort a bad capture

wpr -cancel

Use -cancel when setup failed before the intended measurement began. Do not turn a failed admission into a gameplay sample just because an ETL exists.

Practical capture rule

Start the recorder before the controlled workload, but analyze only the explicitly recorded measurement window. Startup and cleanup events can remain in the ETL without being included in the reported workload.

The retained Vulkan investigation, for example, contained startup/cleanup outside its 20.0090458-second analysis interval. Whole-trace totals were therefore not valid workload totals.


4. Windows Performance Analyzer — WPA

WPA is the interactive analysis tool for ETL traces. It is useful when discovering which thread or interval deserves a narrower export.

The tables we have used most heavily are conceptually:

  • CPU Usage (Sampled) / Samples by Thread Name;
  • Thread Delays / scheduling transitions;
  • GPU activity by process.

What to inspect first

For a lag/stutter interval:

  1. restrict the timeline to the known measurement window;
  2. identify the xemu PID;
  3. identify the target thread by name or independent identity proof;
  4. compare sampled CPU with scheduled/off-CPU time;
  5. inspect GPU intervals over the same window;
  6. resolve hot xemu addresses separately using the exact executable/symbol artifact.

Sampled CPU is not scheduled CPU time

A sampled CPU table estimates where instruction-pointer samples landed.

Example from the retained PFIFO trace:

PFIFO sampled weight:       4,179.0205 ms
PFIFO precisely scheduled:  4,494.3294 ms

Those values answer different questions and should not be substituted for one another.

Running, waiting, and ready are different states

A useful thread partition is:

measurement window
= running
+ waiting
+ ready-for-scheduling

For the retained PFIFO interval:

running: 4.4943294 s
waiting: 15.4398723 s
ready:   0.0748441 s

Interpret carefully

  • Running means the thread was scheduled on a CPU.
  • Waiting means it was not runnable because it was blocked/waiting.
  • Ready means it could run but the scheduler had not put it on a CPU yet.

A thread being off-CPU for 15 seconds does not mean Sleep() consumed 15 seconds of CPU, and it does not identify what dependency made it wait.

GPU overlap is correlation, not dependency

If GPU execution overlaps a PFIFO waiting interval, that does not prove PFIFO was waiting for that GPU execution.

The retained trace had substantial GPU/PFIFO-wait overlap. That was sufficient to justify deeper wait attribution, not sufficient to label every PFIFO wait a fence wait.

Wait-reason trap

Do not automatically read an Old Wait Reason field at PFIFO switch-in as the reason PFIFO was waiting. In our exported scheduling data that field described the other outgoing thread at the switch-in boundary and was excluded from PFIFO cause attribution.


5. WPAExporter — repeatable ETL extraction

Once the useful tables, PID, TID, and time range are known, prefer reproducible export over repeatedly clicking through the GUI.

Our portable tooling lives under:

xemu-perf-tests/utils/wpa-attribution/

See the repository README there for the source-pinned implementation.

Derive a bounded WPA profile

python3 derive-wpa-profile.py \
  --seed-profile "$SEED_PROFILE" \
  --output-profile "$PROFILE" \
  --manifest "$PROFILE_MANIFEST" \
  --pid "$PID" \
  --process-name "$PROCESS_NAME" \
  --pfifo-tid "$PFIFO_TID"

The accepted profile should contain the intended graph types and filters. If the PFIFO TID is not yet proven, omit --pfifo-tid and use the broad sampled-CPU export only for discovery.

Do not select PFIFO merely because a thread has the largest sample total. Use thread name or an independent source-exclusive identity proof.

Export the saved ETL

Run on the Windows analysis host:

powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -File .\invoke-wpa-export.ps1 `
  -Exporter $ExporterPath `
  -Etl $EtlPath `
  -Profile $ProfilePath `
  -OutputDirectory $FreshOutputDirectory `
  -StartNs $StartNanoseconds `
  -EndNs $EndNanoseconds `
  -Prefix $OutputPrefix `
  -ExpectedCsvCount 3 `
  -MaximumOutputBytes 268435456

Read the export result

An exporter process returning exit code 0 is not enough.

Require the wrapper/manifest to confirm:

  • correct ETL hash;
  • correct profile hash;
  • correct exporter/tool identity;
  • exact requested time range;
  • expected CSV count;
  • no error markers;
  • complete final CSV records/terminal line ending;
  • expected row counts;
  • no owned exporter process left behind.

We observed a WPA raw export that returned exit code 0 while logging an export error and stopping mid-row after 7,804 complete records. Those totals were rejected.

That is why export integrity is part of the analysis, not housekeeping.


6. xperf — fast command-line ETL interrogation

xperf is useful for quick trace statistics, sampled-profile summaries, process information, and context-switch exports.

Official action reference: https://learn.microsoft.com/windows-hardware/test/wpt/xperf-actions

Trace health / event inventory

xperf -i .\xemu-lag.etl -a tracestats

Use this to inspect the trace/event inventory and help confirm whether expected event classes were captured.

Sampled CPU profile

xperf -i .\xemu-lag.etl -a profile -detail

Restrict to a measurement interval, where the range is in microseconds relative to the trace timeline:

xperf -i .\xemu-lag.etl -a profile -detail -range 21796062 41805107

Read the output

-detail groups profile samples by process/module, or by decoded function when symbol decoding is available.

Important trap from our investigations: Usage% in the profile report can use all logical CPUs as its denominator. It is not necessarily “percent of one xemu thread.”

Use the profile to identify hot regions, then move to thread-bounded analysis rather than treating a system-wide percentage as the final result.

Context switches

xperf -i .\xemu-lag.etl -a cswitch -range 21796062 41805107

Initial cswitch output is system-wide. It does not automatically isolate PFIFO or distinguish its waiting time from scheduling delay.

Process/thread inventory

xperf -i .\xemu-lag.etl -a process

Use PID/TID identity from the trace rather than assuming the same PID from a different run.

Unsupported action does not mean missing events

On the WPT version used in one investigation, xperf -a gpu was rejected as an unsupported action. That did not prove the ETL lacked GPU events; GPU data was later extracted with WPAExporter.

Separate “this CLI action is unavailable” from “the trace contains no data.”


7. addr2line — source attribution from optimized addresses

x86_64-w64-mingw32-addr2line is one of the most important tools in this workflow because it turns a sampled machine address into source context using the matching DWARF artifact.

GNU reference: https://sourceware.org/binutils/docs/binutils/addr2line.html

Common command

x86_64-w64-mingw32-addr2line \
  -a -f -i -C \
  -e /path/to/xemu-dwarf.exe \
  0x1402d3a7c 0x1402d50fa 0x1401865a1

Options:

Option Meaning
-a print the input address
-f print function names
-i show inline-call chain
-C demangle C++ names
-e exact executable/debug image to use

Read the output

Typical pattern:

0x1402d3a7c
some_function
path/to/source.c:123

With -i, additional function/file/line groups after the first location are inline parents. This is especially important under LTO/O2 where a hot instruction may physically live inside an inlined chain rather than a neat standalone function.

?? or line 0

Output such as:

??
??:0

usually means the tool cannot map that address using the supplied debug information. Common causes include:

  • wrong executable/DWARF pair;
  • stripped/missing debug information;
  • incorrect address adjustment;
  • address belongs to another module;
  • optimizer emitted code without useful line attribution at that point.

Do not guess the source line when symbolization failed.

ASLR / RVA rule

A raw ETW instruction pointer may be a runtime virtual address after ASLR.

Safe model:

RVA = runtime IP - runtime image base
linked VA = preferred PE image base + RVA

Use the form expected by the selected symbolization method.

Do not subtract the preferred PE base directly from an ASLR runtime address. First establish the runtime image base recorded for that process instance.

For nearest-label maps, our retained tooling works in RVA space and explicitly verifies that the sampled IP belongs to the recorded xemu image range.


8. objdump — PE sections, symbols, and disassembly

x86_64-w64-mingw32-objdump is useful when you need to understand the exact optimized binary around a hot address or build a bounded symbol map.

GNU reference: https://sourceware.org/binutils/docs/binutils/objdump.html

Inspect PE/section layout

x86_64-w64-mingw32-objdump -h xemu.exe

Look at section VMAs/sizes and confirm the address/RVA belongs to executable text.

Inspect defined symbols

x86_64-w64-mingw32-objdump -t -C xemu.exe

This is useful for building or checking a defined-text symbol map when full source-line DWARF is not available to the analysis stage.

Disassemble a function

x86_64-w64-mingw32-objdump -d -C --disassemble=tlb_reset_dirty_range_all xemu.exe

If source/debug interleave is available:

x86_64-w64-mingw32-objdump -S -C --disassemble=tlb_reset_dirty_range_all xemu-dwarf.exe

Read the data

Use disassembly to answer questions such as:

  • Did LTO inline this helper?
  • Is the sampled address in the loop body or call setup?
  • Did the compiler vectorize/unroll the code?
  • Is an apparently expensive C helper only a few machine instructions here?
  • Does a proposed source change actually alter the generated hot loop?

Do not infer invocation frequency from static disassembly. It tells you what code exists, not how often runtime execution reached it.


9. Defined-text symbol maps

When full stack/source decoding is unavailable, an exact-PE defined-symbol map can still bound an instruction pointer to a known function extent.

The retained WPA attribution packet generated a map with 60,273 defined text labels from the exact executable.

Its reproducible helper is documented in the evidence packet. Conceptually:

bash extract-defined-text-symbols.sh /path/to/exact/xemu.exe rebuilt-symbols.tsv.gz
sha256sum rebuilt-symbols.tsv.gz

To find the nearest defined label at or below a known RVA:

gzip -dc defined-text-symbols.tsv.gz |
  TARGET_RVA=0x00296c90 perl -F'\t' -ane '
    next if /^#/;
    $last=$_ if hex($F[1]) <= hex($ENV{TARGET_RVA});
    END { print $last if defined $last }
  '

Read the result carefully

A nearest label can say “this IP is within the bounded extent beginning at pgraph_vk_finish.” It cannot recover a missing caller stack or tell whether that common function was copying, submitting, or waiting at the sampled moment.

Treat it as function ownership, not full causality.


10. rg / ripgrep — source-path recovery

Once profiling gives you a symbol, register name, trace label, or FIXME, rg is the fastest way to map it back into the source tree.

Find a function or identifier

rg -n "tlb_reset_dirty_range_all" .
rg -n "create_pipeline" hw include ui

Follow related state and callers

rg -n "tlb_reset_dirty_range_all|tlb_reset_dirty|dirty_range" .
rg -n "vkQueueSubmit|vkWaitForFences|pgraph_vk_finish" hw/xbox

Search known incomplete behavior

rg -n "FIXME|TODO|assert\(" hw/xbox

Read the result

rg gives static source locations. The next job is to reconstruct:

caller
 -> state mutation
 -> callee
 -> synchronization / cache / DMA side effect
 -> return / retry / guest-visible result

A text match is not runtime evidence. Use it to understand the path already identified by the trace.


11. Diagnostic counters

Broad traces often identify a hot framework function but not why it was called.

This is where narrowly scoped diagnostic counters are useful.

Examples from the TCG investigation counted lookup/return causes rather than logging every event.

Analyze XEMU_CAUSE records

The repository helper is:

xemu-perf-tests/utils/analyze_cause_counters.py

Command:

python3 utils/analyze_cause_counters.py \
  cause-records.jsonl \
  --start-utc-us 1788970000000000 \
  --end-utc-us   1788970020000000 \
  --output cause-summary.json

The timestamps above are examples; use the actual measurement anchors from the run.

What the analyzer validates

It rejects:

  • missing thread/timestamp identity;
  • fewer than two samples per thread;
  • counter-schema changes;
  • counter resets/decreases;
  • clock reversal;
  • incomplete lookup partitions.

The output intentionally contains:

"purpose": "causal counts, not benchmark acceptance",
"performance_acceptance": false,
"lookup_partition_check": "PASS"

Read the useful fields

For each thread:

  • delta — count growth across the bounded interval;
  • rates_per_second — event frequency;
  • *_cache_hit_percent — lookup hit ratio;
  • helper_collision_share_of_misses_percent — one identified miss class;
  • zero_arm_scan_percent — share of dirty scans that armed nothing;
  • entries_per_scan — average entries inspected;
  • new_arms_per_scan — average newly armed entries.

A high event rate tells you what repeats. It does not tell you how many milliseconds are removable.

PR #54 is the canonical example: more than 99% of sampled NULL-return leaves were attributed to the STI/shadow pair, but that result was intentionally not reported as a speedup.


12. Bounded WPA attribution scripts

After the target PID/TID and measurement range are established, the saved ETL can be reduced into a reproducible, privacy-minimized dataset.

Sanitize exported tables

python3 sanitize-attribution.py \
  --sampled "$SAMPLED_CSV" \
  --precise "$PRECISE_CSV" \
  --gpu "$GPU_CSV" \
  --output-dir "$COMPACT_DIR" \
  --pid "$PID" \
  --tid "$PFIFO_TID" \
  --image-base "$RUNTIME_IMAGE_BASE" \
  --image-end "$RUNTIME_IMAGE_END"

The compact packet retains the target IDs, timestamps, sample weights, xemu RVAs, limited external module identity, readying scope, and GPU intervals while dropping unrelated host context.

Reproduce the timing partition

python3 analyze-sanitized.py \
  --sampled "$COMPACT_DIR/sampled-pfifo.csv" \
  --precise "$COMPACT_DIR/precise-pfifo.csv" \
  --gpu "$COMPACT_DIR/gpu-process.csv" \
  --output-summary "$COMPACT_DIR/summary.json" \
  --output-leaves "$COMPACT_DIR/sampled-leaf-summary.csv" \
  --pid "$PID" \
  --tid "$PFIFO_TID" \
  --start-s "$START_SECONDS" \
  --end-s "$END_SECONDS" \
  --pe "$EXACT_PE" \
  --symbol-map "$EXACT_SYMBOL_MAP" \
  --expected-pe-sha256 "$EXPECTED_PE_SHA256" \
  --expected-symbol-map-sha256 "$EXPECTED_SYMBOL_MAP_SHA256"

Read the outputs

summary.json is useful for exact bounded totals such as:

  • scheduled running time;
  • waiting time;
  • ready time;
  • partition closure against the selected window;
  • GPU interval union;
  • temporal wait/GPU overlap.

sampled-leaf-summary.csv ranks bounded leaf ownership from sampled CPU records.

Example retained result:

tlb_reset_dirty_range_all
407.9830 ms sampled weight
9.7626% of PFIFO sample weight

That means roughly 9.76% of the PFIFO sample weight mapped to that bounded leaf. It is not 9.76% of total host CPU, not 9.76% of frame time, and not proof that the entire 407.983 ms can be removed.


13. Correctness hash comparison

Performance improvements are unusable if they silently change deterministic guest results.

The repository helper:

xemu-perf-tests/utils/hash_compare.py

compares deterministic hashes such as framebuffer, work, and result checksums from baseline/candidate result records.

Compare one candidate

python3 utils/hash_compare.py \
  /results/baseline \
  /results/candidate

Compare several candidates and save JSON

python3 utils/hash_compare.py \
  /results/baseline \
  /results/candidate-a \
  /results/candidate-b \
  --json-out hash-comparison.json

Read the output

Human output resembles:

BASELINE ...
RUN ...
  test-name framebuffer=1 work=1 result=1
VERDICT PASS checked=3 mismatch=0 missing=0 extra=0
OVERALL PASS

Meaning:

  • 1 = eligible hash matched;
  • 0 = eligible hash differed;
  • - = that hash was not eligible/available;
  • PASS requires checked hashes and no missing/extra/mismatched records;
  • UNVERIFIED means no eligible hashes were actually checked.

Do not convert UNVERIFIED into PASS.


14. PowerShell as orchestration, not evidence by itself

PowerShell is used heavily to launch bounded workloads, invoke WPR/WPAExporter, capture process identity, hash files, and clean up owned helper processes.

Useful primitives:

Get-FileHash .\xemu.exe -Algorithm SHA256
Get-Process xemu -ErrorAction SilentlyContinue
Get-CimInstance Win32_Process | Where-Object Name -eq 'xemu.exe'
wpr -status collectors -details

For automation, record at least:

  • PID;
  • creation time;
  • parent PID when available;
  • originating command/run identity.

A remote command timing out does not prove the server-side process stopped. Always perform bounded cleanup checks.

Do not recursively kill generic python, powershell, wpa, or xperf processes unless ownership has been established; unrelated host tools may be active.


15. Reading common profiling numbers correctly

This section is intentionally repetitive because these mistakes have already cost investigation time.

Sample percentage

Function X = 20% of samples on PFIFO

Means:

About 20% of the selected PFIFO sampled weight landed in Function X under the capture conditions.

It does not automatically mean:

  • Function X consumes 20% of a frame;
  • deleting Function X makes xemu 20% faster;
  • Function X is the root cause;
  • the same share exists on another renderer/game/hardware configuration.

Scheduled-running time

PFIFO running = 4.49 s in a 20 s window

Means the thread was on-CPU for about 4.49 seconds in the selected interval.

It does not explain the remaining 15.5 seconds until waiting/ready states are separated.

Waiting time

Waiting means blocked/not runnable. Find the dependency before optimizing the wait mechanism.

A faster fence poll can make the polling loop cheaper without making the fence signal earlier.

Ready time

Ready means runnable but unscheduled. This can point toward host scheduling/CPU contention rather than an emulator wait.

GPU execution overlap

Temporal overlap is useful evidence for a hypothesis but does not prove causation.

Call counts

A function called 100,000 times may still be cheap. Combine counts with cost or use the count only to justify a targeted experiment.

p95 / p99

Tail metrics describe the slow end of the interval distribution. A higher average FPS with worse p99 can still feel more stuttery.

Always retain both central throughput and tail behavior for hitch-oriented work.

Display-write cadence

Our current runner's NV2A display-write cadence is a guest progress proxy, not rendered FPS or monitor-present FPS.

Label it as cadence, not FPS.


16. Suggested command sequence for a new lag report

This is a practical starting sequence, not a requirement to run every command for every issue.

A. Identify the build

git rev-parse HEAD
git rev-parse HEAD^{tree}
sha256sum xemu.exe xemu.debug

B. Capture optimized runtime behavior

wpr -start GeneralProfile -start GPU -filemode
# launch the controlled xemu workload
wpr -status collectors -details
wpr -stop C:\traces\xemu-lag.etl

C. Check the ETL quickly

xperf -i C:\traces\xemu-lag.etl -a tracestats
xperf -i C:\traces\xemu-lag.etl -a profile -detail

Then identify the exact measurement range and rerun bounded exports:

xperf -i C:\traces\xemu-lag.etl -a profile -detail -range T1 T2
xperf -i C:\traces\xemu-lag.etl -a cswitch -range T1 T2

D. Resolve hot xemu addresses

x86_64-w64-mingw32-addr2line -a -f -i -C \
  -e xemu-dwarf.exe \
  0xADDRESS1 0xADDRESS2 0xADDRESS3

E. Find the source path

rg -n "resolved_function_name" .
rg -n "related_state|related_helper|related_register" hw include ui

F. If the framework function is only a symptom, add bounded diagnostics

python3 utils/analyze_cause_counters.py \
  cause-records.jsonl \
  --start-utc-us START \
  --end-utc-us END \
  --output cause-summary.json

Discard diagnostic timing as acceptance evidence.

G. Test a minimal uninstrumented candidate

Build another optimized Release-equivalent executable, preserve its symbols, and record new hashes.

Then compare correctness and matched performance separately.


17. Tools we deliberately do not let answer the wrong question

Tool/output Valid use Invalid conclusion
xperf profile find sampled hot regions “this percentage is removable frame time”
cswitch inspect scheduler transitions “largest system thread is PFIFO”
WPA GPU overlap correlate GPU execution with thread state “PFIFO definitely waited on this fence”
addr2line resolve exact optimized IP to source/inline chain “resolved line is the root cause”
objdump inspect generated machine code “static loop is hot because it looks long”
rg locate source definitions/call sites “textual caller definitely executed”
diagnostic counter attribute repeated event classes “count reduction equals FPS gain”
fewer Vulkan waits demonstrate API-call reduction “GPU dependency disappeared”
hash PASS deterministic contract matched “all graphics/audio behavior is correct”
one favorable run create a hypothesis “candidate is qualified”

18. Tool locations

Windows Performance Toolkit

Provides WPR, WPA, WPAExporter, and xperf.

Microsoft documentation:

GNU/MinGW binutils

Used for PE/DWARF inspection and source resolution:

  • x86_64-w64-mingw32-addr2line
  • x86_64-w64-mingw32-objdump
  • x86_64-w64-mingw32-objcopy
  • x86_64-w64-mingw32-strip

References:

xemu performance-test helpers

Repository:

https://github.com/Mainkill1/xemu-perf-tests

Important paths:

utils/analyze_cause_counters.py
utils/hash_compare.py
utils/oracle_validation.py
utils/wpa-attribution/

The evidence directory contains source-pinned reports showing exactly how each tool was used in prior investigations.


Related pages

Shortcut evidence campaigns

The commands and profilers on this page provide the measurements; Using Hot-Path Shortcuts defines how to run a hot-path shortcut campaign, validate requested/effective mode, read route counters, and reject mismatched or unreachable A/B data before calculating performance statistics.

Clone this wiki locally