Debugging Tools and Commands

Debugging Tools and Commands

Use this page to choose a debugging tool. Each linked page explains where to obtain the tool, how to verify the installation, how xemu uses it, how to interpret its output, and what evidence to retain.

Use Debugging Lag and Hot Paths for the investigation process and Graphics Transition Debugging Tools for scene, resource, snapshot, and renderer-transition failures.

Tool index

Name	OS	Description / use cases
DSP capture, replay and fuzzing	Linux, Steam Deck tested; Windows unqualified	Experimental PR #384: production GP/EP state/PCM/I/O comparison, private capture and seeded failure reduction. Not merged or released.
Git	Windows, Linux, macOS	Identifies the exact source commit and tree, checks for uncommitted changes, and bounds baseline/candidate diffs.
SHA-256 and artifact identity	Windows, Linux, macOS	Proves byte identity for executables, symbols, traces, configurations, captures, and test media.
Release symbols with objcopy and strip	Windows cross-build, Linux, macOS	Keeps a small optimized executable while retaining matching DWARF for source attribution.
Windows Performance Recorder (WPR)	Windows	Records bounded ETW traces containing CPU, scheduling, and optional GPU activity.
Windows Performance Analyzer (WPA)	Windows	Interactively isolates processes, threads, hot intervals, waits, ready time, and GPU overlap in ETL traces.
WPAExporter	Windows	Exports repeatable, filtered WPA tables from ETL traces for scripted analysis.
xperf	Windows	Performs quick command-line ETL health, sampled CPU, process, and context-switch interrogation.
addr2line	Windows, Linux, macOS	Resolves optimized machine addresses to functions, source lines, and inline chains using matching symbols.
objdump	Windows, Linux, macOS	Inspects PE/ELF sections, symbols, and generated machine code around a hot address.
Defined-text symbol maps	Windows, Linux, macOS	Provides bounded function ownership when full stack or source-line symbolization is unavailable.
ripgrep (rg)	Windows, Linux, macOS	Finds definitions, callers, state transitions, diagnostics, and known incomplete paths in the source tree.
PowerShell automation	Windows, Linux, macOS	Orchestrates Windows GUI runs, process identity, hashing, input, ETW capture, and owned-process cleanup.
Linux perf	Linux, Steam Deck	Samples per-thread CPU stacks, separates scheduler run/wait/delay time, and collects hardware counters.
strace	Linux, Steam Deck	Finds long futex, polling, sleep, driver ioctl, and file-I/O syscalls on a selected thread.
BCC and eBPF	Linux; optional on Steam Deck	Attributes off-CPU stacks, waking threads, and run-queue delays when perf/strace do not explain a wait.
Perfetto	Linux, Steam Deck capture; browser/workstation analysis	Combines scheduler events, wakeups, samples, and process/thread identity on one timeline.
Tracy	Instrumented builds on Windows/Linux/macOS	Adds frame-oriented CPU/GPU zones, locks, allocations, and custom xemu phase markers.
RenderDoc	Windows, Linux, Android	Captures and replays guest Vulkan/OpenGL frames; inspects draws, resources, pixel history, and pipeline state.
Vulkan validation layers	Windows, Linux, macOS, Android	Detects invalid Vulkan use, object-lifetime errors, and synchronization hazards in diagnostic runs.
xemu cause counters	Host-independent analysis	Attributes why a hot framework path repeats without emitting one text log per event.
WPA attribution helpers	Windows capture; cross-platform analysis	Produces integrity-checked, privacy-minimized ETL exports and bounded timing summaries.
Correctness hash comparison	Windows, Linux, macOS	Compares deterministic baseline/candidate results and keeps UNVERIFIED distinct from PASS.

Choose by question

Question	Start with	Continue with
Which xemu thread owns a lag interval?	WPR	WPA or xperf
Is the thread running, waiting, or ready?	WPA	WPA attribution helpers
Which Linux/Deck thread owns a lag interval?	Linux perf	Steam Deck thread diagnosis
Is a Linux thread running, blocked, or runnable but delayed?	perf sched	BCC/eBPF or Perfetto
Is a Linux thread stuck in a syscall or driver call?	strace	Perfetto for cross-thread ordering
Does an executing Linux thread suffer cache/branch pressure?	perf stat	perf c2c only after evidence suggests false sharing
Do xemu-specific frame phases need explicit instrumentation?	Tracy	Validate later with an uninstrumented Release build
Which source line owns a sampled address?	addr2line	objdump and ripgrep
Which draw or resource first corrupts a frame?	RenderDoc	Vulkan validation and the graphics-transition guide
Why does a framework helper repeat?	xemu cause counters	Exact-source inspection with ripgrep
Did candidate output remain equivalent?	Correctness hashes	Visual/gameplay validation for behavior outside the hash contract
Does evidence belong to the reviewed build?	Git and SHA-256	Matching release symbols

Standard investigation flow

flowchart LR
    A[Exact Release executable and symbols] --> B[Bounded capture]
    B --> C[Inspect the correct process, thread, API, and time range]
    C --> D[Resolve addresses and resources]
    D --> E[Inspect exact source and state ownership]
    E --> F[Add one bounded diagnostic if causality is still missing]
    F --> G[Build an uninstrumented candidate]
    G --> H[Correctness and performance validation]

Loading

1. Record commit, tree, working-tree state, executable hash, symbol hash, tool versions, configuration, renderer, GPU, and driver.
2. Start capture before the controlled action, then report only the explicit measurement interval.
3. Verify that the capture contains the intended workload. A file existing does not prove correct admission.
4. Use the least invasive tool that answers the unresolved question.
5. Remove diagnostic timing from acceptance evidence. Validate the final change in an optimized, uninstrumented build.

Linux and Steam Deck tool policy

Start with tools already present on the checked SteamOS appliance: perf and strace. Add BCC/eBPF only for unresolved blocked/waker stacks, Perfetto only when a unified timeline is required, and Tracy only when kernel/system tracing cannot express an xemu-specific phase.

Hotspot is documented as an optional GUI for perf.data; bpftrace is an optional custom-probe layer on the BCC page. They do not need separate default workflows. uftrace, rr, GDB, TSan, and Helgrind are deliberately excluded from this performance set because they duplicate the selected tools or answer a different correctness question.

Evidence rules

• A source commit does not identify a binary. Record executable and symbol hashes.
• A profiler identifies where time or samples appeared; it does not prove root cause or removable cost.
• Temporal GPU/CPU overlap does not prove a dependency.
• A source match does not prove that code executed.
• A hash PASS proves only the hashed contract. UNVERIFIED is not PASS.
• Display-write cadence is a guest-progress proxy, not rendered or monitor-present FPS.
• Keep diagnostic logging, validation, and tracing disabled during ordinary performance acceptance unless that run is explicitly measuring their cost.
• Preserve failed admission, dropped events, incomplete exports, abnormal termination, and unfavorable results instead of relabeling them as passes.

Scenario guides

• Debugging Lag and Hot Paths
• Steam Deck and Linux Thread Diagnosis
• Graphics Transition Debugging Tools
• Performance Methodology
• Performance Research Test Protocol
• PGR2 Race-to-Menu Corruption Lessons
• Pull Request Lessons