Debugging Tool Perf

Linux perf

← Debugging Tools and Commands

Linux perf is the default profiler for xemu on Linux and Steam Deck. It can sample one thread's call stacks, separate scheduler wait/delay/runtime, and collect hardware counters without adding xemu instrumentation.

Property	Details
Supported OS	Linux; event availability depends on the kernel, CPU, and permissions.
Install	Distribution package normally named perf or matching linux-tools. SteamOS 3.8.16 already includes /usr/bin/perf.
Verify	perf --version, perf list, and a short controlled capture.
Primary outputs	perf.data, sched.data, text/JSON counter summaries.

Choose the perf mode by question

Question	Command family
Which code used CPU time?	perf record and perf report
Was a thread running, blocked, or runnable but unscheduled?	perf sched record and perf sched timehist
Is executing code instruction-heavy, cache-bound, or branch-heavy?	perf stat
Are threads bouncing shared cache lines?	perf c2c after simpler evidence points to false sharing

Do not interpret an on-CPU profile as an explanation for off-CPU time. Use the scheduler capture alongside the sampling capture.

Identify the process and thread

Thread IDs are local to one run. Record the process tree and thread list immediately before capture:

pgrep -n -x xemu

ps -L -p 12345 \
  -o pid,tid,psr,pcpu,stat,wchan:32,comm

top -H -p 12345

Replace 12345 with the PID printed by pgrep. In later commands, replace 12346 with the selected thread's TID.

Save the selected PID, TID, thread name, source commit, executable SHA-256, matching symbols, workload boundary, and capture start/stop times. A thread name is useful context but is not immutable identity.

On-CPU call stacks

For an optimized binary with matching DWARF:

perf record \
  -F 499 \
  -g \
  --call-graph dwarf \
  -t 12346 \
  -o perf-thread.data \
  -- sleep 10

perf report -i perf-thread.data

Use a lower sampling rate when the Deck or workload is sensitive to profiling overhead. If the build retains frame pointers, --call-graph fp is cheaper and often more reliable:

-fno-omit-frame-pointer

Do not assume frame-pointer unwinding is valid merely because perf accepts the option. Verify stack depth and unresolved-frame rate in the result. --call-graph dwarf is the fallback for an optimized build without dependable frame pointers.

Optional GUI

Hotspot reads perf.data and provides a flame graph, caller/callee views, and timelines. It is an optional viewer for the same capture, not another source of evidence. Keep perf report available for scripted and remote checks.

Scheduler time: run, wait, and delay

A scheduler trace answers a different question:

interval = task runtime + blocked time + runnable delay

Capture the process only for a smaller file:

sudo perf sched record \
  -p 12345 \
  -o sched.data \
  -- sleep 10

If the investigation must retain external wakers, capture system-wide for a short bounded interval and filter during analysis:

sudo perf sched record \
  -a \
  -o sched.data \
  -- sleep 10

sudo perf sched timehist \
  -i sched.data \
  -t 12346 \
  --state \
  --wakeups

timehist reports wait time, scheduling delay, and run time. Interpret them separately:

• run time: the thread was executing;
• wait time: the thread was not runnable;
• scheduling delay: the thread was runnable but did not have a CPU;
• wakeup: useful attribution evidence, but still inspect the blocked and waking stacks before assigning ownership.

On the checked Steam Deck, unprivileged perf sched record could not read sched:sched_switch; use sudo for scheduler tracing. Ordinary user-space perf record and perf stat worked without privilege for a process owned by the same user.

Hardware counters

When the target thread is genuinely on-CPU, collect a bounded counter set:

perf stat \
  -t 12346 \
  -e cycles:u,instructions:u,branches:u,branch-misses:u,cache-references:u,cache-misses:u \
  -- sleep 10

The :u suffix restricts the example to user space and is useful under restrictive perf_event_paranoid settings. Confirm the event names with perf list; unsupported or multiplexed counters must remain visible in the report.

Interpretation is comparative, not diagnostic by one number:

Observation	Possible direction	Does not prove
High instruction count	Too much actual work	Which work can be removed
Low IPC with cache misses	Memory locality or cache pressure	A specific data structure is wrong
High branch-miss share	Dispatch or unpredictable branches	That branch elimination improves guest progress
High runnable delay	Host CPU contention	That xemu itself consumed the missing time

Normalize counters to the same guest-progress interval when comparing builds.

perf c2c is an escalation tool

Use cache-to-cache analysis only after ordinary samples and counters suggest shared-cache-line contention. It normally needs elevated access and a system-wide capture:

sudo perf c2c record -o c2c.data -- sleep 10
sudo perf c2c report -i c2c.data

False-sharing investigation requires the exact binary layout and symbols. Do not infer a production struct change from an address alone.

Steam Deck notes

The 2026-09-21 Deck check found:

SteamOS 3.8.16
kernel 6.16.12-valve24.5-1-neptune-616-gb2f7cfe85e45
perf 6.15-3
kernel.perf_event_paranoid = 2
kernel.kptr_restrict = 0

perf record with DWARF and a user-space perf stat counter set both completed against a short controlled process. Scheduler tracepoints required elevation. Treat this as an appliance snapshot, not a guarantee for later SteamOS versions; run the preflight in the Steam Deck and Linux Thread Diagnosis guide.

Boundaries and common mistakes

• Sampling frequency changes overhead and sample confidence.
• A high sample share is not removable frame time.
• A thread blocked for 20 ms did not consume 20 ms of CPU.
• perf sched -p can omit an external waker; use a short -a capture when wakeup attribution matters.
• Counter multiplexing and unsupported PMU events can invalidate naive comparisons.
• Symbolize only with the executable/debug artifact from the captured build.
• Profiling results are diagnostic; acceptance still uses an uninstrumented optimized build.

Evidence to retain

• perf --version, kernel version, CPU model, perf_event_paranoid, and kptr_restrict;
• source/executable/symbol hashes and build flags;
• PID/TID/name mapping and exact capture command;
• raw perf.data or sched.data plus exported summaries;
• lost samples, permission warnings, unresolved stacks, and multiplexed counters;
• the exact guest-progress interval and cache/configuration state.

Official references

• perf record manual
• perf sched manual
• perf stat manual
• perf c2c manual
• Linux perf security guidance

Related xemu guides

• Steam Deck and Linux Thread Diagnosis
• BCC and eBPF
• Perfetto
• strace
• Release symbols
• Performance Methodology