Linux Steam Deck Thread Diagnosis

Steam Deck and Linux Thread Diagnosis

← Debugging Tools and Commands

Use this guide when a Linux xemu thread appears to cost a large part of a frame. The goal is to classify the interval before changing code:

on-CPU execution
blocked/off-CPU wait
runnable scheduler delay
syscall/driver wait
GPU dependency
mixed interval

The default Steam Deck stack is deliberately small: Linux perf plus strace. Add BCC/eBPF when blocked/waker stacks remain unresolved, Perfetto when event order across threads matters, and Tracy only for an explicitly instrumented xemu build.

Checked appliance snapshot

Read-only probing on 2026-09-21 found:

Item	Observed state
OS	SteamOS 3.8.16
Kernel	6.16.12-valve24.5-1-neptune-616-gb2f7cfe85e45
Root filesystem	SteamOS read-only mode enabled
perf	/usr/bin/perf, version 6.15-3
strace	/usr/bin/strace, version 6.15
BTF	/sys/kernel/btf/vmlinux readable
debugfs / tracefs	Mounted
kernel.perf_event_paranoid	2
kernel.kptr_restrict	0
BCC / bpftrace	Not installed
Perfetto / Tracy	Not installed

Controlled probes confirmed that unprivileged perf record --call-graph dwarf and user-space perf stat worked. Unprivileged perf sched record failed because the Deck user could not read the scheduler tracepoint, so scheduler tracing requires elevation on this image.

This is a dated appliance result. SteamOS updates can replace packages, the kernel, and policy. Run the preflight before relying on it.

Preflight

cat /etc/os-release
uname -srmo

perf --version
strace --version | head -1

cat /proc/sys/kernel/perf_event_paranoid
cat /proc/sys/kernel/kptr_restrict

test -r /sys/kernel/btf/vmlinux && echo BTF-present
mount | grep -E ' (debugfs|tracefs) '

command -v perf strace
command -v bpftrace || true
command -v tracebox || true

Record the output in the run manifest. Do not permanently disable SteamOS read-only protection merely to install a convenience viewer. User-owned static tools, a separate workstation, or a purpose-built diagnostic bundle are preferable.

Step 1: identify the exact thread

pgrep -n -x xemu

ps -L -p 12345 \
  -o pid,tid,psr,pcpu,stat,wchan:32,comm \
  | tee threads-before.txt

top -H -p 12345

Replace 12345 with the PID printed by pgrep. Choose a run-local TID; later commands use 12346 as the example TID. Keep the whole thread list because names can repeat and IDs change every launch.

Step 2: capture on-CPU work

perf record \
  -F 499 \
  -g \
  --call-graph dwarf \
  -t 12346 \
  -o perf-thread.data \
  -- sleep 10

perf report -i perf-thread.data

If the call graph is shallow or unresolved, verify matching symbols. Use --call-graph fp only for a binary built with dependable frame pointers.

Step 3: classify scheduling time

For a smaller process-scoped trace:

sudo perf sched record \
  -p 12345 \
  -o sched.data \
  -- sleep 10

sudo perf sched timehist \
  -i sched.data \
  -t 12346 \
  --state \
  --wakeups \
  | tee sched-timehist.txt

Use a short system-wide -a capture when the waker may be outside xemu and its identity is required. Scheduler data can contain activity from other applications, so keep raw traces private.

Classify the result:

Dominant time	Next step
Running	Inspect perf report; then collect perf stat counters if the code remains expensive.
Blocked	Use strace for syscall boundaries; escalate to BCC offcputime/offwaketime.
Runnable delay	Inspect host CPU saturation, clocks/power policy, affinity, and competing processes.
Mixed	Use Perfetto to align scheduling, samples, wakeups, and xemu events.

Step 4: check syscall waits

sudo timeout --signal=INT 10s \
  strace \
  -ttT \
  -p 12346 \
  -e trace=futex,poll,ppoll,epoll_wait,ioctl,clock_nanosleep \
  -o strace-thread.log

An ioctl or futex lasting most of the interval redirects the investigation toward the driver/GPU or a cross-thread dependency. It does not by itself identify the producer that must change.

Step 5: explain executing cost

perf stat \
  -t 12346 \
  -e cycles:u,instructions:u,branches:u,branch-misses:u,cache-references:u,cache-misses:u \
  -- sleep 10 \
  2> perf-stat.txt

Compare the counters only across the same guest-progress interval and effective settings. Preserve unsupported-event and multiplexing messages.

Escalation ladder

flowchart TD
    A[Identify PID/TID and guest interval] --> B[perf record: on-CPU stacks]
    A --> C[perf sched: run/wait/delay]
    C -->|Blocked| D[strace: syscall boundary]
    C -->|Blocked dependency unclear| E[BCC offcputime/offwaketime]
    C -->|Runnable delay| F[Host scheduling and power state]
    B -->|Executing cost unclear| G[perf stat]
    B -->|Possible false sharing| H[perf c2c]
    D --> I[Perfetto unified timeline]
    E --> I
    I -->|Emulator phase still ambiguous| J[Optional Tracy build]

Loading

Stop when the unresolved question has been answered. Running every tool creates redundant evidence and increases perturbation.

Optional-tool policy on SteamOS

Tool	Deck policy
perf, strace	Built-in first-line tools on the checked image.
Perfetto tracebox	Good user-owned static addition; scheduler/kernel capture still needs elevation.
BCC/eBPF	Install only when blocked/waker attribution is required; revalidate kernel/package compatibility.
bpftrace	Use only for a reviewed probe that existing BCC tools cannot express.
Hotspot	Analyze perf.data on a workstation; no need to install it on the Deck.
Tracy	Ship only with an explicit instrumented xemu build/tool bundle.

Not included in the default performance stack:

• uftrace substantially overlaps call-graph questions already covered by perf and optional Tracy;
• rr intentionally changes scheduling and is unsuitable for natural frame-time measurement;
• GDB, TSan, and Helgrind answer correctness/race questions and should be added when a specific bug requires them, not to every performance appliance.

Power and workload controls

The checked Deck reported the powersave CPU governor. That label alone does not define effective frequency behavior on modern AMD hardware. Record rather than assume:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
grep -m1 'cpu MHz' /proc/cpuinfo

Keep these stable across comparisons:

• charger/battery state and thermal condition;
• TDP/GPU-clock/CPU-clock limits;
• display refresh and frame cap;
• renderer, resolution scale, VSync, shader/prewarm mode, and caches;
• game/snapshot/XISO identity and exact interval;
• background downloads, updates, overlays, and desktop applications.

Do not change power policy while diagnosing a wait unless the policy itself is the experiment.

Evidence to retain

manifest.txt
threads-before.txt
perf-thread.data
sched.data
sched-timehist.txt
strace-thread.log
perf-stat.txt
xemu executable SHA-256
matching symbol SHA-256

The manifest should also record tool versions, command lines, warnings, start/stop anchors, guest progress, renderer/GPU/driver, settings, and cache state. Raw traces can contain private paths and other processes; keep them private and publish only bounded, reviewed exports.

Automation boundary

The eventual appliance helper should orchestrate these existing tools rather than invent a new profiler. First retain one verified manual capture from the Deck so the wrapper can assert tool admission, target identity, duration, and artifact creation. A future thread-diagnose helper should fail when the TID disappears, a trace is empty, permissions are insufficient, symbols do not match, or a tool reports dropped data. It should not convert those conditions into zero-cost or passing results.

Acceptance boundary

These captures identify ownership and causal paths. They are not performance acceptance. After a fix:

1. remove or disable diagnostic instrumentation;
2. build the normal optimized product;
3. repeat the bounded workload with the same settings;
4. report cadence, p95/p99, CPU/GPU use, correctness, and unfavorable results;
5. retain the diagnostic trace only as explanation for the change.

Official references

• Valve Steam Deck development: loading and running games
• Linux perf security guidance
• Perfetto Linux tracing cookbook
• BCC installation guide
• strace manual

Related tool pages

• Linux perf
• strace
• BCC and eBPF
• Perfetto
• Tracy
• Release symbols
• Debugging Lag and Hot Paths