Debugging Tool BCC eBPF

BCC and eBPF

← Debugging Tools and Commands

BCC's eBPF tools attribute why a Linux thread stopped running, which thread woke it, and how long a runnable task waited for a CPU. They complement perf: perf record explains on-CPU work, while BCC explains important off-CPU intervals.

Property	Details
Supported OS	Linux with the required BPF, tracepoint, symbol, and privilege support.
Install	Use the distribution's BCC tools package and the matching kernel support described by the official BCC installation guide.
Verify	Locate the scripts, confirm /sys/kernel/btf/vmlinux or matching headers, then run a short controlled capture.
Primary outputs	Text or folded stack aggregates from offcputime, offwaketime, and runqslower.

Why BCC belongs in the xemu workflow

A 20 ms thread interval can contain very little CPU execution:

3.1 ms running
11.8 ms blocked
4.6 ms runnable but unscheduled
0.5 ms running

Optimizing the sampled code would miss most of that interval. Use these tools by question:

Tool	Question
offcputime	At which stack did the thread block?
offwaketime	Which blocked stack and waking stack formed the dependency?
runqslower	When was the thread runnable but delayed by the scheduler?
profile	Optional BCC sampling when perf is unsuitable; normally prefer perf for xemu CPU profiling.

The executable names and locations vary by distribution. Debian-derived packages often add a -bpfcc suffix; other packages install scripts under /usr/share/bcc/tools. Resolve them once and record the actual path.

Preflight

uname -r
test -r /sys/kernel/btf/vmlinux && echo BTF-present
mount | grep -E ' (debugfs|tracefs) '
sudo test -r /sys/kernel/tracing/events/sched/sched_switch

command -v offcputime || \
  test -x /usr/share/bcc/tools/offcputime

Do not treat BTF alone as proof that the tools will work. A packaged BCC build, Python bindings, kernel configuration, lockdown mode, tracepoint access, and privileges can each affect admission.

Capture where a thread blocks

Replace 12346 in the commands below with the run-local TID selected from xemu's thread list.

sudo /usr/share/bcc/tools/offcputime \
  -t 12346 \
  -m 1000 \
  10 \
  > offcpu.txt

-m 1000 keeps waits of at least 1,000 microseconds, and the final 10 bounds the capture to ten seconds. Start with a threshold that excludes normal micro-waits without hiding the hitch being investigated.

Useful results resemble:

pthread_cond_wait
qemu_cond_wait
fifo_wait
renderer_wait

The stack is where the thread stopped running. It identifies a wait boundary, not automatically the subsystem that caused the dependency.

Capture blocked and waking sides

sudo /usr/share/bcc/tools/offwaketime \
  -t 12346 \
  -m 1000 \
  10 \
  > offwake.txt

offwaketime joins the blocked thread's stack to the waking thread's stack. This is useful for renderer/PFIFO, vCPU/PFIFO, audio-worker, condition-variable, and fence-service dependencies.

The current upstream script supports -t TID; verify --help because older distribution packages can differ.

Capture scheduler delay

runqslower runs until interrupted, so wrap it in a bounded timeout:

sudo timeout --signal=INT 10s \
  /usr/share/bcc/tools/runqslower \
  -t 12346 \
  1000 \
  > runqslower.txt

The positional 1000 reports run-queue delays of at least 1,000 microseconds. A long delay means the task was ready to execute but did not receive a CPU. Check CPU saturation, frequency policy, affinity, competing processes, and thread priority before changing xemu synchronization.

Optional bpftrace escalation

bpftrace is useful when the packaged BCC tools do not express the required condition. Keep one-off probes in a reviewed script, bound them by PID/TID and duration, and record lost-event counts.

Do not add a custom probe when offcputime, offwaketime, runqslower, or Perfetto already answers the question. Ad hoc probes are harder to reproduce and easier to mis-filter.

Steam Deck status

On the checked SteamOS 3.8.16 appliance:

• BTF was available at /sys/kernel/btf/vmlinux;
• debugfs and tracefs were mounted;
• the kernel advertised CONFIG_BPF, CONFIG_BPF_SYSCALL, CONFIG_BPF_JIT, and CONFIG_DEBUG_INFO_BTF;
• BCC tools and bpftrace were not installed;
• the SteamOS root filesystem was read-only.

That makes eBPF feasible but not the default Deck dependency. Begin with the built-in perf and strace path. Install BCC only when the remaining question requires blocked/waker stacks that those tools cannot provide. Keep optional tools outside the immutable system image where possible, and re-run the preflight after every SteamOS update.

Do not assume a generic container automatically gains usable host tracing. BPF and tracefs access still cross the host-kernel privilege boundary.

Boundaries and common mistakes

• Root access does not fix a kernel/tool ABI mismatch.
• A blocked stack names the wait site, not necessarily the owner of the delay.
• Wakeup attribution can be incomplete if the selected filter excludes the waker.
• Stack aggregation loses event order; use Perfetto when one chronological timeline is required.
• eBPF collection can change timing and must not become ordinary performance acceptance.
• Record dropped stack traces, missing symbols, warnings, and tool exit status.

Evidence to retain

• BCC tool version/package, kernel version, BTF/header source, and exact script path;
• full command, PID/TID mapping, duration, and threshold;
• raw text/folded output and tool warnings;
• executable/symbol hashes and selected workload interval;
• perf sched evidence for the same interval when scheduler classification matters.

Official references

• BCC repository and tool catalog
• BCC installation guide
• offcputime source and options
• offwaketime source and options
• runqslower source and options
• bpftrace documentation

Related xemu guides

• Linux perf
• Perfetto
• Steam Deck and Linux Thread Diagnosis
• Debugging Lag and Hot Paths