Idea: live trace — ordered, thread-scoped call traces of one request

[alex09x]

Problem

The hardest questions I get when reversing a large stripped Rust service binary are about order, not about arithmetic:

• Inside one request, is the old record removed before or after the wrapper that prepares the new one runs?
• Does a cancel settle the account immediately, or only mark it so that a pass at the end of the batch settles it?
• Which of 79 callers of a shared balance-update function ran for this particular input, and with what arguments?

The static tools (c, path, args) give candidates. live probe confirms single points. Ordering across several functions in one request, though, I reconstruct by hand: probe 6–10 addresses at once, dump fetch ts lines, then sort by timestamp and thread in awk. It's slow and error-prone (fetch lines aren't tagged with their probe, timestamps from different CPUs interleave, and nothing tells me where one request ends and the next begins).

What I'd like: live trace

A bounded, ordered call trace of one process, opened and closed by events:

xref --bin B live trace --pid P \
  --start 0x2c5e3c0 --start-filter "asset >= 100000000" --fetch "asset=+16(%dx):u64" \
  --stop-return 0x2c5e3c0 \
  --functions 0x22e2b40,0x22e4bd0,0x2c71dd0,0x2d38090,0x2d37130,0x373aa80 \
  --fetch-at 0x373aa80 "amount=%r8:u64,price=%r9:u64,debit=%cx:u64" \
  --max-windows 5 --json

Output: one window per triggering call, the ordered entries and returns of the listed functions on the triggering thread, with depth, the static VA of the call site, and the fetched values:

window 1  tid 3709093  start 0x2c5e3c0 asset=100215681
  0  > 0x2c5e3c0
  1    > 0x22e2b40   from 0x2c5ecdb
  1    < 0x22e2b40
  1    > 0x2c71dd0   from 0x2c5ef95
  2      > 0x2d38090 from 0x2c71e40
  3        > 0x373aa80 from 0x2d388fa  amount=... price=... debit=0
  ...
  0  < 0x2c5e3c0

Details that matter

1. Thread scoping: only events on the thread that hit --start, from start until the matching return (--stop-return) or --stop VA.
2. Function list modes: an explicit list; --callees-of FN --depth N (static callee tree, like tree); and --all-calls-in FN (every direct call site inside one function, probed at the call and the return site).
3. Exact ordering: per thread, in program order. Merging probes from different CPUs by timestamp isn't enough.
4. --fetch-at VA "..." per function, with the same syntax as live probe --fetch, plus --fetch-ret VA "ret=%ax:u64" on return.
5. Window filters: --start-filter uses the start probe's fetches, so I can trace only the one request I care about. --max-windows and --max-events keep it bounded.
6. Correlation to an external log: --mark-file PATH --mark-regex RE stamps each window with the latest matching line (for me, the batch height), so a window maps to a position in the input.
7. Diffable output: live trace JSON should feed trace diff A B, comparing the ordered call sequences of two windows, for example a case that works vs. one that doesn't.

Backends

The uprobe/uretprobe pair per function is enough for depth-limited traces. Intel PT (now partly present through the AUX import in #89) would make --all-calls-in cheap and exact. Either backend is fine; I care about the CLI and the output.

Why it matters

With this, "is X done before Y in request R" becomes one command instead of an hour of static reading. A real example from this week: verifying that a cancel inside a modify happens before the prepare step of the wrapper, and that a cancelled conditional order only marks the account for an end-of-batch pass.

Related: field report #93 (overall feedback); profiler requests in #78 and #92.

[alex09x]

How I would build it: OS mechanisms, in order of preference

Target box for my use: Linux 6.8 with BTF (/sys/kernel/btf/vmlinux), AMD Zen 5 (no Intel PT, but IBS and the core PMU are available), root via sudo, bpftrace/bpftool installed. The design should not depend on Intel PT.

1. Primary backend: eBPF on uprobes (uprobe_multi), with filtering and ordering in the kernel

Attach. Use one BPF_TRACE_UPROBE_MULTI link (Linux ≥ 6.6) for all traced VAs: a single attach, a single program, with the cookie per address = the event id (static VA + kind). Before 6.6, fall back to one perf_event_open(PERF_TYPE_UPROBE) per address and PERF_EVENT_IOC_SET_BPF. Restrict it to the target with pid on the link.

Avoid uretprobes for returns. Since xref already knows every call site statically, probe the call instruction and the instruction right after it (the return site) inside the traced callers, instead of the callee entry plus a uretprobe:

• there is no return trampoline, so there is no breakage when the stack is walked or unwound (Rust panics unwind through frames, so uretprobes would never fire and depth would drift);
• tail calls (jmp to a function) are known statically and can be emitted as > with no matching <, as a marked tail edge;
• every event carries its exact call-site VA for free (the cookie).

For --functions mode (entry points only), use entry uprobes plus uretprobes, flagged as best-effort for depth.

Thread scoping and windows in BPF maps.

• HASH(tid → window_state {active, depth, seq, window_id}).
• The --start probe evaluates --start-filter in BPF, over register and bpf_probe_read_user fetches compiled from the existing --fetch syntax, and sets active for that tid only.
• Other probes check active first and return immediately otherwise, so the cost for untraced threads is one map lookup.
• --stop-return is the return site of the start call on the same tid with depth == 0, which clears the state.
• Unwinding safety: if a --start fires on a tid that is still active (a previous window was lost to unwinding), close the old window as truncated.

Ordering. Per tid, program order is guaranteed by a per-window seq++ in the map value, so CPU clocks don't matter. Emit records into a BPF_MAP_TYPE_RINGBUF; userspace sorts by (window_id, seq). Record bpf_ktime_get_ns() only as an informational duration.

Fetches. Compile --fetch-at VA "a=%r8:u64,b=+16(%dx):u64" into a small per-cookie fetch table: register index, optional deref offset, width. The BPF program reads pt_regs and bpf_probe_read_user. That makes it the same language as live probe --fetch, so existing filters and scripts carry over.

Bounds and overhead. Use --max-events and --max-windows counters in an ARRAY map; when they are reached, BPF stops emitting and userspace detaches the link. A uprobe costs about 1–3 µs per hit (int3 + trap), which is fine for request-scoped windows, and the cheap active early-out keeps untraced threads close to free. Report ring buffer drops as lost.

Implementation in Rust. Use libbpf-rs or aya with a CO-RE object embedded in the binary. No runtime clang is needed, because the program is generic and the per-trace configuration goes in maps.

2. Fallback backend without BPF: tracefs uprobes (what live probe already uses)

Create the same events through uprobe_events, filter with the filter file, and read trace_pipe_raw. The weakness is that per-thread ordering then relies on timestamps across per-CPU buffers. Make it reliable by setting trace_clock=mono (or x86-tsc when it's invariant) and merging the per-CPU buffers by timestamp. Within one tid, consecutive events can't run on two CPUs at once, so a monotonic clock gives a correct per-tid order. Thread scoping and windows then live in userspace, which costs more events, but it works where BPF is not allowed.

3. Exact "every call in FN" mode (--all-calls-in)

• Intel PT, where available: perf_event_open on the intel_pt PMU, per thread, with an address filter (ADDR_FILTERS on the function range; the hardware has 2–4 filter ranges), an AUX buffer in snapshot mode, and a snapshot taken when the --stop uprobe fires. Decode with libipt against the ELF, then emit calls and returns. This gives exact and cheap results, including indirect calls, and the AUX plumbing from Preserve perf samples alongside AUX traces #89 already exists.
• AMD (no PT): derive the probe set statically, a uprobe on every direct and indirect call site in FN and on each return site, and use backend 1. For indirect calls (call rax, call [rip+x]), also fetch the target register or memory so the callee is named.
• LBR as a cheap complement: PERF_SAMPLE_BRANCH_STACK with PERF_SAMPLE_BRANCH_CALL_STACK (on Zen 4/5 via LBR v2, 16 entries) on a BPF-triggered sample gives the last N taken calls and returns at the stop point. That's enough to recover the short path that led there without probing everything.

4. Last resort: ptrace

Use PTRACE_SEIZE + int3 patches + PTRACE_O_TRACECLONE. It's exact, unprivileged when ptrace_scope allows it, and works anywhere, but it stops the process on every event (around 10 µs+) and changes timing. It's acceptable only for --max-events in the hundreds. The hardware breakpoints DR0–DR3 that live watch uses fit only 4 addresses, so they're not a general solution.

What xref's static side contributes (the reason it belongs in xref, not bpftrace)

• the call-site and return-site VAs for --callees-of FN --depth N and --all-calls-in FN, including tail-jump edges, which bpftrace can't know in a stripped binary;
• PIE slide and build-id verification (it already exists for live probe);
• symbolization of every event: function range, call site, source location from panic Locations;
• rendering against the static tree, so text output can show "expected callee, not called" in a tree-like view.

Pitfalls to handle

• Async runtimes: a request can hop threads at .await points. Thread scoping is correct for synchronous request handling (my case) but should say so. A later option could follow a task id fetched from a register at the poll entry.
• Inlining: an inlined callee has no call site. xref could warn when a requested function has callers only through inlined copies, based on its location data.
• Recursion: depth comes from the per-window counter, so recursion is fine. Mutual recursion through tail calls is marked explicitly.
• Lost events: always print lost per window, and mark the window incomplete instead of silently dropping it.

Suggested order of work

1. Backend 1 with --functions and --start/--stop-return/--start-filter/--fetch-at, text and JSON output (live trace: ordered thread-scoped call trace windows (v1) #111).
2. Call-site/return-site mode for --callees-of --depth.
3. --all-calls-in, statically derived on AMD and through Intel PT where available.
4. --mark-file and trace diff.

[alex09x]

Implemented as xref live trace in #116 (closes #111, main 8349361). A window opens on --start in one thread and closes on the matching --stop-return or on --stop; events inside are in program order per tid. The traced set comes from --functions and/or --callees-of FN --depth N; --start-filter gates windows on start fetches. --max-windows, --max-events and --duration-ms detach all probes and print a summary. Usage and the JSON schema are in docs/live.md.

[alex09x]

xref live trace (#116) is released in v0.4.0. Closing this idea as implemented; follow-ups (for example an eBPF backend) can go in new issues.