DWARF stack unwinding + pprof address/mapping fidelity

[dpsoft]

Summary

This branch lands the DWARF/hybrid stack unwinder and the pprof address + mapping fidelity refactor (S — 115 commits totaling roughly 2 months of design and implementation. End result:

• Release-built C++/Rust binaries unwind correctly even without frame pointers, via per-binary CFI tables compiled from .eh_frame and consulted in BPF on FP-less PCs.
• pprof output preserves real per-binary identity: Mapping entries carry path + start/limit/file-offset + GNU build-id; Location entries are address-keyed (Address - MapStart + MapOff) so two PCs at the same source line stay distinguishable.
• Plumbing for downstream LLVM sample-PGO (Rust -Cprofile-use and friends) is now in place; the conversion tool itself is the next piece and is intentionally out of scope here.

Architecture changes

The hybrid walker shares the FP path's userspace pipeline:

• unwind/ehcompile/ — pure-Go .eh_frame parser. Classifies PC ranges into FP_SAFE / FP_LESS / FALLBACK and emits flat 24-byte CFI entries the BPF walker can binary-search.
• unwind/ehmaps/ — BPF map population + per-PID lifecycle (refcounted CFI tables, MmapWatcher consuming PERF_RECORD_MMAP2/EXIT/FORK to keep maps fresh through dlopen/exec).
• unwind/dwarfagent/ — Profiler / OffCPUProfiler for the DWARF path; shares a session struct (BPF objs, ringbuf reader, symbolizer, MMAP2 watcher). Eager-compile failures (Go binaries, no .eh_frame) are tolerated — the walker's FP fast path covers those.
• unwind/procmap/ (new in S9) — lazy per-PID /proc/<pid>/maps + ELF .note.gnu.build-id resolver. Keyed binary-search lookup feeds the pprof builder. MmapWatcher events invalidate cache entries through an observer callback on PIDTracker.Run.
• pprof/ — Frame gains Address/BuildID/MapStart/MapLimit/MapOff/IsKernel. New intern keys (mappingKey, locationKey, locationFallbackKey, functionKey) replace the name-only dedup. addLocation dispatches: kernel sentinel → JIT sentinel (Python/Node perf-maps) → resolver-driven address path → name-based fallback.
• BPF: bpf/perf_dwarf.bpf.c + bpf/offcpu_dwarf.bpf.c, sharing types via bpf/unwind_common.h. CFI consulted via HASH_OF_MAPS keyed by FNV-1a of GNU build-id.

--unwind flag selects the walker: fp (cheap, FP-only), dwarf (eager CFI compile + per-frame hybrid), auto (= dwarf; the hybrid walker covers FP-safe code via the FP path with no CFI lookup).

Notable behavior changes

• --unwind auto is now the default (previously fp). Most workloads see no change because the hybrid walker's FP fast path is byte-equivalent to pure-FP for FP-safe code; the difference shows up on FP-less binaries that previously truncated.
• pprof Function dedup is now per-binary (functionKey{MappingID, Name}). Same symbol name in two binaries (main.main in the target plus a vendored tool invoked via exec) is now two distinct profile.Function entries — pprof-correct, but a schema-compatible change.
• Mapping[0].File == "" no longer indicates "the binary" — it's a fallback placeholder, kept for synthetic [unknown] and resolver-miss frames. Real binaries get their own Mapping entries.

Test plan

Verified end-to-end on Fedora 43, kernel 6.19, on a setcap-capped binary (no sudo):

• make build — clean (defaulted GOTOOLCHAIN=auto so the Go 1.26 requirement doesn't fail on hosts with older Go).
• make test-unit — all unit packages pass, including new unwind/procmap tests (parser, build-id reader, Resolver lazy cache, InvalidateAddr observable behavior).
• Full integration suite — 18 PASS, 6 SKIP, 0 FAIL (5:34 runtime). Skipped tests load BPF in-process and require capping the test binary itself.
• S9 fidelity assertions on TestProfileMode (go-cpu/go-io/rust-cpu/python-cpu/python-io) and TestPerfAgentSystemWideDwarfProfile: every captured profile has ≥1 real (non-sentinel) Mapping with non-empty BuildID, and every user-space Location has non-zero file-offset Address.
• go tool pprof -raw <output>.pb.gz parses cleanly and shows real binary paths, build-ids, and addresses.

Operational notes

• Test capability gates now honor setcap on the perf-agent binary file (no sudo needed for the integration suite). Tests categorize as Category A (exec perf-agent — file caps suffice) or B (in-process BPF — test binary itself needs caps). See requireBPFRunnable in test/integration_test.go.
• AttachAllMappings is now best-effort: per-binary failures (Go executables lack .eh_frame) log + continue rather than aborting NewProfiler. The hybrid walker covers those binaries via the FP path at runtime.

Out of scope (follow-ups)

• S10 — pprof → LLVM sample-profile converter for Rust PGO. The originally stated end goal; deferred to a fresh session because the LLVM sample-profile format has its own design surface (line discriminators, weight calibration, body-sample vs callsite-sample emission).
• --unwind auto runtime-cost refinement (lazy compile / binary-detect heuristics). Spec at docs/unwind-auto-refinement-design.md. Recommendation: benchmark current eager-compile cost on real workloads first; only act if -a startup is painful.
• GPU profiling (full-stack CPU launch + device-side correlation). Draft spec at docs/superpowers/specs/2026-04-25-gpu-profiling-design.md.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a DWARF-capable hybrid stack unwinder and refactors pprof emission to preserve per-binary mapping identity and address-keyed locations (enabling higher-fidelity downstream consumers like sample-based PGO tooling).

Changes:

• Introduces DWARF/hybrid CPU + off-CPU profilers (BPF programs + Go wiring) with CFI table support and MMAP2-driven mapping lifecycle.
• Refactors pprof builder to accept rich Frame objects, intern real Mappings (path/start/limit/off/build-id), and key Locations by binary-relative address.
• Expands integration/unit tests and workloads (Rust + dlopen probe) and updates CLI/config defaults around --unwind.

Reviewed changes

Copilot reviewed 49 out of 111 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
unwind/dwarfagent/agent_test.go	Adds an end-to-end DWARF profiler test against the Rust workload.
unwind/dwarfagent/agent.go	Introduces DWARF-capable CPU profiler implementation and perf-event attachment.
test/workloads/rust/src/main.rs	Extends Rust workload with optional dlopen + probe function pointer path.
test/workloads/rust/probe/src/lib.rs	Adds a small cdylib probe function for dlopen/MMAP2 tracking tests.
test/workloads/rust/probe/Cargo.toml	Defines probe cdylib build configuration (debug symbols in release).
test/workloads/rust/Cargo.toml	Adds libc dependency to support dlopen/dlsym.
test/integration_test.go	Adds CAP_BPF gating, DWARF walker tests, and pprof fidelity assertions.
test/go.mod	Bumps Go version and promotes several deps to direct requirements for tests.
profile/profiler_test.go	Adds tests for address propagation into frames and resolver wiring.
profile/profiler.go	Switches to frame-based stacks, batches symbolization, and wires procmap resolver into pprof building.
profile/perf_dwarf_x86_bpfel.go	Adds generated bpf2go bindings for perf_dwarf (x86).
profile/perf_dwarf_test.go	Adds a verifier smoke test for loading perf_dwarf BPF.
profile/perf_dwarf_arm64_bpfel.go	Adds generated bpf2go bindings for perf_dwarf (arm64).
profile/offcpu_dwarf_x86_bpfel.go	Adds generated bpf2go bindings for offcpu_dwarf (x86).
profile/offcpu_dwarf_export.go	Exposes an API wrapper to load/use the offcpu_dwarf program and maps.
profile/offcpu_dwarf_arm64_bpfel.go	Adds generated bpf2go bindings for offcpu_dwarf (arm64).
profile/gen.go	Extends go:generate to build new DWARF BPF objects for amd64/arm64.
profile/dwarf_export.go	Exposes an API wrapper to load/use the perf_dwarf program and maps.
pprof/pprof_test.go	Updates tests to frame stacks; adds fidelity and perf-map decoder coverage.
pprof/pprof.go	Introduces Frame model, resolver-based mapping/location fidelity, and perf-map decoding.
perfagent/options.go	Adds config option + option helper for selecting unwind mode.
perfagent/agent.go	Dispatches to FP vs DWARF profilers based on --unwind, adds adapters.
offcpu/profiler.go	Refactors off-CPU symbolization to frame stacks and batched lookups.
main.go	Adds --unwind flag defaulting to auto.
internal/bpfstack/stack_test.go	Adds unit tests and benchmarks for parsing raw stackmap buffers.
internal/bpfstack/stack.go	Adds shared helper to decode stackmap IPs.
go.mod	Bumps module Go version to 1.26.0.
docs/unwind-auto-refinement-design.md	Adds a follow-up design doc for reducing auto mode startup cost.
docs/superpowers/specs/2026-04-25-s10-pgo-converter-idea.md	Adds idea note for future pprof→LLVM sample profile conversion.
docs/superpowers/specs/2026-04-25-gpu-profiling-design.md	Adds draft GPU profiling design spec (out of scope follow-up).
docs/superpowers/specs/2026-04-24-s9-pprof-fidelity-design.md	Adds design spec capturing the S9 fidelity work implemented here.
docs/dwarf-unwinding-design.md	Adds DWARF unwinding design spec (architecture and staging).
cmd/test_blazesym/main.go	Removes an old blazesym diagnostic binary.
bpf/unwind_common.h	Adds shared BPF types/maps/helpers for hybrid FP+DWARF unwinding.
bpf/perf_dwarf.bpf.c	Adds DWARF-capable CPU sampling BPF program emitting ringbuf records.
bpf/offcpu_dwarf.bpf.c	Adds DWARF-capable off-CPU BPF program for sched_switch-based sampling.
README.md	Updates docs to reflect DWARF/hybrid unwinding and pprof fidelity changes.
Makefile	Sets GOTOOLCHAIN=auto and extends workload/test targets (Rust probe + unwind tests).
.claude/settings.json	Adds editor plugin config (non-runtime).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Off-CPU symbolization drops the per-IP address, so pprof.Frame.Address remains 0 for all off-CPU frames. With the new pprof builder, Address=0 forces fallback location-keying (and prevents mapping/file-offset fidelity) even when ips are available. Fix by iterating symbols with an index and threading ips[i] into the frame conversion (mirroring profile/profiler.go’s blazeSymToFrames(s, ips[i]) approach).

[Copilot]

This test only gates on CAP_BPF, but loading/attaching the DWARF profilers also requires additional capabilities (e.g., PERFMON/SYS_PTRACE/CHECKPOINT_RESTORE, and memlock handling). As written, the test can fail (instead of skip) on systems where CAP_BPF is present but the other required caps are missing. Consider reusing the broader capability gate used elsewhere in the repo (or checking the full set needed by the loader) and handling a potential nil/failed cap.GetProc() defensively.

Suggested change

	// TestProfilerEndToEnd runs the full dwarfagent stack against the
	// rust-workload and asserts that the resulting pprof contains at
	// least one sample naming cpu_intensive_work.
	func TestProfilerEndToEnd(t *testing.T) {
	if os.Getuid() != 0 {
	caps := cap.GetProc()
	have, _ := caps.GetFlag(cap.Permitted, cap.BPF)
	if !have {
	t.Skip("requires root or CAP_BPF")
	}
	// haveRequiredProfilerCaps reports whether the current process has the
	// capabilities required to load and attach the DWARF profiler stack when
	// not running as root.
	func haveRequiredProfilerCaps() (bool, string) {
	if os.Getuid() == 0 {
	return true, ""
	}
	caps := cap.GetProc()
	if caps == nil {
	return false, "cannot inspect process capabilities"
	}
	required := []struct {
	value cap.Value
	name string
	}{
	{value: cap.BPF, name: "CAP_BPF"},
	{value: cap.PERFMON, name: "CAP_PERFMON"},
	{value: cap.SYS_PTRACE, name: "CAP_SYS_PTRACE"},
	{value: cap.CHECKPOINT_RESTORE, name: "CAP_CHECKPOINT_RESTORE"},
	{value: cap.IPC_LOCK, name: "CAP_IPC_LOCK"},
	}
	missing := make([]string, 0, len(required))
	for _, req := range required {
	have, err := caps.GetFlag(cap.Permitted, req.value)
	if err != nil || !have {
	missing = append(missing, req.name)
	}
	}
	if len(missing) > 0 {
	return false, "missing required capabilities: " + strings.Join(missing, ", ")
	}
	return true, ""
	}
	// TestProfilerEndToEnd runs the full dwarfagent stack against the
	// rust-workload and asserts that the resulting pprof contains at
	// least one sample naming cpu_intensive_work.
	func TestProfilerEndToEnd(t *testing.T) {
	if ok, reason := haveRequiredProfilerCaps(); !ok {
	t.Skipf("requires root or profiler capabilities: %s", reason)

[Copilot]

Using map[any]... for locations and functions weakens type safety and introduces a sharp edge: inserting a non-comparable key (even accidentally) will panic at runtime. Since these maps are internal and already have a small, closed set of key types, consider splitting them into strongly typed maps (e.g., map[locationKey]*profile.Location and map[locationFallbackKey]*profile.Location, similarly for functionKey) or introducing a single comparable tagged-key struct to keep compile-time guarantees.

[Copilot]

The comment references a staged rollout ('Task 8') and suggests Address might still be nonzero 'before then', but decodePerfMapFrame in this diff unconditionally zeros Address when it recognizes Python/Node formats. Updating this comment to describe the current behavior (without stage references) will keep the code documentation accurate.

[Copilot]

The generated bpf2go bindings import a package named structs via a bare import path. Unless this repo provides a top-level structs package/module with HostLayout, this will not compile. If this is intended, please ensure the structs package is present and in-module; otherwise, regenerate with the correct import path or adjust the generator configuration so the import resolves consistently in module builds.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 111 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

range over an int (ncpu) does not compile. Replace this with an indexed loop (e.g., for cpu := 0; cpu < ncpu; cpu++) to iterate CPUs.

Suggested change

	for cpu := range ncpu {
	for cpu := 0; cpu < ncpu; cpu++ {