v0.9.8

What's Changed

• A few small fixes and clean ups by @anakryiko in #53
• bpftool: Pass additional compile flags as EXTRA_CFLAGS, not CFLAGS by @qmonnet in #52
• sidecar: Update addr2line dependency to 0.21 by @danielocfb in #55

New Contributors

• @danielocfb made their first contribution in #55

Full Changelog: v0.9.7...v0.9.8

retsnoop v0.9.7

What's Changed

• Add release workflow for shipping binaries and combined sources by @qmonnet in #42 and #44
• retsnoop: Fix sign extension failure logic by @anakryiko in #50
• retsnoop: Remove a limit of 4096 attachable functions by @anakryiko in #48
• retsnoop: Remove hard-coded maximum of 256 CPUs supported by @anakryiko in #49
• retsnoop: Use bpf_probe_read_kernel() instead of bpf_probe_read() by @iii-i in #46
• retsnoop: Pass the real envp to the sidecar by @erthalion in #47

New Contributors

• @qmonnet made their first contribution in #42
• @iii-i made their first contribution in #46
• @erthalion made their first contribution in #47

Full Changelog: v0.9.6...v0.9.7

retsnoop v0.9.6

What's Changed

• Fix calibration unreliability on some new kernels by @anakryiko in #41

Full Changelog: v0.9.5...v0.9.6

retsnoop v0.9.5

What's Changed

Massive improvements in how retsnoop determines whether kprobes are attachable:

• add --debug multi-kprobe mode to bisect failing multi-kprobe attachment; it quickly narrows down and logs which kprobes were attempted but failed to be attached;
• skip attaching to kernel functions that have non-unique name and some of instances are not traceable;
• resolve internal mix up of function and data ksyms;
• internal fixes to consistently take into account kernel module to which ksym/kprobe belongs to.

Overall, these fixes and improvements make retsnoop's mass-attach behavior more reliable.

Full Changelog: v0.9.4...v0.9.5

retsnoop v0.9.4

Bug fixes

• fix IP (instruction pointer) fetching on non-x86_64 architectures on older kernels;
• handle io_uring source code files better, after Linux code reorganization;
• automatically pick debugfs (/sys/kernel/debug/tracing) or tracefs (/sys/kernel/tracing), whichever is available;
• handle very old kernels that don't support BPF global data more gracefully.

retsnoop v0.9.3

What's Changed

retsnoop now supports DWARF-based symbolization (i.e.,
source code file/line info and inline functions) on
KASLR-enabled Linux kernels.

retsnoop v0.9.2

What's Changed

• -F (fentry/fexit) mode now supports tracing void-returning functions;
• few fixes for -F (fentry/fexit) mode interacting weirdly with source code globs;
• retsnoop now can be compiled across x86_64, i686, aarch64, ppc64le, s390x, riscv64 architectures by using per-architecture pre-generated minimal vmlinux.h (see gen-vmlinux-headers.sh script);
• retsnoop now builds bootstrap (lightweight) version of bpftool from submodule, which allows it to be compilable on multiple-architectures. Previously retsnoop's Makefile relied on checked in pre-built x86_64 bpftool binary;
• massive revamp of README.md;
• usage text fixes and improvements.

Full Changelog: v0.9.1...v0.9.2

retsnoop v0.9.1

Few nice improvements with no major new features:

• dropped the requirement for /proc/config.gz presence for multi-kprobe detection (just using BPF CO-RE now for detection);
• use dynamically allocated internal formatting buffers for stacks and traces, thus allowing much larger traces without dropping any information (at the expense of more memory usage, of course);
• force-flush stdout before (potentially very long) detachment to improve retsnoop usage in scripts;
• emit detected features when printing version and --verbose flag is specified:

$ sudo ./retsnoop -Vv
retsnoop v0.9.1
Feature detection:
        BPF ringbuf map supported: yes
        bpf_get_func_ip() supported: yes
        bpf_get_branch_snapshot() supported: yes
        BPF cookie supported: yes
        multi-attach kprobe supported: yes
Feature calibration:
        kretprobe IP offset: 4
        fexit sleep fix: yes
        fentry re-entry protection: yes

All just nice quality of life improvements. Enjoy!

retsnoop v0.9

--trace (-T) function calls trace mode

Add function call trace output, in addition to default stack trace and LBR output.

Example:

$ sudo ./retsnoop -e '*sys_bpf' -v -n simfail -a ':kernel/bpf/syscall.c' -a ':kernel/bpf/verifier.c' -T
...
Receiving data...
15:50:12.413878 -> 15:50:12.414193 TID/PID 1755152/1755152 (simfail/simfail):

FUNCTION CALLS TRACE                    RESULT                 DURATION
-------------------------------------   --------------------  ---------
→ __x64_sys_bpf
    → __sys_bpf
        ↔ bpf_check_uarg_tail_zero      [0]                     0.341us
        → bpf_raw_tracepoint_open
            ↔ __bpf_prog_get            [0xffffc9000c93d000]    0.255us
            → bpf_tracing_prog_attach
                ↔ bpf_link_prime        [0]                     2.530us
                ↔ bpf_link_cleanup      [void]                  3.435us
            ← bpf_tracing_prog_attach   [-ENOTSUPP]           306.161us
        ← bpf_raw_tracepoint_open       [-ENOTSUPP]           310.147us
    ← __sys_bpf                         [-ENOTSUPP]           314.846us
← __x64_sys_bpf                         [-ENOTSUPP]           315.515us

                      entry_SYSCALL_64_after_hwframe+0x44  (arch/x86/entry/entry_64.S:112:0)
                      do_syscall_64+0x2d                   (arch/x86/entry/common.c:46:12)
   315us [-ENOTSUPP]  __x64_sys_bpf+0x1c                   (kernel/bpf/syscall.c:4749:1)
   314us [-ENOTSUPP]  __sys_bpf+0x867                      (kernel/bpf/syscall.c:4689:9)
   310us [-ENOTSUPP]  bpf_raw_tracepoint_open+0x9a         (kernel/bpf/syscall.c:3063:6)
!  306us [-ENOTSUPP]  bpf_tracing_prog_attach

As you can see from the above, function calls trace mode allows to peer into exact control flow inside the kernel, but filter to according to allow/deny lists, taking into account all the filters (process name, latency, etc). In addition to call sequence, function results and duration is emitted.

Note that leaf function calls (e.g., bpf_link_prime above) are collapsed, if they don't call any other functions. This makes call trace more readable and compact. This is marked with ↔ marker, while otherwise function entry is marked with →, and function exit is marked with ←.

This mode is perfectly augments stack trace output for deeper kernel internals inspection, but also is great for discovering how kernel internals work, in general.

retsnoop v0.8.3

Add ability to filter functions by kernel modules.

General glob format is now <name-glob> [<module-glob>], where module glob is optional. This allows to, e.g., attach to all kprobes within some module: '* [fuse]' will attach to all the functions within fuse module. Note that module glob is also a glob, so one can capture multiple modules within one glob, e.g. -a '* [kvm*]' will capture functions defined in kvm and kvm_intel modules.