Missing a lot of syscall probes in bpftrace for RPI

[OevreFlataeker]

When I do a sudo bpftrace -l | grep syscalls | wc -l on an Ubuntu 22.04. machine I get 694 tracepoints which cover syscalls.
When I execute the same command on an RPI3 I only get 2 results:

pi@rpi3:~ $ sudo bpftrace -l | grep syscalls
tracepoint:raw_syscalls:sys_enter
tracepoint:raw_syscalls:sys_exit

I cross compiled bpftrace on the Ubuntu machine as bpftrace wasn't available in Raspbian bookworm as far as I understood.
I followed the generic best practices to setup a x-compile environment and everything worked so far.

This is the Output of the cmake run

daubsi@playbox:~/bpftrace$ cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=cmake/armv8-rpi-linux-gnueabihf.cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_TESTING=OFF
-- The C compiler identification is GNU 12.3.0
-- The CXX compiler identification is GNU 12.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/daubsi/opt/x-tools/armv8-rpi3-linux-gnueabihf/bin/armv8-rpi3-linux-gnueabihf-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/daubsi/opt/x-tools/armv8-rpi3-linux-gnueabihf/bin/armv8-rpi3-linux-gnueabihf-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found ZLIB: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/lib/arm-linux-gnueabihf/libz.a (found version "1.2.13")
-- Found LibBcc: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/lib/arm-linux-gnueabihf/libbcc.so
-- Looking for bcc_usdt_foreach
-- Looking for bcc_usdt_foreach - found
-- Found LibBpf: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/lib/arm-linux-gnueabihf/libbpf.so
-- Performing Test HAVE_LIBBPF_UPROBE_MULTI
-- Performing Test HAVE_LIBBPF_UPROBE_MULTI - Failed
-- Found LibElf: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/lib/arm-linux-gnueabihf/libelf.so
-- Found LibCereal: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/include
-- Found BISON: /usr/bin/bison (found version "3.8.2")
-- Found FLEX: /usr/bin/flex (found version "2.6.4")
-- Looking for name_to_handle_at
-- Looking for name_to_handle_at - found
-- Found LibBfd: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/lib/arm-linux-gnueabihf/libbfd.so
-- Found LibOpcodes: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/lib/arm-linux-gnueabihf/libopcodes.so
-- Performing Test LIBBFD_DISASM_FOUR_ARGS_SIGNATURE
-- Performing Test LIBBFD_DISASM_FOUR_ARGS_SIGNATURE - Success
-- Performing Test LIBBFD_INIT_DISASM_INFO_FOUR_ARGS_SIGNATURE
-- Performing Test LIBBFD_INIT_DISASM_INFO_FOUR_ARGS_SIGNATURE - Success
-- Please install the libcap development package (missing: LIBPCAP_LIBRARIES LIBPCAP_INCLUDE_DIRS)
-- Looking for sys/sdt.h
-- Looking for sys/sdt.h - found
-- Performing Test HAVE_FFI_CALL
-- Performing Test HAVE_FFI_CALL - Success
-- Found FFI: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/lib/arm-linux-gnueabihf/libffi.so
-- Performing Test Terminfo_LINKABLE
-- Performing Test Terminfo_LINKABLE - Success
-- Found Terminfo: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/lib/arm-linux-gnueabihf/libtinfo.a
-- Found LibXml2: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/lib/arm-linux-gnueabihf/libxml2.so (found version "2.9.14")
-- Found LLVM 14.0.6: /home/daubsi/chroots/rpi3-bookworm-armhf/usr/lib/llvm-14/lib/cmake/llvm
-- Please install the lldb development package (missing: LLDB_LIBRARIES LLDB_INCLUDE_DIRS)
-- Using SYSTEM_INCLUDE_PATHS=auto
-- Configuring done (1.5s)
-- Generating done (0.0s)
-- Build files have been written to: /home/daubsi/bpftrace/build

Did I miss something during the configure step that all those tracepoints have not been found?

[ajor]

bpftrace isn't compiled with knowledge of what probes are available - it discovers them at runtime from the kernel. I think the most likely explanation is that your Raspbian kernel doesn't make the "syscalls" tracepoints available (but it does provide "raw_syscalls" tracepoints).

You can see what tracepoints are available on your system by looking in this directory: /sys/kernel/tracing/events
You should see a subdirectory called raw_syscalls. Is there another subdirectory called syscalls, or is it missing?

[OevreFlataeker]

Indeed there is no such directory "syscall"!

oot@rpi3:/sys/kernel/tracing/events# ls
alarmtimer    compaction    fib6	  i2c	    lock	net		power	      rpm     sunrpc		vmscan
block	      cpuhp	    filelock	  initcall  maple_tree	netfs		preemptirq    rseq    task		watchdog
bpf_test_run  dev	    filemap	  io_uring  mdio	netlink		printk	      rtc     tcp		workqueue
bpf_trace     dma_fence     fscache	  iomap     migrate	nfs		pwm	      sched   thermal		writeback
bridge	      drm	    ftrace	  ipi	    mmap	nfs4		qdisc	      scsi    thermal_pressure	xdp
cachefiles    enable	    gadget	  irq	    mmap_lock	oom		raw_syscalls  signal  timer
cfg80211      error_report  gpio	  iscsi     mmc		page_isolation	rcu	      skb     tlb
cgroup	      ext4	    header_event  jbd2	    module	page_pool	regmap	      smbus   udp
clk	      f2fs	    header_page   kmem	    napi	pagemap		regulator     sock    v4l2
cma	      fib	    hwmon	  kyber     neigh	percpu		rpcgss	      spi     vb2
root@rpi3:/sys/kernel/tracing/events# ls raw_syscalls/
enable	filter	sys_enter  sys_exit

The kernel is

Linux rpi3 6.1.21-v7+ #1642 SMP Mon Apr 3 17:20:52 BST 2023 armv7l GNU/Linux

So the Raspberry Pi kernel did explicitly leave out this kernel option to expose the syscalls you presume?

[OevreFlataeker]

I extracted the kernel config of the running kernel via:

sudo modprobe configs; zcat /proc/config.gz > /tmp/kernel.config
grep TRACEPOINTS /tmp/kernel.config 
CONFIG_TRACEPOINTS=y
# CONFIG_FTRACE_SYSCALLS is not set
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_PREEMPTIRQ_TRACEPOINTS=y

Is there more needed to get the syscall tracepoints in the directory above? Is the FTRACE line relevant?

[OevreFlataeker]

Just found... seems this answers the question....
https://gitlab.alpinelinux.org/alpine/aports/-/issues/16077

[viktormalik]

Thanks, this is a useful reference! Marking it as the answer and closing the discussion.