Benchmark for tracee

[liusy58]

Hi, I know tracee uses eBPF to trace a lot syscall event, so how to measure the cost of these eBPF functions? I am wondering if you have some benchamrks to test it. Any help is appreciated!

[rafaeldtinoco]

#!/bin/bash

# sysctl -w kernel.bpf_stats_enabled=1

bpftool prog show > /tmp/progshow.$$

checkfile() {
    prognum=$(echo $line | cut -d' ' -f1)
    progname=$(echo $line | cut -d' ' -f2)

    runtime=$(cat /tmp/progshow.$$ | grep ^$prognum | awk '{print $9}')
    amount=$(cat /tmp/progshow.$$ | grep ^$prognum | awk '{print $11}')
    if [[ $runtime -eq 0 ]]; then runtime=1; fi
    if [[ $amount -eq 0 ]]; then amount=1; fi
    average=$((runtime/amount))

    echo "PROGRAM: $progname (runtime: $runtime ns, amount: $amount times, average: $average ns)"

}

# kprobes
bpftool perf | grep kprobe | awk '{print $6" "$9}' | \
while read line; do
    checkfile
done

# traces
bpftool perf | grep trace | awk '{print $6" "$8}' | \
while read line; do
    checkfile
done

rm -f /tmp/progshow.$$

Hope it helps.

[rafaeldtinoco]

I'm not sure what you mean. What environment? An empty hook handler just with "return" and then add logic to it (bpf helpers, bpf map updates/queries) and compare ?

[liusy58]

Not yet. Takte the syscall__execve eBPF for example, we need to call execve syscall to trigger this eBPF function, right? So I think I need to call many execve to calculate the cost?

[liusy58]

Not sure whther https://aquasecurity.github.io/tracee/v0.15/getting-started/kubernetes-quickstart/ is what I am looking for.

[rafaeldtinoco]

Yep, but it would be better to also filter for the syscall calling context (like as long as the caller of the syscall is from UID X, or else return). Then you can control the workload generator as well. It is not perfect, but it will "tend" to real numbers if you make if often (and pin the workload to a single CPU, disable RCU callbacks, NOHZ, etc).

Other way would be to instrument enter/exit of each handler with a TOD monotonic clock getter like bpf_ktime_get_ns() but that would disturb measurements a bit I suppose. I remember I saw some projects instrumenting LLVM do generate those automatically when doing the eBPF bytecode (and picking values from managed maps).

Anyway, multiple ways of solving the issue, the script is just an idea for a quick help

[liusy58]

Yep, in fact I have successfully instrumented them. Now I cannot find a suitable workload to do some replay...