runtime test suite

[brendangregg]

The /tests currently test parsing and BPF code generation, but does not test tracing functionality at runtime (actually running bpftrace and tracing an event). This is necessary to ensure everything actually works. It's a little tricky, as any given event (kprobe, tracepoint, etc) may or may not fire on its own, and may need to be triggered by the test suite.

I could imagine using the tracepoint:syscalls:sys_exit_nanosleep event as much as possible, as it's a stable API and can be easily triggered (from the shell: sleep 1).

Here's a very basic hacky example:

test=pid; sleep 1 & sleep 15 & ./src/bpftrace -e 'tracepoint:syscalls:sys_exit_nanosleep { printf("SUCCESS '$test' %d\n", pid); exit(); }' | grep '^SUCCESS '$test' [0-9][0-9]*$' || echo "FAILURE $test"
test=uid; sleep 1 & sleep 15 & ./src/bpftrace -e 'tracepoint:syscalls:sys_exit_nanosleep { printf("SUCCESS '$test' %d\n", uid); exit(); }' | grep '^SUCCESS '$test' [0-9][0-9]*$' || echo "FAILURE $test"
test=reg; sleep 1 & sleep 15 & ./src/bpftrace -e 'tracepoint:syscalls:sys_exit_nanosleep { printf("SUCCESS '$test' %d\n", reg("ip")); exit(); }' | grep '^SUCCESS '$test' [0-9][0-9]*$' || echo "FAILURE $test"

The idea of spawning a "sleep 1" and a "sleep 15" is to allow the test to complete quickly (sleep 1), but if bpftrace was slow to initialize, then there is a 15 second timeout (or shorter, since prior tests that complete in 1 second leave 15 second sleepers still running).

Running these produces the output

SUCCESS pid 1446
SUCCESS uid 0
SUCCESS reg 0

So the output says "SUCCESS" or "FAILURE", the name of the test, and some argument to aid debugging.

I think it would be better to write this in C++, or C, or Perl, or Python, and have better control over the sleeps.

The functionality that should be tested should be everything from the reference guide.

[brendangregg]

I'll paste some commands I've run to test various things in development. These can be put in the runtime test suite:

bpftrace -e 'BEGIN { @aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa = 1; exit(); }'

bpftrace -e 'BEGIN { $x = -123; printf("%d %lld %lld\n", $x, $x, - $x); exit(); }'
bpftrace -e 'BEGIN { $x = -2; printf("%d\n", $x >= 0 ? 0 : - $x); exit();  }'

bpftrace -e 'kprobe:vfs_read { @ = count(); } interval:s:1 { print(@); clear(@); exit(); }'
bpftrace -e 'kprobe:vfs_read { @a = count(); } interval:s:1 { print(@a); clear(@a); exit(); }'
bpftrace -e 'kprobe:vfs_read { @ab = count(); } interval:s:1 { print(@ab); clear(@ab); exit(); }'
bpftrace -e 'kprobe:vfs_read { @abc = count(); } interval:s:1 { print(@abc); clear(@abc); exit(); }'
bpftrace -e 'kprobe:vfs_read { @abcd = count(); } interval:s:1 { print(@abcd); clear(@abcd); exit(); }'
bpftrace -e 'kprobe:vfs_read { @abcde = count(); } interval:s:1 { print(@abcde); clear(@abcde); exit(); }'
bpftrace -e 'kprobe:vfs_read { @abcdef = count(); } interval:s:1 { print(@abcdef); clear(@abcdef); exit(); }'
bpftrace -e 'kprobe:vfs_read { @abcdefg = count(); } interval:s:1 { print(@abcdefg); clear(@abcdefg); exit(); }'
bpftrace -e 'kprobe:vfs_read { @abcdefgh = count(); } interval:s:1 { print(@abcdefgh); clear(@abcdefgh); exit(); }'
bpftrace -e 'kprobe:vfs_read { @abcdefghi = count(); } interval:s:1 { print(@abcdefghi); clear(@abcdefghi); exit(); }'
bpftrace -e 'kprobe:vfs_read { @abcdefghij = count(); } interval:s:1 { print(@abcdefghij); clear(@abcdefghij); exit(); }'
bpftrace -e 'kprobe:vfs_read { @abcdefghijk = count(); } interval:s:1 { print(@abcdefghijk); clear(@abcdefghijk); exit(); }'
bpftrace -e 'kprobe:vfs_read { @abcdefghijkl = count(); } interval:s:1 { print(@abcdefghijkl); clear(@abcdefghijkl); exit(); }'

I'll keep adding here when I remember.

[mmarchini]

Maybe we should mark this as priority to help us identify regressions like #103 sooner?

Also, it would be nice to have a CI for the project :)

[brendangregg]

Yeah, I'd thought about making this priority. And please file a separate ticket for CI.

I'll keep adding one-liners here for the runtime test. Just something basic that can run a bunch of one-liners and see if they fail would be great. bcc has a test suite for running tools, I haven't looked into it to see if that's the same approach we want.

I suppose each one-liner can end with 'interval:ms:50 { exit(); }', to ensure it exits quickly.

[brendangregg]

More one-liners:

bpftrace -e 'tracepoint:syscalls:sys_enter_r* { @[name] = count(); }'
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @ = count(); } interval:s:1 { print(@); clear(@); }'
bpftrace -e 'tracepoint:syscalls:sys_enter_open { @a = count(); } tracepoint:syscalls:sys_enter_open { @b = count(); }'

[williangaspar]

Should it run with the other tests or separated? The test suite takes about 20s to run.
Right now I'm running them with:

cd tests/python
sudo python -m unittest discover --pattern=*.py

[ajor]

I think it should have it's own separate way of running the tests as these new ones will require root and probably won't work in the CI