datapath: Improved BPF testing framework

[dylandreimerink]

This PR adds a new BPF testing framework for datapath code. This
framework will allow us to write tests in C which will be executed,
as eBPF programs in the kernel. This allows us to test our code against
the actual kernel facilities and helper calls as well as confirming that
code will pass the verifier.

This framework differs from other related works in the sense that we do
the test setup, execution and verification all in C/eBPF code instead
of having to do half the work in userspace and the other half in eBPF.
A lightweight loader program is used to automatically detect test
programs in ELF files in a given directory, load, and execute them using
the BPF_PROG_TEST_RUN feature. The test results are passed back to the
loader which will convert them into sub-tests within the golang testing
framework to allow for easy integration into existing tooling.

Doing setup in C allows us to easily mock out code with #define
preprocessor tricks, replace tailcalls with stubs/code, and leverage
conditional testing depending on which flags are set / features enabled.
It also decouples datapath testing from the agent allowing us to verify
that the eBPF programs work as expected separately from the agent /
userspace.

Each ELF file can contain multiple CHECK's each of which will become
a separate program which can fail(test_fail/test_fail_now()) or pass
(default if not failed). Tests can log messages with test_log("msg")
for debugging purposes or as fail message(test_fatal("msg") logs and
test_fail_now()). These messages can include parameters analogous to
bpf_trace_printk, for example:
test_log("Expected 123, found: %llu", some_var). Or for compact
checks the assert(some_var == 123) style can be used which will
report the file and line of failed asserts and stop the test.

These check programs can also define sub-tests with a TEST("name", {
block which can pass/fail independently from the parent test. The appeal
is that such sub-tests run after each other in the exact same context
and can for example test different aspects of the same test setup(ctx,
map state).

Both CHECK and TEST can be skipped by calling test_skip()/
test_skip_now(), to mark a test as skipped, for example if applies to
a feature which is disabled or depends on features which are not
available in the current kernel version. Skipped tests are explicitly
marked as such instead of silently ignored. Tests can use #ifdef
preprocessor statements to determine to skip or not.

Tests that do not require tailcalls can do setup, execution and
pass/fail checking all in the CHECK program. Tests that do need to
tests across multiple tailcalls need to use a SETUP program in
addition, in which case the setup and execution will happen in the
SETUP program which will exit after the last tailcall is done. The
loader will run any SETUP program before the CHECK program of the
same name, the CHECK program will be invoked with the result context
of the SETUP program, the result number of the SETUP program will
be pretended to the context(4 bytes), this allows the CHECK program
to verify the return value, context and map state and determine a pass/
fail result.

Fixes: #18480

[joestringer]

It looked like others focused on the details of the test progs themselves, so I focused more on the build systems and general bits that were different from the way that other go tests work in the tree.

I explicitly didn't review the actual test content, because if you delete the full files and then add full files, it's much more difficult to review the diffs. If you're able to combine these in the same commit with minimal diff, that would make review easier for those pieces.

[joestringer]

Similar feedback here, we don't need yet another directory (test/bpf, bpf/tests, test/bpf_tests, bpf/tests/bpf_tests) to confuse people even more about where the code should live :-)

IIRC the rationale behind test/bpf/ vs. bpf/tests is the language distinction of Go vs. C, but if we have a reasonable way to satisfy whatever tooling is analyzing these directories then my preference would actually be to just put them all in the same directory. If the tooling doesn't like them being in the same directory (eg go tooling complaining about the C files) then we can keep two separate dirs.

[dylandreimerink]

Currently the distinction seems to be that /bpf is for C code and the rest for Go. You can't put Go and C files in the same directory since Go will automatically attempt to compile .c files as CGO, so any Go code would have to live in a separate directory anyway. Having 1 Go file in among all the C code felt strange, but I agree that it would be better to co-locate code that works together, perhaps not just for the tests but other parts of cilium as well

[joestringer]

Ah I see. Maybe we should move the unit-test.c (and elf-demo.c?) into bpf/ ..?

elf-demo.c I'm less sure about, IIRC that's actually used for the pkg/elf Go testing. Could potentially even move that one under something like test/elf.

Well anyways this is not such a big deal, we can always move the files later.

[joestringer]

🤔 I'm always suspicious about sleeps, is there a way we can flush the log or something instead?

[dylandreimerink]

cilium/ebpf currently doesn't provide this, though I agree it should, and I believe it is possible, I will see if I can fix that. Would it be acceptable to leave it like this and fix it as soon as a flush feature has been upstreamed in cilium/ebpf? (if that takes longer than this PR)

[joestringer]

Maybe a question first, what's the consequences if 50 Milliseconds is not enough time to flush the log? Can we easily detect that problem and react to it?

[dylandreimerink]

I guess the condition we are looking for is:

1. Did the test complete
2. Are we blocking

But we can't simply use a select to check if we are blocked, since it uses epoll. It would be great if we the library had a "check without blocking" function, but it doesn't. So to really fix it we need to update cilium/ebpf. Worst case if the reader does take longer than 50ms, is that we lose some debug information while in verbose mode, so I personally feel like this doesn't have to be a blocking issue as long as we improve this in a follow-up.

[dylandreimerink]

I have gotten the CI as far as I can take it. The only remaining complaint from checkpatch is about the macros used:

CHECK:MACRO_ARG_REUSE: Macro argument reuse 'ptr' - possible side-effects?
  #379: FILE: bpf/tests/common.h:28:
  +#define __bpf_log_arg1(ptr, arg) *(ptr++) = MKR_LOG_ARG; *(__u64 *)(ptr) = arg; ptr += sizeof(__u64)
  
  CHECK:MACRO_ARG_PRECEDENCE: Macro argument 'ptr' may be better as '(ptr)' to avoid precedence issues
  #379: FILE: bpf/tests/common.h:28:
  +#define __bpf_log_arg1(ptr, arg) *(ptr++) = MKR_LOG_ARG; *(__u64 *)(ptr) = arg; ptr += sizeof(__u64)

But I can't really "fix" these without altering/breaking the behavior of the macros