Moving to libbpf

[yanivagman]

I recently opened a PR making major changes to tracee:
#250

This PR removes the dependancy of Tracee on gobpf and bcc, and instead moves to using libbpf (which is part of the linux kernel sources).

As such, I want us to merge this change only after we gain some confidence that it works on as many environments as possible.

My request is simple, and I'll appreciate if you can pull the changes from the above branch, and report back if you notice any change - whether it's compilation issues, bpf verifier issues, performance issues, or features not working. If everything works - please report as well.
If you can, please also provide details about the environment you are using - distro name, kernel version and clang version used to compile tracee

Thanks

[itaysk]

Updating that we have created a branch called libbpf for ongoing work on this.
Even if you just tell us "it works for me" it will be helpful.

git fetch origin
git checkout libbpf
make build
sudo tracee

[grantseltzer]

I'd add git submodule update --init to this list before make build

[grantseltzer]

Upon running the above commands:

...
clang -S \
	-D __BPF_TRACING__ -D __KERNEL__ -D__TARGET_ARCH_x86 \
	-I dist/libbpf/usr/include \
	-include /lib/modules/5.8.14-300.fc33.x86_64/build/include/linux/kconfig.h \
	-I /lib/modules/5.8.14-300.fc33.x86_64/build/arch/x86/include \
	-I /lib/modules/5.8.14-300.fc33.x86_64/build/arch/x86/include/uapi \
	-I /lib/modules/5.8.14-300.fc33.x86_64/build/arch/x86/include/generated \
	-I /lib/modules/5.8.14-300.fc33.x86_64/build/arch/x86/include/generated/uapi \
	-I /lib/modules/5.8.14-300.fc33.x86_64/build/include \
	-I /lib/modules/5.8.14-300.fc33.x86_64/build/include/uapi \
	-I /lib/modules/5.8.14-300.fc33.x86_64/build/include/generated \
	-I /lib/modules/5.8.14-300.fc33.x86_64/build/include/generated/uapi \
	-I 3rdparty/include  \
	-Wno-address-of-packed-member \
	-Wno-compare-distinct-pointer-types \
	-Wno-deprecated-declarations \
	-Wno-gnu-variable-sized-type-not-at-end \
	-Wno-pointer-sign \
	-Wno-pragma-once-outside-heade \
	-Wno-unknown-warning-option \
	-Wno-unused-value \
	-Wunused \
	-Wall \
	-fno-stack-protector \
	-fno-jump-tables \
	-fno-unwind-tables \
	-fno-asynchronous-unwind-tables \
	-x c \
	-nostdinc \
	-O2 -emit-llvm -c -g tracee/tracee.bpf.c -o dist/tracee.bpf.ll
<built-in>:1:10: fatal error: '/lib/modules/5.8.14-300.fc33.x86_64/build/include/linux/kconfig.h' file not found
#include "/lib/modules/5.8.14-300.fc33.x86_64/build/include/linux/kconfig.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [Makefile:61: dist/tracee.bpf.o] Error 1

fedo grant in ~> tracee on ~> libbpf
[*] uname -a                                                                                                                                                                                                                              2 ↵
Linux localhost.localdomain 5.8.14-300.fc33.x86_64 #1 SMP Wed Oct 7 21:44:23 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

At least on my system (Fedora 33) /lib/modules/<kernel>/build is a link to /usr/src/kernels/<kernel>, perhaps there's an issue related to evaluating that?

[yanivagman]

@grantseltzer the kernel headers package version and the running kernel version should match. This is why the build fails.

Regrading your question - at this stage we don't generate vmlinux.h, and require the user to supply the correct kernel headers.
The reason for that, is that generating vmlinux.h requires the kernel to have BTF support (CONFIG_DEBUG_INFO_BTF), or generating it using pahole with the host's vmlinux image. As many kernels not yet configured with BTF at this stage, we don't want to limit the range of supported kernels. We also don't want to request the user to supply the vmlinux image as required when generating vmlinux.h using pahole.

For this reason, we currently move to libbpf, but not yet support CO-RE. This is a mid-term solution until kernels with BTF support will become mainstream.

[grantseltzer]

Ah very interesting, that makes sense. I'm upgrading my kernel at the moment and will report back when my headers package matches my running kernel version. Though it may be good to think of a workaround for this as the linux-headers package doesn't get a new version for every new kernel version.

[yanivagman]

I still wonder how bcc tools worked on your environment.
If kernel and headers didn't match, and as you say your kernel is not compiled with CONFIG_IKHEADERS - how was bcc able to compile the code? It must have got the correct headers somehow...

[grantseltzer]

I believe CONFIG_IKHEADERS=m was enabled though the /sys/kernel/kheaders.tar.xz did not exist. Whic bcc tools are you talking about compiling specifically?

[grantseltzer]

Ok upgraded kernel to 5.9.2 with correct headers and everything built/ran perfectly :)

[yanivagman]

Libbpfgo is now part of the master branch.
Any issue related to this, should now be opened as a regular issue