btfgen: Embed btf generated files into the binary #2044

mauriciovasquezbernal · 2023-09-15T15:59:04Z

This commit changes the approach regarding btfgen, now the btf generated files are embeded in the golang binary, which means that now it's also available for ig.

This also updates the CI to execute btfgen before building ig or ig-k8s.

How to test

Getting a host without BTF

Ironically it's difficult to get a machine without BTF support these days. I used a ubuntu focal64 through vagrant (config.vm.box = "ubuntu/focal64"). It ships a kernel version that already has btf support, hence I installed another kernel that doesn't include BTF and rebooted the machine:

$ uname -r
5.4.0-99-generic
$ sudo apt-get install linux-image-5.8.0-45-generic -y
...

After reboot

$ uname -r 
5.8.0-45-generic

$ cat /boot/config-$(uname -r) | grep -i btf
CONFIG_VIDEO_SONY_BTF_MPX=m
# CONFIG_DEBUG_INFO_BTF is not se

Install docker (it should not be required, I'll open an issue soon about this)

$ sudo apt-get install docker.io -y

Before this PR

$ make ig-linux-amd64
...

$ ls -lh ig-linux-amd64 
-rwxr-xr-x 1 mvb mvb **102M** Sep 15 11:13 ig-linux-amd64

$ sudo ./ig-linux-amd64 trace open --host
WARN[0000] Failed to initialize BTF: open btfs/x86/ubuntu/20.04/x86_64/5.8.0-45-generic.btf: file does not exist 
Error: initializing operators: initializing operator "LocalManager": initializing manager: starting container fanotify: readPrivateDataFromFd: loading maps and programs: field IgFgetX: program ig_fget_x: apply CO-RE relocations: load kernel spec: no BTF found for kernel version 5.8.0-45-generic: not supported

After this PR

Option A: Locally compiling ig with btf enabled

$ make ENABLE_BTFGEN=true ig-linux-amd64 -j$(nproc)
#  some minutes after 

$ ls -lh ig-linux-amd64 
-rwxr-xr-x 1 mvb mvb **104M** Sep 15 11:09 ig-linux-amd64
                       ^^^ binary is only ~2MB bigger.

$ sudo ./ig-linux-amd64 trace open --host
RUNTIME.CONTAINERNAME                                      PID        COMM             FD    ERR PATH                                                         
                                                           3936       ig-linux-amd64   44    0   /proc                                                        
                                                           3936       ig-linux-amd64   44    0   /proc                                                        
                                                           3936       ig-linux-amd64   44    0   /proc                                                        
                                                           572        multipathd       9     0   /sys/devices/pci0000:00/0000:00:14.0/host2/target2:0:0/2:0:0…
                                                           572        multipathd       9     0   /sys/devices/pci0000:00/0000:00:14.0/host2/target2:0:0/2:0:0…
                                                           416        systemd-journal  37    0   /proc/572/status

Option B: Getting ig from a CI run

ig is compiled in the CI with btfgen enabled, hence downloading the ig binary from a run of this PR should work too.

Fixes #752

eiffel-fl

I like the idea as we only get the BTF files at compile time and we will use the good one at runtime only if ENABLE_BTFGEN was set a compile time and if current kernel does not have BTF.

I would advice to split the commit by having one which adds the btfgen package and the other using it.
I will test it later though.

.github/workflows/inspektor-gadget.yml

pkg/btfgen/btfgen.go

mauriciovasquezbernal · 2023-09-18T18:51:29Z

I would advice to split the commit by having one which adds the btfgen package and the other using it.

I wanted to do that but given it involves changes to the CI, makefile and gadges, it's difficult to split in logic pieces that work alone, so I prefer to have a big commit with all changes in.

eiffel-fl

Sort of tested it on custom built kernel but as btfhub does not know my kernel this of course does not work:

root@vm-amd64:~# zgrep DEBUG_INFO /proc/config.gz 
CONFIG_DEBUG_INFO_NONE=y
# CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
root@vm-amd64:~# ./share/kinvolk/inspektor-gadget/ig-linux-amd64 trace open --host
Error: initializing operators: initializing operator "LocalManager": initializing manager: starting container fanotify: readPrivateDataF0
root@vm-amd64:~# more /tmp/foo 
Error: initializing operators: initializing operator "LocalManager": initializin
g manager: starting container fanotify: readPrivateDataFromFd: private_data is 0
root@vm-amd64:~# uname -a
Linux vm-amd64 6.5.0-rc7-00021-ga6fda6d4d5fc #96 SMP PREEMPT_DYNAMIC Tue Sep 19 10:12:26 EEST 2023 x86_64 GNU/Linux

I will create the corresponding BTF file for this kernel and test again but the logic seems to be good.

eiffel-fl

It works like a charm! Awesome job:

# Just removed the one about map.
$ bpftool gen min_core_btf ../../linux/linux/.tmp_vmlinux.btf foo.btf $(find . -name '*.o')
$ mv foo.btf pkg/btfgen/btfs/x86/debian/11/x86_64/6.5.0-rc7-00021-ga6fda6d4d5fc.btf
# Inside VM
root@vm-amd64:~# uname -a
Linux vm-amd64 6.5.0-rc7-00021-ga6fda6d4d5fc #98 SMP PREEMPT_DYNAMIC Tue Sep 19 14:13:16 EEST 2023 x86_64 GNU/Linux
root@vm-amd64:~# zgrep DEBUG_INFO /proc/con
config.gz  consoles   
root@vm-amd64:~# zgrep DEBUG_INFO /proc/config.gz 
CONFIG_DEBUG_INFO_NONE=y
# CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
root@vm-amd64:~# ./share/kinvolk/inspektor-gadget/ig-linux-amd64 trace open --host | head
RUNTIME.CONTAINERNAME          PID              COMM             FD  ERR PATH                            
                               283              ig-linux-amd64   43  0   /proc                           
                               283              ig-linux-amd64   43  0   /proc                           
                               283              ig-linux-amd64   43  0   /proc                           
                               283              ig-linux-amd64   43  0   /proc                           
                               283              ig-linux-amd64   43  0   /sys/kernel/tracing/events/sysc…
                               283              ig-linux-amd64   43  0   /proc                           
                               283              ig-linux-amd64   43  0   /proc                           
                               283              ig-linux-amd64   43  0   /proc                           
                               283              ig-linux-amd64   43  0   /proc                           
root@vm-amd64:~#

I am wondering if some documentation which links to a tutorial regarding bpftool gen min_core_btf would help?

mauriciovasquezbernal · 2023-09-19T15:21:01Z

I am wondering if some documentation which links to a tutorial regarding bpftool gen min_core_btf would help?

I wrote this blog post some time ago -> https://www.inspektor-gadget.io//blog/2022/03/btfgen-one-step-closer-to-truly-portable-ebpf-programs/, however it's not clear for me where to link this in IG documentation.

eiffel-fl · 2023-09-20T06:23:41Z

I am wondering if some documentation which links to a tutorial regarding bpftool gen min_core_btf would help?

I wrote this blog post some time ago -> https://www.inspektor-gadget.io//blog/2022/03/btfgen-one-step-closer-to-truly-portable-ebpf-programs/, however it's not clear for me where to link this in IG documentation.

I was refeering to this blog post.
I do not know where you can put, maybe in devel.md.

alban

Initial comments. I'll continue soon.

pkg/btfgen/btfgen.go

alban

One blocking comment about the network dispatcher.

The rest looks good to me.

I didn't review the CI part.

alban · 2023-09-20T15:31:00Z

Makefile.btfgen

 # Filter out BPF objects that only contain BPF maps without BPF programs
-BPF_PROGS_O_FILES = $(filter-out pkg/gadgets/trace/network/tracer/graphmap_bpfel%,$(BPF_ALL_O_FILES))
+BPF_PROGS_O_FILES = $(filter-out pkg/gadgettracermanager/containers-map/containersmap_bpfel%,$(BPF_ALL_O_FILES))


Similarly to the discussion in #2003 (comment), the map from the pkg/gadgettracermanager/containers-map package could be created programmatically.

But that can be in a different PR.

Right, the difference is that in this case providing BTF information manually for the map is not that easy, and in this case it could be appreciated to pretty print the contents of the map while debugging.

pkg/gadgets/internal/networktracer/tracer.go

alban · 2023-09-20T15:35:33Z

pkg/gadgets/internal/socketenricher/tracer.go

@@ -73,6 +74,12 @@ func (se *SocketEnricher) start() error {

 	kallsyms.SpecUpdateAddresses(specIter, []string{"socket_file_ops"})

+	opts := ebpf.CollectionOptions{
+		Programs: ebpf.ProgramOptions{
+			KernelTypes: btfgen.GetBTFSpec(),


Happy to see that the socket enricher can use btfgen!

This commit changes the approach regarding btfgen, now the btf generated files are embeded in the golang binary, which means that now it's also available for ig. This also updates the CI to execute btfgen before building ig or ig-k8s. Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

mauriciovasquezbernal requested review from eiffel-fl, alban, blanquicet and mqasimsarfraz as code owners September 15, 2023 15:59

mauriciovasquezbernal linked an issue Sep 15, 2023 that may be closed by this pull request

local-gadget: Use BTFGen #752

Closed

mauriciovasquezbernal force-pushed the mauricio/btfgen-ig branch 3 times, most recently from 2c3c81d to 33dd5c8 Compare September 15, 2023 18:03

eiffel-fl reviewed Sep 18, 2023

View reviewed changes

.github/workflows/inspektor-gadget.yml Outdated Show resolved Hide resolved

pkg/btfgen/btfgen.go Outdated Show resolved Hide resolved

Base automatically changed from mauricio/fix-ig-image-artifact to main September 18, 2023 15:36

mauriciovasquezbernal force-pushed the mauricio/btfgen-ig branch from 33dd5c8 to f35098d Compare September 18, 2023 18:50

eiffel-fl reviewed Sep 19, 2023

View reviewed changes

eiffel-fl approved these changes Sep 19, 2023

View reviewed changes

mauriciovasquezbernal force-pushed the mauricio/btfgen-ig branch from f35098d to d75a044 Compare September 19, 2023 15:19

mauriciovasquezbernal force-pushed the mauricio/btfgen-ig branch from d75a044 to 0fe1ba2 Compare September 20, 2023 13:08

alban reviewed Sep 20, 2023

View reviewed changes

pkg/btfgen/btfgen.go Outdated Show resolved Hide resolved

pkg/btfgen/btfgen.go Outdated Show resolved Hide resolved

alban reviewed Sep 20, 2023

View reviewed changes

mauriciovasquezbernal force-pushed the mauricio/btfgen-ig branch from 0fe1ba2 to d8f37ff Compare September 20, 2023 15:55

mauriciovasquezbernal merged commit dfd5a39 into main Sep 20, 2023
48 of 49 checks passed

mauriciovasquezbernal deleted the mauricio/btfgen-ig branch September 20, 2023 16:59

mauriciovasquezbernal mentioned this pull request Sep 22, 2023

Get rid / consider alternatives to entrypoint.sh #801

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

btfgen: Embed btf generated files into the binary #2044

btfgen: Embed btf generated files into the binary #2044

mauriciovasquezbernal commented Sep 15, 2023 •

edited

eiffel-fl left a comment

mauriciovasquezbernal commented Sep 18, 2023

eiffel-fl left a comment

eiffel-fl left a comment

mauriciovasquezbernal commented Sep 19, 2023

eiffel-fl commented Sep 20, 2023

alban left a comment

alban left a comment

alban Sep 20, 2023

mauriciovasquezbernal Sep 20, 2023

alban Sep 20, 2023

btfgen: Embed btf generated files into the binary #2044

btfgen: Embed btf generated files into the binary #2044

Conversation

mauriciovasquezbernal commented Sep 15, 2023 • edited

How to test

Getting a host without BTF

Before this PR

After this PR

Option A: Locally compiling ig with btf enabled

Option B: Getting ig from a CI run

eiffel-fl left a comment

Choose a reason for hiding this comment

mauriciovasquezbernal commented Sep 18, 2023

eiffel-fl left a comment

Choose a reason for hiding this comment

eiffel-fl left a comment

Choose a reason for hiding this comment

mauriciovasquezbernal commented Sep 19, 2023

eiffel-fl commented Sep 20, 2023

alban left a comment

Choose a reason for hiding this comment

alban left a comment

Choose a reason for hiding this comment

alban Sep 20, 2023

Choose a reason for hiding this comment

mauriciovasquezbernal Sep 20, 2023

Choose a reason for hiding this comment

alban Sep 20, 2023

Choose a reason for hiding this comment

mauriciovasquezbernal commented Sep 15, 2023 •

edited