program: relocation of program targeting a module fails if CONFIG_DEBUG_INFO_BTF_MODULES is disabled

[orishuss]

Describe the bug

Support for btf parsing of kernel modules, recently introduced, was long awaited, but I believe it breaks some bpf programs. The programs affected are ones that are attached to loaded kernel module functions, on certain systems according to kernel configuration.

CONFIG_DEBUG_INFO_BTF_MODULES is a kernel flag, introduced in kernel 5.11, that makes the btf files for kernel modules appear in /sys/kernel/btf. The machine on which I encountered the error is kernel 5.10 which explains why this bug happens, but the same should happen on newer kernels if the flag is not set.

A practical solution could be to make the dependence on the module's specific btf file optional (so that, for example, if /sys/kernel/btf/nf_conntrack_netlink doesn't exist, it just skips it). This way, the module could still use CORE relocations for normal vmlinux structs.

Notice that changing the section name (for example SEC("kprobe/ctnetlink_fill_info") to SEC("kprobe/ctnetlink_fill_info_")) "fixes" the problem because it makes the library only check vmlinux btf, as it can't find what module loaded the function.

Commands outputs

kernel version 5.10.213

package version github.com/cilium/ebpf v0.14.0

missing btf file for module:

~ ls -larth /sys/kernel/btf/
total 0      
drwxr-xr-x   14 root     root           0 Apr 15 10:42 ..
-r--r--r--    1 root     root        3.8M Apr 16 09:48 vmlinux
drwxr-xr-x    2 root     root           0 Apr 16 09:48 .

kernel module is indeed loaded:

~ lsmod | grep nf_conntrack_netlink
nf_conntrack_netlink    61440  0 
nfnetlink              20480  2 nf_tables,nf_conntrack_netlink
nf_conntrack          180224  8 xt_REDIRECT,xt_state,xt_connmark,nf_conntrack_netlink,xt_nat,xt_MASQUERADE,nf_nat,xt_conntrack

kallsyms (where the module name of the function comes from):

~ cat /proc/kallsyms | grep ctnetlink_fill_info
ffffffffxxxxxxxx t ctnetlink_fill_info  [nf_conntrack_netlink]

Output difference between version 0.12.3 and 0.14.0 (the version of the package is the only thing that was changed)

~ ./kmerror-0.12.3
it worked!
~ ./kmerror-0.14.0
2024/04/16 09:46:00 loading objects: field CtnetlinkFillInfo: program ctnetlink_fill_info: apply CO-RE relocations: load kernel module spec: open /sys/kernel/btf/nf_conntrack_netlink: no such file or directory

How to reproduce

use make and run ./kmerror on a machine that is misconfigured

kmerror.zip

[lmb]

Thanks for the detailed report. @brycekahle maybe this is as easy as not generating an error if module BTF doesn't exist?

[brycekahle]

Yeah, that would probably work. If the program needs the module BTF then the CO-RE relocations will fail and poison the instructions. That error is a little harder to debug, but will still fail regardless.

[lmb]

The fix is going to be here somewhere:

ebpf/linker.go

Lines 149 to 155 in 7719d2f

	if kmodName != "" {
	kmodTarget, err := btf.LoadKernelModuleSpec(kmodName)
	if err != nil {
	return fmt.Errorf("load kernel module spec: %w", err)
	}
	targets = append(targets, kmodTarget)
	}

• LoadKernelModuleSpec should return a discernible error when the load fails due to the file not existing. Something like errors.Is(err, os.ErrNotFound).
• The code above should ignore the error if it is due to not being found.

@orishuss are you up for contributing a fix? We can probably test this by using a bogus kmodName and asserting that we don't get an error.