Unable to run kprobe example

[Darrilla]

I commented on #68 about how I could not get this to run on ubuntu 18.04. I was able to get minimal working, by removing the global variable. I am using kernel version 4.15.0-180-generic. The kprobe example, however, is not running for some other reason:

sudo ./kprobe
libbpf: loading object 'kprobe_bpf' from buffer
libbpf: elf: section(2) kprobe/do_unlinkat, size 152, link 0, flags 6, type=1
libbpf: sec 'kprobe/do_unlinkat': found program 'do_unlinkat' at insn offset 0 (0 bytes), code size 19 insns (152 bytes)
libbpf: elf: section(3) kretprobe/do_unlinkat, size 88, link 0, flags 6, type=1
libbpf: sec 'kretprobe/do_unlinkat': found program 'do_unlinkat_exit' at insn offset 0 (0 bytes), code size 11 insns (88 bytes)
libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1
libbpf: license of kprobe_bpf is Dual BSD/GPL
libbpf: elf: section(5) .rodata, size 72, link 0, flags 2, type=1
libbpf: elf: section(6) .BTF, size 1482, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 364, link 0, flags 0, type=1
libbpf: elf: section(8) .symtab, size 240, link 14, flags 0, type=2
libbpf: elf: section(9) .relkprobe/do_unlinkat, size 16, link 8, flags 0, type=9
libbpf: elf: section(10) .relkretprobe/do_unlinkat, size 16, link 8, flags 0, type=9
libbpf: looking for externs among 10 symbols...
libbpf: collected 0 externs total
libbpf: map 'kprobe_b.rodata' (global data): at sec_idx 5, offset 0, flags 480.
libbpf: map 0 is "kprobe_b.rodata"
libbpf: sec '.relkprobe/do_unlinkat': collecting relocation for section(2) 'kprobe/do_unlinkat'
libbpf: sec '.relkprobe/do_unlinkat': relo #0: insn #12 against '.rodata'
libbpf: prog 'do_unlinkat': found data map 0 (kprobe_b.rodata, sec 5, off 0) for insn 12
libbpf: sec '.relkretprobe/do_unlinkat': collecting relocation for section(3) 'kretprobe/do_unlinkat'
libbpf: sec '.relkretprobe/do_unlinkat': relo #0: insn #3 against '.rodata'
libbpf: prog 'do_unlinkat_exit': found data map 0 (kprobe_b.rodata, sec 5, off 0) for insn 3
libbpf: Kernel doesn't support BTF, skipping uploading it.
libbpf: map 'kprobe_b.rodata': skipped auto-creating...
libbpf: sec 'kprobe/do_unlinkat': found 2 CO-RE relocations
libbpf: CO-RE relocating [2] struct pt_regs: found target candidate [14] struct pt_regs in [vmlinux]
libbpf: prog 'do_unlinkat': relo #0: <byte_off> [2] struct pt_regs.si (0:13 @ offset 104)
libbpf: prog 'do_unlinkat': relo #0: non-matching candidate #0 <byte_off> [14] struct pt_regs (0 @ offset 0)
libbpf: prog 'do_unlinkat': relo #0: no matching targets found
libbpf: prog 'do_unlinkat': relo #0: substituting insn #0 w/ invalid insn
libbpf: prog 'do_unlinkat': relo #1: <byte_off> [7] struct filename.name (0:0 @ offset 0)
libbpf: prog 'do_unlinkat': relo #1: no matching targets found
libbpf: prog 'do_unlinkat': relo #1: substituting insn #3 w/ invalid insn
libbpf: sec 'kretprobe/do_unlinkat': found 1 CO-RE relocations
libbpf: prog 'do_unlinkat_exit': relo #0: <byte_off> [2] struct pt_regs.ax (0:10 @ offset 80)
libbpf: prog 'do_unlinkat_exit': relo #0: matching candidate #0 <byte_off> [14] struct pt_regs.ax (0:0 @ offset 80)
libbpf: prog 'do_unlinkat_exit': relo #0: patched insn #0 (LDX/ST/STX) off 80 -> 80
libbpf: prog 'do_unlinkat': relo #2: poisoning insn #12 that loads map #0 'kprobe_b.rodata'
libbpf: prog 'do_unlinkat_exit': relo #1: poisoning insn #3 that loads map #0 'kprobe_b.rodata'
libbpf: prog 'do_unlinkat': BPF program load failed: Invalid argument
libbpf: prog 'do_unlinkat': -- BEGIN PROG LOAD LOG --
0: <invalid CO-RE relocation>
failed to resolve CO-RE relocation <byte_off> [2] struct pt_regs.si (0:13 @ offset 104)
-- END PROG LOAD LOG --
libbpf: failed to load program 'do_unlinkat'
libbpf: failed to load object 'kprobe_bpf'
libbpf: failed to load BPF skeleton 'kprobe_bpf': -22
Failed to load and verify BPF skeleton

[anakryiko]

yes, you need kernel BTF for CO-RE to work. You can work around that for such trivial example by adding

#define BPF_NO_PRESERVE_ACCESS_INDEX

before vmlinux.h include, but is there any reason why you can't upgrade your Ubuntu to something a bit more modern that will restrict you less in what you can do with BPF?

[Darrilla]

Yes, I have a huge set of BCC-style kprobes that we are using with Centos 7/RHEL servers, amazon Linux 1 & 2, ubuntu 18.04 +, etc. The goal is to migrate these to libbpf and BPF CO-RE using btfhub files for the older kernels. I assumed this would be possible after seeing the BTF files for the kernels we currently support via BCC and kernel headers.

[Darrilla]

BTW, I did modify this kprobe example to load the BTF for my kernel using the opts:

struct bpf_object_open_opts openopts = {};
openopts.sz = sizeof(struct bpf_object_open_opts);
openopts.btf_custom_path = "/tmp/vmlinux.btf";
...
...
/* Load and verify BPF application */
skel = kprobe_bpf__open_opts(&openopts);

/tmp/vmlinux.btf is the file I obtained for my kernel from btfhub-archives.

[Darrilla]

#define BPF_NO_PRESERVE_ACCESS_INDEX

Adding this define did not resolve the issue.

[Darrilla]

Even after adding

#define BPF_NO_GLOBAL_DATA
#define BPF_NO_PRESERVE_ACCESS_INDEX

there is still a problem due to, I believe, the usage of BPF_CORE_READ:

filename = BPF_CORE_READ(name, name);

When I code around this using bpf_probe_read_user, I am able to get things to work. This brings me back to a core question (no pun intented), are the bpf core helpers not going to work on older kernels even if you have the BTF for that kernel available?

[anakryiko]

it should work with btf_custom_path. What's the libbpf debug log output once you specify custom BTF?

[Darrilla]

I got this working. My BTF file didn't have all of the symbols I needed for this. I have run this example on 4.15.0-180-generic and on 3.10.0-957 (centos 7.6).