libbpf-tools: add CO-RE xfsslower

[ethercflow]

• Parse fsync's offset to start and size to end - start instead of zero;

Signed-off-by: Wenbo Zhang ethercflow@gmail.com

[ethercflow]

Hi @anakryiko , I want to try fentry/fexit to replace kprobe/kretprobe like this PR. But got runtime error:

libbpf: fentry/xfs_file_read_iter is not found in vmlinux BTF
libbpf: failed to load object 'xfsslower_bpf'
libbpf: failed to load BPF skeleton 'xfsslower_bpf': -2
failed to load BPF object: -2

Would you please help me to find the faults in the code? Thank you.

[ethercflow]

I've xfs module:

➜ ~ cat /proc/kallsyms | grep xfs_file_read_iter
ffffffffc0cd7860 t xfs_file_read_iter [xfs]
➜ ~ lsmod | grep xfs
xfs 1699840 1
libcrc32c 16384 5 nf_conntrack,nf_nat,bnx2x,xfs,ip_vs

[anakryiko]

Right, xfs is built as a module, that's what I suspected. BTF is generated and supported only for vmlinux itself right now, not for kernel modules, unfortunately. So to make fentry/fexit work with them you'll need to build-in XFS support.

[ethercflow]

Right, xfs is built as a module, that's what I suspected. BTF is generated and supported only for vmlinux itself right now, not for kernel modules, unfortunately. So to make fentry/fexit work with them you'll need to build-in XFS support.

Thank you @anakryiko , I just tried with build-in XFS, but still get the same errror:

➜ libbpf-tools git:(libbpf-tools) ✗ ./xfsslower
libbpf: fentry/xfs_file_read_iter is not found in vmlinux BTF
libbpf: failed to load object 'xfsslower_bpf'
libbpf: failed to load BPF skeleton 'xfsslower_bpf': -2
failed to load BPF object: -2
➜ libbpf-tools git:(libbpf-tools) ✗ cat /proc/kallsyms| grep xfs_file_read_iter
ffffffff92035aa0 t xfs_file_read_iter
➜ libbpf-tools git:(libbpf-tools) ✗

To use fentry/fexit, does it need any CONFIG_XX? I've checked

CONFIG_HAVE_FENTRY=y
CONFIG_FUNCTION_TRACER=y

[anakryiko]

You have pahole 1.15, isn't that right? fentry/fexit relies on func infos in BTF, which are produced starting from pahole 1.16, can you please try upgrading?

[anakryiko]

implementation looks good, few suggestions only

[ethercflow]

You have pahole 1.15, isn't that right? fentry/fexit relies on func infos in BTF, which are produced starting from pahole 1.16, can you please try upgrading?

Thank you @anakryiko , I updated to 1.16 at host then make clean && rebuild kernel with gcc 8.3.0, and restart qemu to load the recompiled kernel. But still met that error. Did I miss anything else?

1. env

guest os version: 5.6.0-rc3+
host pahole version: v1.16
compile kernel with gcc (Ubuntu 8.3.0-26ubuntu1~18.04) 8.3.0

2. Upload full Kconfig (zcat /proc/config.gz)
   config.txt

3. Upload verbose log in xfsslower (add -v command line arg) and upload it in full.
   xfsslower_log.txt

4. Upload full dump of bin/bpftool btf dump file /sys/kernel/btf/vmlinux (not a C one)
   vmlinux.txt

[anakryiko]

So I have bad news for you w.r.t. fentry/fexit usage here. xfs_file_read_iter is static function, and those functions are not recorded (at least right now) in BTF. Which means you can't attach to them with fentry/fexit programs. Your best bet for now is to stick to kprobe, probably.

[ethercflow]

So I have bad news for you w.r.t. fentry/fexit usage here. xfs_file_read_iter is static function, and those functions are not recorded (at least right now) in BTF. Which means you can't attach to them with fentry/fexit programs. Your best bet for now is to stick to kprobe, probably.

Get it, I'll change this to kprobe and try fentry/fexit with sth else later. Thank you.

[ethercflow]

So I have bad news for you w.r.t. fentry/fexit usage here. xfs_file_read_iter is static function, and those functions are not recorded (at least right now) in BTF. Which means you can't attach to them with fentry/fexit programs. Your best bet for now is to stick to kprobe, probably.

Get it, I'll change this to kprobe and try fentry/fexit with sth else later. Thank you.

Hi @anakryiko , are BPF_KPROBE && PT_REGS_PARM available now?

I tried BPF_KRPOBE from your blog's example,

SEC("kprobe/acct_collect")
int BPF_KPROBE(kprobe__acct_collect, long exit_code, int group_dead)
{

}

with libbpf/libbpf@9a424be, but got some warning and error at compile time:

kprobe.txt

BTW, from

So I have bad news for you w.r.t. fentry/fexit usage here. xfs_file_read_iter is static function, and those functions are not recorded (at least right now) in BTF. Which means you can't attach to them with fentry/fexit programs. Your best bet for now is to stick to kprobe, probably.

so if the functions are from kernel modules or defined as static. I need to use as bcc's way like:

#ifdef __BCC__
int kprobe__acct_collect(struct pt_regs *ctx, long exit_code, int group_dead)

instead of CO-RE right? Thank you.

[anakryiko]

Wenbo, right, once libbpf is updated (#2814), BPF_KPROBE and PT_REGS_PARM macros should be available. But there is a bit of complication due to using -target bpf, which loses track of host architecture. I forgot about that, given kernel selftests solved that with extra __TARGET_ARCH_$(SRCARCH) defined. I've attempted to fix that in #2815, let me know if that works for you. I've only tested on x86_64, though.

[anakryiko]

As for the other part of your question. You are conflating few things. When talking about kprobes, we can talk about BCC and libbpf version only. CO-RE is, in the most strict sense, only about field relocations (so all those BPF_CORE_READ macro). Then fentry/fexit are another type of BPF programs, can work without using CO-RE relocations, but in practice those two are used together most frequently. So for your use case, you need libbpf-style kprobes. I'd recomment using BPF_KPROBE, but you can go with:

SEC("kprobe/whatever")
int my_kprobe(struct pt_regs *ctx)
{
    int arg0 = PT_REGS_PARM1(ctx);
   ...
}

BPF_KPROBE conveniently hides those translations from you, nothing more. We can call this libbpf-style kprobes.

[ethercflow]

As for the other part of your question. You are conflating few things. When talking about kprobes, we can talk about BCC and libbpf version only. CO-RE is, in the most strict sense, only about field relocations (so all those BPF_CORE_READ macro). Then fentry/fexit are another type of BPF programs, can work without using CO-RE relocations, but in practice those two are used together most frequently. So for your use case, you need libbpf-style kprobes. I'd recomment using BPF_KPROBE, but you can go with:

SEC("kprobe/whatever")
int my_kprobe(struct pt_regs *ctx)
{
    int arg0 = PT_REGS_PARM1(ctx);
   ...
}

BPF_KPROBE conveniently hides those translations from you, nothing more. We can call this libbpf-style kprobes.

As for the other part of your question. You are conflating few things. When talking about kprobes, we can talk about BCC and libbpf version only. CO-RE is, in the most strict sense, only about field relocations (so all those BPF_CORE_READ macro). Then fentry/fexit are another type of BPF programs, can work without using CO-RE relocations, but in practice those two are used together most frequently. So for your use case, you need libbpf-style kprobes. I'd recomment using BPF_KPROBE, but you can go with:

SEC("kprobe/whatever")
int my_kprobe(struct pt_regs *ctx)
{
    int arg0 = PT_REGS_PARM1(ctx);
   ...
}

BPF_KPROBE conveniently hides those translations from you, nothing more. We can call this libbpf-style kprobes.

Thank you @anakryiko so much, I did think that the clang 's precompiler might handle BPF_xxx to do special processing to let xxx's bpf prog use with BTF and with BCC macro, it'll fallback to work without BTF, so I got

➜ libbpf-tools git:(libbpf-tools) ✗ ./xfsslower
libbpf: fentry/xfs_file_read_iter is not found in vmlinux BTF
libbpf: failed to load object 'xfsslower_bpf'
libbpf: failed to load BPF skeleton 'xfsslower_bpf': -2
failed to load BPF object: -2
➜ libbpf-tools git:(libbpf-tools) ✗ cat /proc/kallsyms| grep xfs_file_read_iter
ffffffff92035aa0 t xfs_file_read_iter
➜ libbpf-tools git:(libbpf-tools) ✗

It is a bad habit to speculate on the principle, I should throw my doubts out and ask. :(

I think I misunderstood and I have a confusion about:

So I have bad news for you w.r.t. fentry/fexit usage here. xfs_file_read_iter is static function, and those functions are not recorded (at least right now) in BTF. Which means you can't attach to them with fentry/fexit programs. Your best bet for now is to stick to kprobe, probably.

As xfs_file_read_iter is static and not recorded in BTF (I think BTF == CO-RE right?), and fentry/fexit are another type of BPF programs, can work without using CO-RE relocations. So how can I fix

➜ libbpf-tools git:(libbpf-tools) ✗ ./xfsslower
libbpf: fentry/xfs_file_read_iter is not found in vmlinux BTF
libbpf: failed to load object 'xfsslower_bpf'
libbpf: failed to load BPF skeleton 'xfsslower_bpf': -2
failed to load BPF object: -2
➜ libbpf-tools git:(libbpf-tools) ✗ cat /proc/kallsyms| grep xfs_file_read_iter
ffffffff92035aa0 t xfs_file_read_iter
➜ libbpf-tools git:(libbpf-tools) ✗

with fentry/fexit + non-BTF?

Does it mean I need to use another libbpf's APIs in userspace like libbcc did? (I think I only changed from fentry/fexit to kprobe may still meet the similar error as xfs_file_read_iter's BTF doesn't exit.)

[ethercflow]

Wenbo, right, once libbpf is updated (#2814), BPF_KPROBE and PT_REGS_PARM macros should be available. But there is a bit of complication due to using -target bpf, which loses track of host architecture. I forgot about that, given kernel selftests solved that with extra __TARGET_ARCH_$(SRCARCH) defined. I've attempted to fix that in #2815, let me know if that works for you. I've only tested on x86_64, though.

Thank you @anakryiko , I tested on x86_64, it works for me. No warnings and errors at compile time. But got runtime error:

libbpf: prog 'kprobe/xfs_file_read_iter': relo #0: matching error: -22
libbpf: prog 'kprobe/xfs_file_read_iter': relo #0: failed to relocate: -22
libbpf: failed to perform CO-RE relocations: -22
libbpf: failed to load object 'xfsslower_bpf'
libbpf: failed to load BPF skeleton 'xfsslower_bpf': -22
failed to load BPF object: -22

Does it related to #2806 (comment)?

[anakryiko]

@ethercflow, BTF is used for many things. At it's essence it's just a type information available form kernel at runtime. BTF is used for CO-RE relocations, of course. But it's also used for fentry/fexit to find kernel's function type definiton. It's also used for .struct_ops, but that's for another day. My point is that BTF != CO-RE, BTF is more generally used.

Now, you can't use fentry/fexit without BTF. Which means you are stuck with kprobe. If you further need to use CO-RE with that kprobe with types defined in a module -- then again, you can't do that. It all comes down to BTF availability for functions and types you need from kernel.

In your case, seems like you only care about struct kiocb and struct file, both should be in kernel. So it seems like kprobe + CO-RE will work for your use-case, no? So please try that approach.

[ethercflow]

@ethercflow, BTF is used for many things. At it's essence it's just a type information available form kernel at runtime. BTF is used for CO-RE relocations, of course. But it's also used for fentry/fexit to find kernel's function type definiton. It's also used for .struct_ops, but that's for another day. My point is that BTF != CO-RE, BTF is more generally used.

@anakryiko Thank you for the details, I think I understand this now.

Now, you can't use fentry/fexit without BTF. Which means you are stuck with kprobe. If you further need to use CO-RE with that kprobe with types defined in a module -- then again, you can't do that. It all comes down to BTF availability for functions and types you need from kernel.

Please help me check my understanding of this passage：

1. The fentry/fexit needs BTF to work, the related BTF must include function types and parameter types, no matter whether with CO-RE relocations or not;
2. The kprobe + CO-RE neeeds BTF to work, but the related BTF can only include the necessary paramters' types which the kprobe handlers need;

In your case, seems like you only care about struct kiocb and struct file, both should be in kernel. So it seems like kprobe + CO-RE will work for your use-case, no? So please try that approach.

Thank you, I tried with kprobe + CO-RE and got runtime error:

libbpf: prog 'kprobe/xfs_file_read_iter': relo #0: matching error: -22
libbpf: prog 'kprobe/xfs_file_read_iter': relo #0: failed to relocate: -22
libbpf: failed to perform CO-RE relocations: -22
libbpf: failed to load object 'xfsslower_bpf'
libbpf: failed to load BPF skeleton 'xfsslower_bpf': -22
failed to load BPF object: -22

Then I replace xfsslower.bpf.c's kprobes with your example from your blog:

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_tracing.h>

SEC("kprobe/acct_collect")
int BPF_KPROBE(kprobe__acct_collect, long exit_code, int group_dead)
{
        bpf_printk("%ld,%d\n", exit_code, group_dead);
        return 0;
}

char LICENSE[] SEC("license") = "GPL";

got similar error (with -v):

libbpf: loading object 'xfsslower_bpf' from buffer
libbpf: section(1) .text, size 0, link 0, flags 6, type=1
libbpf: skip section(1) .text
libbpf: section(2) kprobe/acct_collect, size 104, link 0, flags 6, type=1
libbpf: found program kprobe/acct_collect
libbpf: section(3) license, size 4, link 0, flags 3, type=1
libbpf: license of xfsslower_bpf is GPL
libbpf: section(4) .rodata.str1.1, size 9, link 0, flags 32, type=1
libbpf: skip section(4) .rodata.str1.1
libbpf: section(5) .BTF, size 837, link 0, flags 0, type=1
libbpf: section(6) .BTF.ext, size 188, link 0, flags 0, type=1
libbpf: section(7) .symtab, size 120, link 11, flags 0, type=2
libbpf: section(8) .rel.BTF, size 16, link 7, flags 0, type=9
libbpf: skip relo .rel.BTF(8) for section(5)
libbpf: section(9) .rel.BTF.ext, size 128, link 7, flags 0, type=9
libbpf: skip relo .rel.BTF.ext(9) for section(6)
libbpf: section(10) .llvm_addrsig, size 2, link 0, flags 80000000, type=1879002115
libbpf: skip section(10) .llvm_addrsig
libbpf: section(11) .strtab, size 147, link 0, flags 0, type=3
libbpf: skip section(11) .strtab
libbpf: looking for externs among 5 symbols...
libbpf: collected 0 externs total
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: prog 'kprobe/acct_collect': performing 2 CO-RE offset relocs
libbpf: prog 'kprobe/acct_collect': relo #0: kind 0, spec is [2] pt_regs + 0:13 => 104.0 @ &x[0].si
libbpf: [2] pt_regs: found candidate [198] pt_regs
libbpf: [2] pt_regs: found candidate [9195] pt_regs
libbpf: prog 'kprobe/acct_collect': relo #0: matching candidate #0 pt_regs against spec [198] pt_regs + 0:13 => 104.0 @ &x[0].si: 1
libbpf: prog 'kprobe/acct_collect': relo #0: matching candidate #1 pt_regs against spec [9195] pt_regs + 0 => 0.0 @ &x[0]: -22
libbpf: prog 'kprobe/acct_collect': relo #0: matching error: -22
libbpf: prog 'kprobe/acct_collect': relo #0: failed to relocate: -22
libbpf: failed to perform CO-RE relocations: -22
libbpf: failed to load object 'xfsslower_bpf'
libbpf: failed to load BPF skeleton 'xfsslower_bpf': -22
failed to load BPF object: -22

What's the meaning of this error and how to fix this?

[anakryiko]

@anakryiko Thank you for the details, I think I understand this now.

You're welcome! Glad it was helpful.

Please help me check my understanding of this passage：

1. The `fentry/fexit` needs `BTF` to work, the related `BTF` must include function types and parameter types, no matter whether with CO-RE relocations or not;

Correct.

2. The `kprobe + CO-RE` neeeds `BTF` to work, but the related `BTF` can only include the necessary paramters' types which the kprobe handlers need;

Correct again.

libbpf: prog 'kprobe/acct_collect': performing 2 CO-RE offset relocs
libbpf: prog 'kprobe/acct_collect': relo #0: kind 0, spec is [2] pt_regs + 0:13 => 104.0 @ &x[0].si
libbpf: [2] pt_regs: found candidate [198] pt_regs
libbpf: [2] pt_regs: found candidate [9195] pt_regs
libbpf: prog 'kprobe/acct_collect': relo #0: matching candidate #0 pt_regs against spec [198] pt_regs + 0:13 => 104.0 @ &x[0].si: 1
libbpf: prog 'kprobe/acct_collect': relo #0: matching candidate #1 pt_regs against spec [9195] pt_regs + 0 => 0.0 @ &x[0]: -22
libbpf: prog 'kprobe/acct_collect': relo #0: matching error: -22
libbpf: prog 'kprobe/acct_collect': relo #0: failed to relocate: -22
libbpf: failed to perform CO-RE relocations: -22
libbpf: failed to load object 'xfsslower_bpf'
libbpf: failed to load BPF skeleton 'xfsslower_bpf': -22
failed to load BPF object: -22

What's the meaning of this error and how to fix this?

Ok, I think this is actually a bug in libbpf. It's trying to do relocation against struct pt_regs forward declaration and fails. I'm gonna work on a fix, but meanwhile to unblock yourself you can add the following #define before including vmlinux.h:

#define BPF_NO_PRESERVE_ACCESS_INDEX
#include "vmlinux.h"

This will disable preserve_access_index attribute to be applied to all types, which you don't rely on either way (it's mostly useful for fentry/fexit/tp_btf BPF programs).

[ethercflow]

Ok, I think this is actually a bug in libbpf. It's trying to do relocation against struct pt_regs forward declaration and fails. I'm gonna work on a fix, but meanwhile to unblock yourself you can add the following #define before including vmlinux.h:

#define BPF_NO_PRESERVE_ACCESS_INDEX
#include "vmlinux.h"
This will disable preserve_access_index attribute to be applied to all types, which you don't rely on either way (it's mostly useful for fentry/fexit/tp_btf BPF programs).

I'll keep going after work, thanks. :)

[anakryiko]

alright, should be fixed as part of this series: https://patchwork.ozlabs.org/project/netdev/list/?series=164156&state=*

[ethercflow]

Hi @anakryiko , the current PR with

#define BPF_NO_PRESERVE_ACCESS_INDEX
#include "vmlinux.h"

works as expected, PTAL.

➜  libbpf-tools git:(master) ✗ ./xfsslower -m 0
Tracing XFS operations... Hit Ctrl-C to end.
TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
01:44:27 qemu-system-x8 24565  W 4096    553672      0.05 stretch.img
01:44:27 qemu-system-x8 24565  W 4096    167248      0.02 stretch.img
01:44:27 qemu-system-x8 24565  W 4096    12255028    0.02 stretch.img
01:44:33 qemu-system-x8 24565  F LL_MAX  0           0.47 stretch.img
01:44:33 qemu-system-x8 24565  W 4096    19009300    0.05 stretch.img
01:44:33 qemu-system-x8 24565  F LL_MAX  0           0.09 stretch.img
01:44:50 qemu-system-x8 24565  W 16384   19009304    0.08 stretch.img
01:44:50 qemu-system-x8 24565  F LL_MAX  0           0.17 stretch.img
01:44:50 qemu-system-x8 24565  W 4096    19009320    0.02 stretch.img
01:44:50 qemu-system-x8 24565  F LL_MAX  0           0.10 stretch.img
01:45:00 qemu-system-x8 24565  W 16384   19009324    0.07 stretch.img
01:45:00 qemu-system-x8 24565  F LL_MAX  0           0.16 stretch.img
01:45:00 qemu-system-x8 24565  W 4096    19009340    0.02 stretch.img
01:45:00 qemu-system-x8 24565  F LL_MAX  0           0.09 stretch.img
01:45:15 qemu-system-x8 24565  W 4096    553672      0.05 stretch.img
01:45:15 qemu-system-x8 24565  W 4096    167248      0.02 stretch.img
01:45:15 qemu-system-x8 24565  W 4096    12255028    0.02 stretch.img
01:45:20 qemu-system-x8 24565  W 4096    2097292     0.05 stretch.img
01:45:20 qemu-system-x8 24565  W 4096    2097444     0.02 stretch.img
01:45:20 qemu-system-x8 24565  W 4096    2097452     0.02 stretch.img
01:45:21 qemu-system-x8 24565  F LL_MAX  0           0.37 stretch.img
01:45:21 qemu-system-x8 24565  W 4096    19009344    0.04 stretch.img
01:45:21 qemu-system-x8 24565  F LL_MAX  0           0.09 stretch.img
01:46:00 qemu-system-x8 24565  W 16384   19009348    0.10 stretch.img
01:46:00 qemu-system-x8 24565  F LL_MAX  0           0.17 stretch.img
01:46:00 qemu-system-x8 24565  W 4096    19009364    0.02 stretch.img
01:46:00 qemu-system-x8 24565  F LL_MAX  0           0.09 stretch.img

With a little regret as https://patchwork.ozlabs.org/patch/1211861/'s state is still Awaiting Upstream, we can't get a full path FILENAME. :(

BTW, I found another two issues:

1. I've pulled and merged libbpf-tools: adjust Kconfig and re-build vmlinux.h #2820, but found I still need to

#define NULL    0

Actually, verifier might not allow direct memory access in this particular case, because you are storing struct file * in map, at which point verifier has no way to guarantee that this pointer won't be overwritten from userspace. Please try it, but you might need to stick to BPF_CORE_READ for this one, unfortunately.

I found

qs = BPF_CORE_READ(pidstatp, fp, f_path.dentry, d_name);

will be rejected by verifier.

libbpf: load bpf program failed: Invalid argument
libbpf: -- BEGIN DUMP LOG ---
libbpf:
Unrecognized arg#0 type PTR
... ...
... ...
48: (85) call unknown#195896080
invalid func unknown#195896080
processed 43 insns (limit 1000000) max_states_per_insn 0 total_states 2 peak_states 2 mark_read 2

The full log: xfsslower_log.txt Is this behavior as expected?

[anakryiko]

yes, NULL is not enum, so you'll have to re-define it (though I guess we can add that to vmlinux as well, I suppose, I'll think about that).

Looks great, few more comments to make this perfect, but I think it's almost ready to land.

[anakryiko]

@ethercflow, BPF_KRETPROBE fix is being pulled into BCC in #2823

[ethercflow]

@ethercflow, BPF_KRETPROBE fix is being pulled into BCC in #2823

Great! I'll update the PR after work. Thanks!

[anakryiko]

Looks great to me, thanks for doing this and helping hammer out remaining bugs!

[ethercflow]

@yonghong-song @brendangregg Would you please help me to see if it‘s good enough to be merged? After merge I want to submit another PR with fentry/fexit example. Thanks!

[yonghong-song]

[buildbot, test this please]

[yonghong-song]

LGTM. Thanks!