centos SIGSEV on custom kernel.

[BenSimsCitrix]

Seems like I have a rookie configuration issue, when trying to get ebpf running on a custom kernel. It may be related to bpftrace/bpftrace#515 but i am seeing this consistently on any tracing I run.

(gdb) run hello_world.py
Starting program: /usr/bin/python hello_world.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: JITed object file architecture unknown is not compatible with target architecture i386:x86-64.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6e8b901 in __strlen_sse2_pminub () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install elfutils-libelf-0.172-2.el7.x86_64 glibc-2.17-260.el7_6.4.x86_64 libffi-3.0.13-18.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libstdc++-4.8.5-28.el7_5.1.x86_64 llvm-private-6.0.1-2.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0  0x00007ffff6e8b901 in __strlen_sse2_pminub () from /lib64/libc.so.6
#1  0x00007fffef8fc4cb in bpf_object__new (obj_buf_sz=0, obj_buf=0x0, path=0x2 <Address 0x2 out of bounds>) at /usr/src/debug/bcc-0.6.1/src/cc/libbpf/src/libbpf.c:483
#2  __bpf_object__open (path=0x2 <Address 0x2 out of bounds>, obj_buf=obj_buf@entry=0x0, obj_buf_sz=obj_buf_sz@entry=0, needs_kver=<optimized out>, flags=flags@entry=0)
    at /usr/src/debug/bcc-0.6.1/src/cc/libbpf/src/libbpf.c:2096
#3  0x00007fffef8fdf0e in __bpf_object__open (flags=0, needs_kver=<optimized out>, obj_buf_sz=0, obj_buf=0x0, path=<optimized out>)
    at /usr/src/debug/bcc-0.6.1/src/cc/libbpf/src/libbpf.c:2125
#4  __bpf_object__open_xattr (attr=attr@entry=0x7fffffffd010, flags=flags@entry=0) at /usr/src/debug/bcc-0.6.1/src/cc/libbpf/src/libbpf.c:2122
#5  0x00007fffef8fdf57 in bpf_object__open_xattr (attr=attr@entry=0x7fffffffd010) at /usr/src/debug/bcc-0.6.1/src/cc/libbpf/src/libbpf.c:2129
#6  0x00007fffef8ffaac in bpf_prog_load_xattr (attr=attr@entry=0x7fffffffd070, pobj=0x10a8300, prog_fd=0x78) at /usr/src/debug/bcc-0.6.1/src/cc/libbpf/src/libbpf.c:3294
#7  0x00007fffef8ffc72 in bpf_prog_load (file=<optimized out>, type=<optimized out>, pobj=<optimized out>, prog_fd=<optimized out>)

The rpm I have built contains

rpm -ql bcc
/usr/lib64/libbcc.so.0
/usr/lib64/libbcc.so.0.6.1
/usr/lib64/libbcc_bpf.so.0
/usr/lib64/libbcc_bpf.so.0.6.1
/usr/share/doc/bcc-0.6.1
/usr/share/doc/bcc-0.6.1/README.md
/usr/share/licenses/bcc-0.6.1
/usr/share/licenses/bcc-0.6.1/LICENSE.txt

I'm aware that the rpm is currently not picking libbpf, but I am installing that manually (The source code is consistent)

[root@dt74 ~]# ls -l /usr/lib64/libbpf*
-rwxr-xr-x 1 root root 401048 Apr 25 09:23 /usr/lib64/libbpf.so
-rwxr-xr-x 1 root root 401048 Apr 25 09:23 /usr/lib64/libbpf.so.0
-rwxr-xr-x 1 root root 401048 Apr 25 09:23 /usr/lib64/libbpf.so.0.0.2

Any pointers on debugging the configuration on this system?

[BenSimsCitrix]

(gdb) b bpf_prog_load
Function "bpf_prog_load" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (bpf_prog_load) pending.
(gdb) run hello_world.py
Starting program: /usr/bin/python hello_world.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: JITed object file architecture unknown is not compatible with target architecture i386:x86-64.

Breakpoint 1, bpf_prog_load (file=0x2 <Address 0x2 out of bounds>, type=3897170964, pobj=0x10a8300, prog_fd=0x78) at /usr/src/debug/bcc-0.6.1/src/cc/libbpf/src/libbpf.c:3264

So looks like i have a duff pointer coming into libbc.so.

[yonghong-song]

The environment does not seem right. libbcc 0.6.1 has not used the libbpf repo yet. You can verify it by checking out the corresponding sources with tag v0.6.1. Therefore, for 0.6.1, we should not use file /usr/src/debug/bcc-0.6.1/src/cc/libbpf/src/libbpf.c at all.

[BenSimsCitrix]

Hi Yonghong,

Thanks for pointing out my error.

I think this stems from my attempt to populate the libbpf directory, which i'm doing incorrectly.

There is a submodule which populates libbpf from the mainline kernel, Is there another script which keeps all the instances of libbpf.c (for example) synced.

[BenSimsCitrix]

What i'm looking for is the script/s which are keeping the code in sync the mainline kernel, so that i can subvert them to point at my kernel source code.

[yonghong-song]

@BenSimsCitrix There is no script to automate syncing with kernel source. bcc syncs with libbpf repo. The steps are at https://github.com/iovisor/bcc/blob/master/src/cc/README. Directly pointing to the kernel code is not the best way as kernel code may contain some other .c files (e.g., testing files) which is not libbpf yet.

[BenSimsCitrix]

Ok thank you yonghong,

I will close this bug, as i think i have a good hold on the situation now.