New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use kernel libbpf in bcc #2149

Merged
merged 5 commits into from Jan 21, 2019

Conversation

Projects
None yet
4 participants
@yonghong-song
Copy link
Collaborator

yonghong-song commented Jan 17, 2019

The kernel tools/lib/bpf implements some core bpf related functionality, prog/map loading, probing, btf etc.. The kernel/lib/bpf code has been mirrored at https://github.com/libbpf/libbpf.

This pull request brought libbpf mirror into bcc, as a git submodule, in src/cc/libbpf directory.
The bcc will use bpf.h/btf.h uapi header files from libbpf. Due to naming conflicts between bcc and libbpf, the following two bcc functions are renamed:

   bpf_create_map => bcc_create_map
   bpf_prog_load => bcc_prog_load

This pull request reimplemented bcc_create_map using libbpf bpf_create_map and bpf_create_map_name functions. In the future, more bcc functions can be converted to use
libbpf functions.

To facilitate review, there are total 5 commits for this pull request:
- add kernel libbpf as a submodule
- get submodules in cmake and package build
- use linux bpf uapi header from libbpf submodule
- include libbpf/src/*.c files in the build system
- rewrite bcc_create_map with libbpf primitives

@yonghong-song

This comment has been minimized.

Copy link
Collaborator Author

yonghong-song commented Jan 17, 2019

All tests failed. In my local setup, I see git submodule update successfully checked out libbpf repo.

...
Submodule 'src/cc/libbpf' (https://github.com/libbpf/libbpf.git) registered for path 'src/cc/libbpf'
Cloning into '/data/users/yhs/work/bcc-test/src/cc/libbpf'...
Submodule path 'src/cc/libbpf': checked out 'd5b146fec50d7aa126fe98323aeaee688d4af289'
...

The jenkins testbot failed the checkout with error message,

fatal: Not a git repository (or any of the parent directories): .git

Not sure what is the issue here.

add kernel libbpf as a submodule
The kernel libbpf library
  https://github.com/torvalds/linux/tree/master/tools/lib/bpf
provides common functionality for handling bpf programs, maps,
ELF parsing, probing, BTF manipulation, etc. The library has
been recommended to the community as a common library for
bpf/btf handling. A separate repository
  https://github.com/libbpf/libbpf
has been created for this purpose.

This patch brings in libbpf to bcc. The following command
is executed to crate a libbpf submodule at src/cc/libbpf:
  git submodule add https://github.com/libbpf/libbpf.git src/cc/libbpf

Signed-off-by: Yonghong Song <yhs@fb.com>

@yonghong-song yonghong-song force-pushed the yhs_dev branch 2 times, most recently from def5664 to 847460c Jan 17, 2019

yonghong-song added some commits Jan 16, 2019

get submodules in cmake and package build
If the target directory src/cc/libbpf/ does not exist,
the top level CMakeLists.txt is changed to fetch
submodules with the following command
  git submodule update --init --recursive

The dev/rpm build scripts are also changed
to do git submodule update.

Signed-off-by: Yonghong Song <yhs@fb.com>
use linux bpf uapi header from libbpf submodule
The uapi header
  src/cc/libbpf/include/uapi/linux/bpf.h
is used instead of the current way
  src/cc/compat/linux/bpf.h

Signed-off-by: Yonghong Song <yhs@fb.com>
@yonghong-song

This comment has been minimized.

Copy link
Collaborator Author

yonghong-song commented Jan 17, 2019

now all tests passed.

@drzaeus77 Please take a look just in case I made any mistakes
@brendangregg I assume this should not impact bpftrace, right?
Does people has concern/suggestions about the bcc API changes:

  • bpf_create_map => bpf_create_map2
  • bpf_prog_load => bpf_prog_load2

due to name conflicts with libbpf repo?

yonghong-song added some commits Jan 17, 2019

include libbpf/src/*.c files in the build system
The libbpf/src/*.c files are included in the build system,
so those functions will be available for bcc internals to use them.

There are two name conflicts, bpf_create_map and bpf_prog_load,
between src/cc/libbpf.c and src/cc/libbpf/src/{bpf.c,libbpf.c}.
To keep src/cc/libbpf intact, the following renaming happened
in bcc repo:
  bpf_create_map  =>  bcc_create_map
  bpf_prog_load   =>  bcc_prog_load

Signed-off-by: Yonghong Song <yhs@fb.com>
rewrite bcc_create_map with libbpf primitives
This patch demonstrated how to use libbpf function calls.
Specially, the bcc_map_create is changed to use libbpf
functions bpf_create_map_name and bpf_create_map.

Signed-off-by: Yonghong Song <yhs@fb.com>
@yonghong-song

This comment has been minimized.

Copy link
Collaborator Author

yonghong-song commented Jan 18, 2019

Changed the conflict function renaming to:

  • bpf_create_map => bcc_create_map
  • bpf_prog_load => bcc_prog_load

since this is more clear comparing bpf_create_map => bpf_create_map2.

@4ast

4ast approved these changes Jan 20, 2019

Copy link
Member

4ast left a comment

big milestone.
all patches look great.

@yonghong-song yonghong-song merged commit dc1254e into master Jan 21, 2019

1 check passed

default
Details
@brendangregg

This comment has been minimized.

Copy link
Member

brendangregg commented Jan 22, 2019

Yes, bpftrace needs updating. I haven't dug in yet, but these just came in iovisor/bpftrace#374 iovisor/bpftrace#377

@danobi

This comment has been minimized.

Copy link
Member

danobi commented Jan 23, 2019

@yonghong-song is there an ETA on the next tagged release with these changes?

@yonghong-song

This comment has been minimized.

Copy link
Collaborator Author

yonghong-song commented Jan 24, 2019

The bcc release roughly aligned with kernel release. So next release will be around the end of February.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment