Add support for remote target devices via BPFd #1675
Conversation
jcanseco
requested review from
4ast,
brendangregg,
drzaeus77,
goldshtn and
yonghong-song
as code owners
|
[buildbot, ok to test] |
jcanseco
force-pushed
the
upstream_submit
branch
from
242e749
to
ea26f69
Compare
|
Update: applied style changes suggested by the style-check.sh script |
jcanseco
force-pushed
the
upstream_submit
branch
5 times, most recently
from
43fcf37
to
1d2b3f8
Compare
|
@drzaeus77 Could you add python module |
jcanseco
force-pushed
the
upstream_submit
branch
from
ab1aa72
to
1d2b3f8
Compare
|
Looking into it. |
|
[buildbot, test this please] This is just a fc27 test, if that passes I will apply the update to the other vm images. |
|
The fc27 test failed, though pexpect was installed: |
|
Hi @drzaeus77, thanks for taking the time. Is I'll take a look the moment I get into the office on Monday, but I think I know what the issue may be. |
|
Yes, it is still installed. If you push more changes to this PR it will run against the updated fc27 with pexpect. |
jcanseco
force-pushed
the
upstream_submit
branch
3 times, most recently
from
8917d1b
to
e9e4e65
Compare
Libclang uses functions from libbcc, so it does depend on it. When defining new symbols in bpf_common.cc, linker errors appear when the same symbols are used in libclang. This is because of incorrect linker dependency. This patch fixes the issue by making sure the dependency is correctly tracked. Signed-off-by: Joel Fernandes <joelaf@google.com>
jcanseco
force-pushed
the
upstream_submit
branch
from
e9e4e65
to
260fa23
Compare
|
Hi @drzaeus77, I think I figured out why For instance, I believe that the VMs build the Is my understanding here correct? If so, what am I supposed to do to have the VMs build the |
|
We |
jcanseco
force-pushed
the
upstream_submit
branch
from
8a03059
to
ff62cba
Compare
|
Thank you @drzaeus77, you were completely right. I updated the rpm and deb build steps. The tests for fc27 are passing now. Can you please now install the |
Signed-off-by: Jazel Canseco <jcanseco@google.com>
jcanseco
force-pushed
the
upstream_submit
branch
from
bc003fa
to
e1412fb
Compare
|
Can you try now? I applied a modified variant of your patches and fixed other Python 3 support issues that I identified. |
|
[buildbot, ok to test] |
|
much better now. The test The The We still have some failures like below, which I suspect they all related with this byte vs. str thing. How do you debug bpfd? I could add codes to have debug output to a special file. But I am wondering you have a better way to do this. Or the bpfd logs can go to a special location for easy debugging. |
|
Do not add any uses of |
|
Thanks for reminding. These will be just workaround to pinpoint where is the issue. I will make sure final code will not use decode(). |
|
[buildbot, test this please] |
|
I did some experiments on the following additional hack. With the following change, python3 test_tools_on_remote can pass. Most decode() here might be justified since we need to base64 conversion and send the commands to bpfd. I prefer to do One annoying thing is related to |
|
@yonghong-song: thanks for investigating. I've been trying to fix this problem today, and I came up with a patch somewhat similar to the one you just posted. I got the tests passing, but there were other problems that required investigating. We also need to do a thorough test of BCC and BPFd on our end on both Python-2 and Python-3, since this is a non-trivial code change. So please sit tight, this'll probably take some time. Also, to answer your question about debugging from before: I usually run the individual tools that failed if the test fails. You can set the following environment variables to hookup BCC to BPFd. This will make BCC think that there is a remote target device (in the form of a shell process on the same machine): And the following variable to turn on debugging logs: This should show you what BCC is sending to BPFd and what BPFd is sending back. |
|
Thanks, @jcanseco. Additional testing is definitely helpful. Thanks for the tip for run on the shell with env variable. I indeed used it this morning by looking at the test. One thing I noticed is that the error log inside the bpfd is not visible now. But I see a comment in the bpfd.c which says for later work. Indeed, this can be improved after initial patch. |
|
@jcanseco How would I try bpfd in action, can you point me to some readme/doc ? |
|
Hello everyone, I talked to Joel recently, and we both agreed that we should close this PR for now. We realize that the patch is probably not yet fully ready. There are a few issues that we need to fix on our end first which then need to be thoroughly tested. We don't wish to take up too much of everyone's time during this process, so we've decided to close this for now and reopen it later. Thank you @yonghong-song and @drzaeus77 for the time you've spent and all the help you've provided! We really appreciate it, and we hope that you look forward to a more refined version of this patch. @nirmoy If you still wish to try out BPFd in action, you can read the README and INSTALL docs in the original BPFd repository. If you need help, you can contact Joel (joelaf@google.com). Although I recommend waiting before we resubmit this PR so that you'll be able to try out a more refined version of BPFd. |
|
@jcanseco Sure. Just post when you are ready. Thanks! |
|
Thanks a lot @yonghong-song @drzaeus77 and @jcanseco . Currently I am working on improving BCC tools on Android and I found another way to run tools without needing BPFd called androdeb : http://tinyurl.com/androdeb So, I am moving my own focus slightly away from BPFd since I use The other thing to consider is, this PR only makes BCC's python front-end work with BPFd, where as with About the BPFd project: Several nice things and side-effects like BCC fixes came out of the @nirmoy If you want to continue with BPFd work and have usecases for it, please find the latest work in the following branches: BCC tree (BCC master with changes to make it work with BPFd): Stand-alone BPFd which you can cross-compile and push as a proxy to targets: Thanks a lot. |
|
Is this PR opened yet ? We are really looking run bcc on host target environment. Any leads would be a great help. |
|
@jcanseco were you trying to use bpfd on host and target environment (2 VMs, non-android platforms). You tried it on same machine (remote process). |
|
When it was written you could use it either way. The way of running it
locally on the host was good for testing.
…On Fri, Oct 12, 2018, 12:18 AM Kullu14 ***@***.***> wrote:
@jcanseco <https://github.com/jcanseco> were you trying to use bpfd on
host and target environment (2 VMs, non-android platforms). You tried it on
same machine (remote process).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1675 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AACSVLSpOwP8bGLfwt2qIwPfbNzFZUMjks5ukEItgaJpZM4TO-RN>
.
|
|
@joelagnel I have already tried some of bcc examples locally on the same host. I would really like to test it on host target environment (non-android). Is there anything specific do we need to do to just try some of the examples on host target environment? |
|
@joelagnel, Sorry to bother you again. I could run BPFd on host target environment (non-android). But I need root permission on host to run any tools(filetop.py). Is there any way to run those tools (filetop.py) as normal user ? |
|
unfortunately, filetop.py uses kprobe which requires CAP_SYS_ADMIN. |
|
Thanks Yonghong.
…On Mon, Nov 19, 2018 at 9:07 AM yonghong-song ***@***.***> wrote:
unfortunately, filetop.py uses kprobe which requires CAP_SYS_ADMIN.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
@yonghong-song Thanks for the information. |
|
Could you help debug this a little bit since you have a test case here? |
|
The target loads the BPF program. If the program is of type KPROBE, then you need to running as CAP_SYS_ADMIN. I actually hit this too. Only the networking related program types like SOCK_FILTER donot need CAP_SYS_ADMIN |
|
@joel Sorry.. My mistake.. My question was why do I need root permission for host not target? |
|
@joel I have root permission available on target. But I dont have root permission on host while I am running a script there. why do I need to have root permission on host any idea ? |
|
Your username or shell may not have the cap_sys_admin needed for kprobes. Many bcc tools use kprobes under the hood |
Overview
These patches add features that allow BCC to support cross-development workflows where the development machine and the target machine running the BPF program are different.
This is achieved by integrating the BPFd (BPF Daemon) project into BCC. BPFd is a standalone executable that can be loaded onto a remote target device, and which then can act as a proxy for BCC for whenever BCC wishes to perform an operation on the system (e.g. load BPF programs, read /proc/kallsyms, attach kprobes, etc.).
Through this arrangement, BCC can be used to profile a remote target device while it mostly runs on a separate host machine. The advantage of this arrangement is that it removes the need to have kernel sources and the LLVM stack on the target machine. These can, instead, be kept on the host. This arrangement therefore reduces the space required on a target for BCC tools to run, which is a key benefit for devices that have a more limited disk space (e.g. embedded devices).
In addition, the above set-up also allows embedded developers who use a cross-compiler in their workflow to run clang on a different architecture than the target's architecture, thus facilitating cross-compilation development.
For more information, please check out this LWN article which explains the purpose of BPFd and how it works in more detail.
Integration of BPFd sources into BCC
These patches add the sources for the BPFd executable into the BCC source tree. This is done to ensure that BCC and BPFd remain compatible with each other.
BPFd depends on existing BCC components such as
libbpf.candbcc_syms.ccto function. However, the converse is also true: BCC makes calls to BPFd via BCC's Python interface. The Python interface is the main way by which communication happens between BCC and BPFd, and so any changes there could break interoperability. As a result, it is not sufficient for BPFd to just uselibbccand remain independent like other projects (e.g. bpftrace).Therefore, to ensure that BCC and BPFd are always compatible with each other, it is more feasible to keep them in the same tree instead of keeping them separate. This is also why these patches come with smoke tests which ensure that the interoperability between BCC and BPFd isn't broken silently when changes are made to either.
Tools
The tools that currently work for remote devices with these patches are as follows:
Reviewed-by: Joel Fernandes (joelaf@google.com)