Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpf: Add batch support for attaching trampolines #2158

Closed
wants to merge 30 commits into from
Closed

bpf: Add batch support for attaching trampolines #2158

wants to merge 30 commits into from

Conversation

kernel-patches-bot
Copy link

Pull request for series with
subject: bpf: Add batch support for attaching trampolines
version: 5
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=582267

@kernel-patches-bot
Copy link
Author

Master branch: dd7f091
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=582267
version: 5

@kernel-patches-bot
Copy link
Author

Master branch: 29ad850
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=582267
version: 5

Nobody and others added 24 commits November 19, 2021 07:31
adding ci files
3e5160c
ftrace: Use direct_ops hash in unregister_ftrace_direct
b3915fd 
Now when we have *direct_multi interface the direct_functions
hash is no longer owned just by direct_ops. It's also used by
any other ftrace_ops passed to *direct_multi interface.

Thus to find out that we are unregistering the last function
from direct_ops, we need to check directly direct_ops's hash.

Cc: Steven Rostedt <srostedt@vmware.com>
Fixes: f64dd46 ("ftrace: Add multi direct register/unregister interface")
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
ftrace: Add cleanup to unregister_ftrace_direct_multi
55de4e0 
Adding ops cleanup to unregister_ftrace_direct_multi,
so it can be reused in another register call.

Cc: Steven Rostedt <srostedt@vmware.com>
Fixes: f64dd46 ("ftrace: Add multi direct register/unregister interface")
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
ftrace: Add ftrace_set_filter_ips function
579864e 
Adding ftrace_set_filter_ips function to be able to set filter on
multiple ip addresses at once.

With the *direct_multi interface we have cases where we need to
initialize ftrace_ops object with thousands of functions, so having
single function diving into ftrace_hash_move_and_update_ops with
ftrace_lock is better.

The functions ips are passed as unsigned ong array with count.

Cc: Steven Rostedt <srostedt@vmware.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Factor bpf_check_attach_target function
2461fbf 
Separating the check itself from model distilling and
address search into __bpf_check_attach_target function.

This way we can easily add function in following patch
that gets only function model without the address search,
while using the same code as bpf_check_attach_target.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add bpf_check_attach_model function
4b25b1a 
Adding bpf_check_attach_model function that returns
model for function specified by btf_id. It will be
used in following patches.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
4478414 
Adding bpf_arg/bpf_ret_value helpers for tracing programs
that returns traced function arguments.

Get n-th argument of the traced function:
  long bpf_arg(void *ctx, int n)

Get return value of the traced function:
  long bpf_ret_value(void *ctx)

The trampoline now stores number of arguments on ctx-8
address, so it's easy to verify argument index and find
return value argument.

Moving function ip address on the trampoline stack behind
the number of functions arguments, so it's now stored
on ctx-16 address.

Both helpers are inlined by verifier.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf, x64: Allow to use caller address from stack
34ac25a 
Currently we call the original function by using the absolute address
given at the JIT generation. That's not usable when having trampoline
attached to multiple functions. In this case we need to take the
return address from the stack.

Adding support to retrieve the original function address from the stack
by adding new BPF_TRAMP_F_ORIG_STACK flag for arch_prepare_bpf_trampoline
function.

Basically we take the return address of the 'fentry' call:

   function + 0: call fentry    # stores 'function + 5' address on stack
   function + 5: ...

The 'function + 5' address will be used as the address for the
original function to call.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Keep active attached trampoline in bpf_prog
c0c7b2e 
Keeping active attached trampoline in bpf_prog so it can be used
in following changes to account for multiple functions attachments
in program.

As EXT programs are not going to be supported in multiple functions
attachment for now, I'm keeping them stored in link.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add support to load multi func tracing program
61b978d 
Adding support to load tracing program with new BPF_F_MULTI_FUNC flag,
that allows the program to be loaded without specific function to be
attached to.

Such program will be allowed to be attached to multiple functions
in following patches.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add bpf_trampoline_id object
6e5d66f 
Replacing the bpf_trampoline's key with struct bpf_tramp_id object,
that currently holds only obj_id/btf_id, so same data as key.

Having the key in the struct will allow us to add more ids (functions)
to single trampoline in following patches.

No functional change is intended.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add addr to bpf_trampoline_id object
19e0630 
Adding addr to bpf_trampoline_id object so it's not associated
directly with trampoline directly. This will help us to easily
support multiple ids/addresses support for trampolines coming
in following changes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add struct bpf_tramp_node layer
e393729 
Currently each trampoline holds a list of programs that
are attached to it. With multi func attach support we need
a way for a single program to be connected to multiple
trampolines.

Adding struct bpf_tramp_node object that holds bpf_prog
pointer, so it can be resolved directly. We can now
have multiple struct bpf_tramp_node being attached to
different trampolines pointing to single bpf_prog.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add bpf_tramp_attach layer for trampoline attachment
4f345c2 
Adding bpf_tramp_attach layer for trampoline attachment to
have extra layer on top of the trampoline. The reason is
that in following changes we will add multiple trampolines
for single program and we need entity to hold them.

The api in nutshell:

  - each bpf_prog holds 'bpf_tramp_attach' object, which holds
    list of 'struct bpf_tramp_node' objects:

    struct bpf_tramp_attach {
      struct bpf_tramp_id *id;
      struct hlist_head nodes;
    };

    This allow us to hold multiple trampolines for each program.

  - bpf_tramp_attach returns 'bpf_tramp_attach' object that
    finds trampoline for given 'id' and adds it to the attach
    object, no actuall program attachment is done, just trampoline
    allocation

  - bpf_tramp_attach_link does the actual attachment of the
    program to trampoline

  - bpf_tramp_attach_unlink unlinks all the trampolines present
    in the attach object

  - bpf_tramp_detach frees all the trampolines in attach object

Currently there'll be only single node added in attach object.

Following patches add support for multiple id trampolines,
and uses multiple nodes in attach object to hold trampoline
for given program.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add support to store multiple ids in bpf_tramp_id object
6faafe6 
Adding support to store multiple ids in bpf_tramp_id object,
to have id for trampolines with multiple functions assigned.

Extra array of u32 values is allocated within bpf_tramp_id
object allocation.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add support to store multiple addrs in bpf_tramp_id object
bb635b4 
Adding support to store multiple addrs in bpf_tramp_id object,
to provide address values for id values stored in the object.

The id->addr[idx] returns address value for id->id[idx] id.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add bpf_tramp_id_single function
16814c5 
Adding bpf_tramp_id_single function as interface to
create trampoline with single ID and grouping together
the trampoline allocation with init that is used on
several places and save us few lines.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Resolve id in bpf_tramp_id_single
d57be74 
Moving the id resolving in the bpf_tramp_id_single function
so it's centralized together with the trampoline's allocation
and init.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add refcount_t to struct bpf_tramp_id
ce27c83 
Adding refcount_t to struct bpf_tramp_id so we can
track its allocation and safely use one object on
more places in following changes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add support to attach trampolines with multiple IDs
14daeca 
Adding support to attach trampolines with multiple IDs.

This patch adds support to bpf_tramp_attach function to
attach given program to bpf_tramp_id object that holds
multiple BTF function IDs.

The process of attaching in bpf_tramp_attach is as follows:

  - IDs in bpf_tramp_id object are sorted out to several new
    bpf_tramp_id objects based on number of arguments of each
    ID - so we end up with up to 6 bpf_tramp_id objects, that
    we will create or find suitable trampoline for

  - separating function IDs that have same number of arguments
    save us troubles of handling different argument counts
    within one trampoline

  - now for each such bpf_tramp_id object we do following:

     * search existing trampolines to find match or intersection

     * if there's full match on IDs, we add program to existing
       trampoline and we are done

     * if there's intersection with existing trampoline,
       we split it and add new program to the common part,
       the rest of the IDs are attached to new trampoline

  - we keep trampoline_table as place holder for all trampolines,
    (while the has works only for single ID trampolines) so in case
    there is no multi-id trampoline defined, we still use the fast
    hash trampoline lookup

The bpf_tramp_attach assumes ID array is coming in sorted so it's
possible to run bsearch on it to do all the needed searches.

The splitting of the trampoline use the fact that we carry
'bpf_tramp_attach' object for each bpf_program, so when we split
trampoline that the program is attached to, we just add new
'bpf_tramp_node' object to the program's attach 'nodes'. This way
we keep track of all program's trampolines and it will be properly
detached when the program goes away.

The splitting of the trampoline is done with following steps:

   - lock the trampoline
   - unregister trampoline
   - alloc the duplicate, which means that for all attached programs
     of the original trampoline we create new bpf_tramp_node objects
     and add them to these programs' attach objects
   - then we assign new IDs (common and the rest) to both (original
     and the duplicated) trampolines
   - register both trampolines
   - unlock the original trampoline

This patch only adds bpf_tramp_attach support to attach multiple
ID bpf_tramp_id object. The actual user interface for that comes
in following patch.

Now when each call to bpf_tramp_attach can change any program's attach
object, we need to take trampoline_mutex in both bpf_tramp_attach_link
and bpf_tramp_attach_unlink functions. Perhaps we could add new lock
to bpf_tramp_attach object to get rid of single lock for all.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
bpf: Add support for tracing multi link
3d1bc2b 
Adding new link to allow to attach program to multiple
function BTF IDs.

New fields are added to bpf_attr::link_create to pass
array of BTF IDs:

  struct {
    __aligned_u64   btf_ids;        /* addresses to attach */
    __u32           btf_ids_cnt;    /* addresses count */
  } multi;

The new link code will load these IDs into bpf_tramp_id
and resolve their ips.

The resolve itself is done as per Andrii's suggestion:

  - lookup all names for given IDs
  - store and sort them by name
  - go through all kallsyms symbols and use bsearch
    to find it in provided names
  - if name is found, store the address for the name
  - resort the names array based on ID

If there are multi symbols of the same name the first one
will be used to resolve the address.

The new link will pass them to the bpf_tramp_attach that
does all the work of attaching.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
libbpf: Add btf__find_by_glob_kind function
131d9cb 
Adding btf__find_by_glob_kind function that returns array of
BTF ids that match given kind and allow/deny patterns.

int btf__find_by_glob_kind(const struct btf *btf, __u32 kind,
                           const char *allow_pattern,
                           const char *deny_pattern,
                           __u32 **__ids);

The __ids array is allocated and needs to be manually freed.

At the moment the supported pattern is '*' at the beginning or
the end of the pattern.

Kindly borrowed from retsnoop.

Suggested-by: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
libbpf: Add support to link multi func tracing program
7814d05 
Adding support to link multi func tracing program
through link_create interface.

Adding special types for multi func programs:

  fentry.multi
  fexit.multi

so you can define multi func programs like:

  SEC("fentry.multi/bpf_fentry_test*")
  int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f)

that defines test1 to be attached to bpf_fentry_test* functions.
The test1 program is loaded with BPF_F_MULTI_FUNC flag.

If functions are not specified the program needs to be attached
manually.

Adding new btf_ids/btf_ids_cnt fields to bpf_link_create_opts,
that define functions to attach the program to.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
selftests/bpf: Add bpf_arg/bpf_ret_value test
e0343ef 
Adding tests for bpf_arg/bpf_ret_value helpers on
both fentry and fexit programs.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Jiri Olsa added 6 commits November 19, 2021 07:31
selftests/bpf: Add fentry multi func test
28669a0 
Adding selftest for fentry multi func test that attaches
to bpf_fentry_test* functions and checks argument values
based on the processed function.

We need to cast to real arguments types in multi_arg_check,
because the checked value can be shorter than u64.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
selftests/bpf: Add fexit multi func test
53334b6 
Adding selftest for fexit multi func test that attaches
to bpf_fentry_test* functions and checks argument values
based on the processed function.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
selftests/bpf: Add fentry/fexit multi func test
9dbe9b9 
Adding selftest for fentry/fexit multi func tests that attaches
to bpf_fentry_test* functions and checks argument values based
on the processed function.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
selftests/bpf: Add mixed multi func test
c747545 
Adding selftest for fentry/fexit multi func tests that attaches
to bpf_fentry_test* functions, where some of them have already
attached trampoline.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
selftests/bpf: Add ret_mod multi func test
cc933fc 
Adding extra test to existing modify_return test to
test this with multi func program attached on top
of the modify return program.

Because the supported wildcards do not allow us to
match both bpf_fentry_test* and bpf_modify_return_test,
adding extra code to look it up in kernel's BTF.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
selftests/bpf: Add attach multi func test
dcb00eb 
Adding test code to check on trampolines spliting.

The tests attached various bpf_fetry_* functions in a way
so there's always non trivial IDs intersection, that leads
to trampoline splitting in kenrel code.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
@kernel-patches-bot
Copy link
Author

Master branch: 7615209
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=582267
version: 5

@kernel-patches-bot
Copy link
Author

At least one diff in series https://patchwork.kernel.org/project/netdevbpf/list/?series=582267 expired. Closing PR.

@kernel-patches-bot kernel-patches-bot deleted the series/582267=>bpf-next branch November 21, 2021 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bpf-next changes-requested V5
Projects
None yet
Milestone
No milestone
1 participant
@kernel-patches-bot