bpf: Add kprobe multi link #2715

kernel-patches-bot · 2022-03-16T12:34:31Z

Pull request for series with
subject: bpf: Add kprobe multi link
version: 1
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=623878

kernel-patches-bot · 2022-03-16T12:34:32Z

Master branch: 6585abe
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=623878
version: 1

kernel-patches-bot · 2022-03-16T22:17:28Z

Master branch: aaccdf9
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=623878
version: 1

kernel-patches-bot · 2022-03-16T22:37:38Z

Master branch: ad13baf
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=623878
version: 1

kernel-patches-bot · 2022-03-17T19:39:31Z

Master branch: 7cda76d
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=623878
version: 1

kernel-patches-bot · 2022-03-17T19:47:42Z

Master branch: a50cbac
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=623878
version: 1

kernel-patches-bot · 2022-03-17T21:09:02Z

Master branch: a50cbac
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=623878
version: 1

kernel-patches-bot · 2022-03-17T23:16:28Z

Master branch: e0999c8
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=623878
version: 1

Adding support to have priv pointer in swap callback function. Following the initial change on cmp callback functions [1] and adding SWAP_WRAPPER macro to identify sort call of sort_r. Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk> [1] 4333fb9 ("media: lib/sort.c: implement sort() variant taking context argument") Reviewed-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Jiri Olsa <jolsa@kernel.org>

When kallsyms_lookup_name is called with empty string, it will do futile search for it through all the symbols. Skipping the search for empty string. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

Adding new link type BPF_LINK_TYPE_KPROBE_MULTI that attaches kprobe program through fprobe API. The fprobe API allows to attach probe on multiple functions at once very fast, because it works on top of ftrace. On the other hand this limits the probe point to the function entry or return. The kprobe program gets the same pt_regs input ctx as when it's attached through the perf API. Adding new attach type BPF_TRACE_KPROBE_MULTI that allows attachment kprobe to multiple function with new link. User provides array of addresses or symbols with count to attach the kprobe program to. The new link_create uapi interface looks like: struct { __u32 flags; __u32 cnt; __aligned_u64 syms; __aligned_u64 addrs; } kprobe_multi; The flags field allows single BPF_TRACE_KPROBE_MULTI bit to create return multi kprobe. Acked-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Jiri Olsa <jolsa@kernel.org>

Adding support to call bpf_get_func_ip helper from kprobe programs attached by multi kprobe link. Acked-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Jiri Olsa <jolsa@kernel.org>

Adding support to inline it on x86, because it's single load instruction. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

Adding support to call bpf_get_attach_cookie helper from kprobe programs attached with kprobe multi link. The cookie is provided by array of u64 values, where each value is paired with provided function address or symbol with the same array index. When cookie array is provided it's sorted together with addresses (check bpf_kprobe_multi_cookie_swap). This way we can find cookie based on the address in bpf_get_attach_cookie helper. Suggested-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Jiri Olsa <jolsa@kernel.org>

Move the kallsyms parsing in internal libbpf_kallsyms_parse function, so it can be used from other places. It will be used in following changes. Acked-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Jiri Olsa <jolsa@kernel.org>

Adding new kprobe_multi struct to bpf_link_create_opts object to pass multiple kprobe data to link_create attr uapi. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

Adding bpf_program__attach_kprobe_multi_opts function for attaching kprobe program to multiple functions. struct bpf_link * bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog, const char *pattern, const struct bpf_kprobe_multi_opts *opts); User can specify functions to attach with 'pattern' argument that allows wildcards (*?' supported) or provide symbols or addresses directly through opts argument. These 3 options are mutually exclusive. When using symbols or addresses, user can also provide cookie value for each symbol/address that can be retrieved later in bpf program with bpf_get_attach_cookie helper. struct bpf_kprobe_multi_opts { size_t sz; const char **syms; const unsigned long *addrs; const __u64 *cookies; size_t cnt; bool retprobe; size_t :0; }; Symbols, addresses and cookies are provided through opts object (syms/addrs/cookies) as array pointers with specified count (cnt). Each cookie value is paired with provided function address or symbol with the same array index. The program can be also attached as return probe if 'retprobe' is set. For quick usage with NULL opts argument, like: bpf_program__attach_kprobe_multi_opts(prog, "ksys_*", NULL) the 'prog' will be attached as kprobe to 'ksys_*' functions. Also adding new program sections for automatic attachment: kprobe.multi/<symbol_pattern> kretprobe.multi/<symbol_pattern> The symbol_pattern is used as 'pattern' argument in bpf_program__attach_kprobe_multi_opts function. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

Adding kprobe_multi attach test that uses new fprobe interface to attach kprobe program to multiple functions. The test is attaching programs to bpf_fentry_test* functions and uses single trampoline program bpf_prog_test_run to trigger bpf_fentry_test* functions. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

Adding bpf_cookie test for programs attached by kprobe_multi links. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

Adding tests for bpf_program__attach_kprobe_multi_opts function, that test attach with pattern, symbols and addrs. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

kernel-patches-bot · 2022-03-18T03:39:46Z

Master branch: e0999c8
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=623878
version: 1

Adding bpf_cookie test for programs attached by bpf_program__attach_kprobe_multi_opts API. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

kernel-patches-bot · 2022-03-18T03:46:59Z

Master branch: 5a5c11e
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=623878
version: 1

Pull request is NOT updated. Failed to apply https://patchwork.kernel.org/project/netdevbpf/list/?series=623878
error message:

Cmd('git') failed due to: exit code(128)
  cmdline: git am -3
  stdout: 'Applying: lib/sort: Add priv pointer to swap function
Using index info to reconstruct a base tree...
M	include/linux/sort.h
M	include/linux/types.h
M	lib/sort.c
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
Applying: kallsyms: Skip the name search for empty string
Using index info to reconstruct a base tree...
M	kernel/kallsyms.c
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
Applying: bpf: Add multi kprobe link
Using index info to reconstruct a base tree...
M	include/linux/bpf_types.h
M	include/linux/trace_events.h
M	include/uapi/linux/bpf.h
M	kernel/bpf/syscall.c
M	kernel/trace/bpf_trace.c
M	tools/include/uapi/linux/bpf.h
Falling back to patching base and 3-way merge...
Auto-merging tools/include/uapi/linux/bpf.h
CONFLICT (content): Merge conflict in tools/include/uapi/linux/bpf.h
Auto-merging kernel/trace/bpf_trace.c
CONFLICT (content): Merge conflict in kernel/trace/bpf_trace.c
Auto-merging kernel/bpf/syscall.c
CONFLICT (content): Merge conflict in kernel/bpf/syscall.c
Auto-merging include/uapi/linux/bpf.h
CONFLICT (content): Merge conflict in include/uapi/linux/bpf.h
Patch failed at 0003 bpf: Add multi kprobe link
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".'
  stderr: 'error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch'

conflict:

diff --cc include/uapi/linux/bpf.h
index 7604e7d5438f,d77f47af7752..000000000000
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@@ -1487,7 -1487,6 +1487,10 @@@ union bpf_attr 
  				__u32		cnt;
  				__aligned_u64	syms;
  				__aligned_u64	addrs;
++<<<<<<< HEAD
 +				__aligned_u64	cookies;
++=======
++>>>>>>> bpf: Add multi kprobe link
  			} kprobe_multi;
  		};
  	} link_create;
diff --cc kernel/bpf/syscall.c
index cdaa1152436a,b8bb67ee6c57..000000000000
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@@ -4261,7 -4261,7 +4261,11 @@@ static int tracing_bpf_link_attach(cons
  	return -EINVAL;
  }
  
++<<<<<<< HEAD
 +#define BPF_LINK_CREATE_LAST_FIELD link_create.kprobe_multi.cookies
++=======
+ #define BPF_LINK_CREATE_LAST_FIELD link_create.kprobe_multi.addrs
++>>>>>>> bpf: Add multi kprobe link
  static int link_create(union bpf_attr *attr, bpfptr_t uattr)
  {
  	enum bpf_prog_type ptype;
diff --cc kernel/trace/bpf_trace.c
index 9a7b6be655e4,fffa2171fae4..000000000000
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@@ -18,8 -18,6 +18,11 @@@
  #include <linux/btf_ids.h>
  #include <linux/bpf_lsm.h>
  #include <linux/fprobe.h>
++<<<<<<< HEAD
 +#include <linux/bsearch.h>
 +#include <linux/sort.h>
++=======
++>>>>>>> bpf: Add multi kprobe link
  
  #include <net/bpf_sk_storage.h>
  
@@@ -2220,13 -2188,6 +2223,16 @@@ struct bpf_kprobe_multi_link 
  	struct bpf_link link;
  	struct fprobe fp;
  	unsigned long *addrs;
++<<<<<<< HEAD
 +	/*
 +	 * The run_ctx here is used to get struct bpf_kprobe_multi_link in
 +	 * get_attach_cookie helper, so it can't be used to store data.
 +	 */
 +	struct bpf_run_ctx run_ctx;
 +	u64 *cookies;
 +	u32 cnt;
++=======
++>>>>>>> bpf: Add multi kprobe link
  };
  
  static void bpf_kprobe_multi_link_release(struct bpf_link *link)
@@@ -2243,7 -2204,6 +2249,10 @@@ static void bpf_kprobe_multi_link_deall
  
  	kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link);
  	kvfree(kmulti_link->addrs);
++<<<<<<< HEAD
 +	kvfree(kmulti_link->cookies);
++=======
++>>>>>>> bpf: Add multi kprobe link
  	kfree(kmulti_link);
  }
  
@@@ -2252,60 -2212,10 +2261,66 @@@ static const struct bpf_link_ops bpf_kp
  	.dealloc = bpf_kprobe_multi_link_dealloc,
  };
  
++<<<<<<< HEAD
 +static void bpf_kprobe_multi_cookie_swap(void *a, void *b, int size, const void *priv)
 +{
 +	const struct bpf_kprobe_multi_link *link = priv;
 +	unsigned long *addr_a = a, *addr_b = b;
 +	u64 *cookie_a, *cookie_b;
 +	unsigned long tmp1;
 +	u64 tmp2;
 +
 +	cookie_a = link->cookies + (addr_a - link->addrs);
 +	cookie_b = link->cookies + (addr_b - link->addrs);
 +
 +	/* swap addr_a/addr_b and cookie_a/cookie_b values */
 +	tmp1 = *addr_a; *addr_a = *addr_b; *addr_b = tmp1;
 +	tmp2 = *cookie_a; *cookie_a = *cookie_b; *cookie_b = tmp2;
 +}
 +
 +static int __bpf_kprobe_multi_cookie_cmp(const void *a, const void *b)
 +{
 +	const unsigned long *addr_a = a, *addr_b = b;
 +
 +	if (*addr_a == *addr_b)
 +		return 0;
 +	return *addr_a < *addr_b ? -1 : 1;
 +}
 +
 +static int bpf_kprobe_multi_cookie_cmp(const void *a, const void *b, const void *priv)
 +{
 +	return __bpf_kprobe_multi_cookie_cmp(a, b);
 +}
 +
 +static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx, u64 ip)
 +{
 +	struct bpf_kprobe_multi_link *link;
 +	unsigned long *addr;
 +	u64 *cookie;
 +
 +	if (WARN_ON_ONCE(!ctx))
 +		return 0;
 +	link = container_of(ctx, struct bpf_kprobe_multi_link, run_ctx);
 +	if (!link->cookies)
 +		return 0;
 +	addr = bsearch(&ip, link->addrs, link->cnt, sizeof(ip),
 +		       __bpf_kprobe_multi_cookie_cmp);
 +	if (!addr)
 +		return 0;
 +	cookie = link->cookies + (addr - link->addrs);
 +	return *cookie;
 +}
 +
++=======
++>>>>>>> bpf: Add multi kprobe link
  static int
  kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link,
  			   struct pt_regs *regs)
  {
++<<<<<<< HEAD
 +	struct bpf_run_ctx *old_run_ctx;
++=======
++>>>>>>> bpf: Add multi kprobe link
  	int err;
  
  	if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) {
@@@ -2315,9 -2225,7 +2330,13 @@@
  
  	migrate_disable();
  	rcu_read_lock();
++<<<<<<< HEAD
 +	old_run_ctx = bpf_set_run_ctx(&link->run_ctx);
 +	err = bpf_prog_run(link->link.prog, regs);
 +	bpf_reset_run_ctx(old_run_ctx);
++=======
+ 	err = bpf_prog_run(link->link.prog, regs);
++>>>>>>> bpf: Add multi kprobe link
  	rcu_read_unlock();
  	migrate_enable();
  
@@@ -2403,11 -2311,9 +2422,17 @@@ int bpf_kprobe_multi_link_attach(const 
  {
  	struct bpf_kprobe_multi_link *link = NULL;
  	struct bpf_link_primer link_primer;
++<<<<<<< HEAD
 +	void __user *ucookies;
 +	unsigned long *addrs;
 +	u32 flags, cnt, size;
 +	void __user *uaddrs;
 +	u64 *cookies = NULL;
++=======
+ 	unsigned long *addrs;
+ 	u32 flags, cnt, size;
+ 	void __user *uaddrs;
++>>>>>>> bpf: Add multi kprobe link
  	void __user *usyms;
  	int err;
  
@@@ -2447,19 -2353,6 +2472,22 @@@
  			goto error;
  	}
  
++<<<<<<< HEAD
 +	ucookies = u64_to_user_ptr(attr->link_create.kprobe_multi.cookies);
 +	if (ucookies) {
 +		cookies = kvmalloc(size, GFP_KERNEL);
 +		if (!cookies) {
 +			err = -ENOMEM;
 +			goto error;
 +		}
 +		if (copy_from_user(cookies, ucookies, size)) {
 +			err = -EFAULT;
 +			goto error;
 +		}
 +	}
 +
++=======
++>>>>>>> bpf: Add multi kprobe link
  	link = kzalloc(sizeof(*link), GFP_KERNEL);
  	if (!link) {
  		err = -ENOMEM;
@@@ -2479,21 -2372,6 +2507,24 @@@
  		link->fp.entry_handler = kprobe_multi_link_handler;
  
  	link->addrs = addrs;
++<<<<<<< HEAD
 +	link->cookies = cookies;
 +	link->cnt = cnt;
 +
 +	if (cookies) {
 +		/*
 +		 * Sorting addresses will trigger sorting cookies as well
 +		 * (check bpf_kprobe_multi_cookie_swap). This way we can
 +		 * find cookie based on the address in bpf_get_attach_cookie
 +		 * helper.
 +		 */
 +		sort_r(addrs, cnt, sizeof(*addrs),
 +		       bpf_kprobe_multi_cookie_cmp,
 +		       bpf_kprobe_multi_cookie_swap,
 +		       link);
 +	}
++=======
++>>>>>>> bpf: Add multi kprobe link
  
  	err = register_fprobe_ips(&link->fp, addrs, cnt);
  	if (err) {
@@@ -2506,7 -2384,6 +2537,10 @@@
  error:
  	kfree(link);
  	kvfree(addrs);
++<<<<<<< HEAD
 +	kvfree(cookies);
++=======
++>>>>>>> bpf: Add multi kprobe link
  	return err;
  }
  #else /* !CONFIG_FPROBE */
@@@ -2514,8 -2391,4 +2548,11 @@@ int bpf_kprobe_multi_link_attach(const 
  {
  	return -EOPNOTSUPP;
  }
++<<<<<<< HEAD
 +static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx, u64 ip)
 +{
 +	return 0;
 +}
++=======
++>>>>>>> bpf: Add multi kprobe link
  #endif
diff --cc tools/include/uapi/linux/bpf.h
index 7604e7d5438f,d77f47af7752..000000000000
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@@ -1487,7 -1487,6 +1487,10 @@@ union bpf_attr 
  				__u32		cnt;
  				__aligned_u64	syms;
  				__aligned_u64	addrs;
++<<<<<<< HEAD
 +				__aligned_u64	cookies;
++=======
++>>>>>>> bpf: Add multi kprobe link
  			} kprobe_multi;
  		};
  	} link_create;

kernel-patches-bot · 2022-03-18T03:54:01Z

At least one diff in series https://patchwork.kernel.org/project/netdevbpf/list/?series=623878 irrelevant now. Closing PR.

kernel-patches-bot added bpf-next new V1 labels Mar 16, 2022

kernel-patches-bot force-pushed the series/616820=>bpf-next branch from bebbef3 to c40649f Compare March 16, 2022 22:17

kernel-patches-bot force-pushed the series/616820=>bpf-next branch from c40649f to d3f5a5a Compare March 16, 2022 22:37

kernel-patches-bot force-pushed the series/616820=>bpf-next branch from d3f5a5a to cd47995 Compare March 17, 2022 19:39

kernel-patches-bot force-pushed the series/616820=>bpf-next branch from cd47995 to 8c23ecf Compare March 17, 2022 19:47

kernel-patches-bot force-pushed the series/616820=>bpf-next branch from 8c23ecf to 00db917 Compare March 17, 2022 21:09

kernel-patches-bot force-pushed the bpf-next branch from a50cbac to e0999c8 Compare March 17, 2022 23:12

kernel-patches-bot force-pushed the series/616820=>bpf-next branch from 00db917 to c452a94 Compare March 17, 2022 23:16

Nobody and others added 13 commits March 17, 2022 20:39

adding ci files

6389715

kallsyms: Skip the name search for empty string

172ad37

When kallsyms_lookup_name is called with empty string, it will do futile search for it through all the symbols. Skipping the search for empty string. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

bpf: Add bpf_get_func_ip kprobe helper for multi kprobe link

1c1f61a

Adding support to call bpf_get_func_ip helper from kprobe programs attached by multi kprobe link. Acked-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Jiri Olsa <jolsa@kernel.org>

bpf: Add support to inline bpf_get_func_ip helper on x86

9510306

Adding support to inline it on x86, because it's single load instruction. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

libbpf: Add libbpf_kallsyms_parse function

4029e81

Move the kallsyms parsing in internal libbpf_kallsyms_parse function, so it can be used from other places. It will be used in following changes. Acked-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Jiri Olsa <jolsa@kernel.org>

libbpf: Add bpf_link_create support for multi kprobes

316a36c

Adding new kprobe_multi struct to bpf_link_create_opts object to pass multiple kprobe data to link_create attr uapi. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

selftests/bpf: Add kprobe_multi bpf_cookie test

34a8aba

Adding bpf_cookie test for programs attached by kprobe_multi links. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

selftests/bpf: Add attach test for bpf_program__attach_kprobe_multi_opts

dfc512f

Adding tests for bpf_program__attach_kprobe_multi_opts function, that test attach with pattern, symbols and addrs. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

selftests/bpf: Add cookie test for bpf_program__attach_kprobe_multi_opts

1ccfdb6

Adding bpf_cookie test for programs attached by bpf_program__attach_kprobe_multi_opts API. Signed-off-by: Jiri Olsa <jolsa@kernel.org>

kernel-patches-bot force-pushed the series/616820=>bpf-next branch from c452a94 to 1ccfdb6 Compare March 18, 2022 03:39

kernel-patches-bot added the merge-conflict label Mar 18, 2022

kernel-patches-bot added accepted and removed merge-conflict new labels Mar 18, 2022

kernel-patches-bot closed this Mar 18, 2022

kernel-patches-bot deleted the series/616820=>bpf-next branch March 18, 2022 03:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpf: Add kprobe multi link #2715

bpf: Add kprobe multi link #2715

kernel-patches-bot commented Mar 16, 2022

kernel-patches-bot commented Mar 16, 2022

kernel-patches-bot commented Mar 16, 2022

kernel-patches-bot commented Mar 16, 2022

kernel-patches-bot commented Mar 17, 2022

kernel-patches-bot commented Mar 17, 2022

kernel-patches-bot commented Mar 17, 2022

kernel-patches-bot commented Mar 17, 2022

kernel-patches-bot commented Mar 18, 2022

kernel-patches-bot commented Mar 18, 2022

kernel-patches-bot commented Mar 18, 2022

bpf: Add kprobe multi link #2715

bpf: Add kprobe multi link #2715

Conversation

kernel-patches-bot commented Mar 16, 2022

kernel-patches-bot commented Mar 16, 2022

kernel-patches-bot commented Mar 16, 2022

kernel-patches-bot commented Mar 16, 2022

kernel-patches-bot commented Mar 17, 2022

kernel-patches-bot commented Mar 17, 2022

kernel-patches-bot commented Mar 17, 2022

kernel-patches-bot commented Mar 17, 2022

kernel-patches-bot commented Mar 18, 2022

kernel-patches-bot commented Mar 18, 2022

kernel-patches-bot commented Mar 18, 2022