tools/tcpaccept.py's kretprobes not triggered due to low maxactive #1072

alban · 2017-03-27T09:40:19Z

When a kretprobe is installed on a kernel function, there is a maximum limit of how many calls in parallel it can catch (aka "maxactive"). In the case of a eBPF kretprobe, the maxactive is let to the default as defined in kernel/kprobes.c:

        /* Pre-allocate memory for max kretprobe instances */
        if (rp->maxactive <= 0) {
#ifdef CONFIG_PREEMPT
                rp->maxactive = max_t(unsigned int, 10, 2*num_possible_cpus());
#else
                rp->maxactive = num_possible_cpus();
#endif
        }

The maxactive can be as low as 1 on single-cores on kernel without CONFIG_PREEMPT.

tools/tcpaccept.py installs a kretprobe on inet_csk_accept() but that function can sleep for a long time (waiting for an incoming connection).

How to reproduce:

Start the tracer: sudo ./tools/tcpaccept.py
In another shell, start a bunch of processes waiting on the accept() system call: for i in $(seq 1 25) ; do (busybox nc -l -p $((8081 + i)) &) ; done
Start nginx locally: sudo docker run -d nginx
Connect to nginx: curl 172.17.0.2
Notice that there is no "accept" event

Note: use nc from busybox to reproduce this issue. nc from https://nmap.org/ncat (the default on Fedora) will not reproduce the issue because it does not block on the accept() system call (it waits on the select() system call to know when there is an incoming connection and only start the accept() system call at that time).

Similar issue also reported on weaveworks/tcptracer-bpf#24 & weaveworks/tcptracer-bpf#34

The text was updated successfully, but these errors were encountered:

goldshtn · 2017-03-27T10:46:59Z

This is in fact looks like a potential issue for any of our tool that uses kprobes/uprobes. @4ast @brendangregg

alban · 2017-03-27T11:10:25Z

When setting up a kretprobe from a kernel module, it's possible to set up the maxactive to a different value.

But when creating a kretprobe by writing a command to /sys/kernel/debug/tracing/kprobe_events, the maxactive cannot be specified there. Could the command parser for kprobe_events be updated to accept a maxactive= parameter?

I also got this issue on another kernel function that (AFAIK) does not sleep and I don't understand why:
https://lists.iovisor.org/pipermail/iovisor-dev/2017-March/000694.html

alban · 2017-03-27T22:35:17Z

I've tested a patch to make maxactive configurable in /sys/kernel/debug/tracing/kprobe_events: kinvolk/linux@3c713f1

Tested with kinvolk/gobpf@e21cc33

Do you think this approach would be acceptable in upstream?
And do you think there is a workaround possible on current kernels without such a patch?

4ast · 2017-03-28T00:07:06Z

the patch looks good to me.
pls send it upstream and explain in commit log that we need to increase maxactive not only for recursive functions, but for functions that sleep or resched.

When a kretprobe is installed on a kernel function, there is a maximum limit of how many calls in parallel it can catch (aka "maxactive"). A kernel module could call register_kretprobe() and initialize maxactive (see example in samples/kprobes/kretprobe_example.c). But that is not exposed to userspace and it is currently not possible to choose maxactive when writing to /sys/kernel/debug/tracing/kprobe_events The default maxactive can be as low as 1 on single-core with a non-preemptive kernel. This is too low and we need to increase it not only for recursive functions, but for functions that sleep or resched. This patch updates the format of the command that can be written to kprobe_events so that maxactive can be optionally specified. I need this for a bpf program attached to the kretprobe of inet_csk_accept, which can sleep for a long time. BugLink: iovisor/bcc#1072 Signed-off-by: Alban Crequy <alban@kinvolk.io>

alban · 2017-03-28T13:56:38Z

v1: https://lkml.org/lkml/2017/3/28/629
v2: https://lkml.org/lkml/2017/3/31/426
v3: https://lkml.org/lkml/2017/4/3/225
linux-trace for-next: https://lkml.org/lkml/2017/4/5/586
In linux-trace: https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git/commit/?h=for-next&id=696ced4fb1d76802f864d8848aa4716633f83c17
In Linus' tree: torvalds/linux@696ced4

Expected to land in Linux 4.12.

When a kretprobe is installed on a kernel function, there is a maximum limit of how many calls in parallel it can catch (aka "maxactive"). A kernel module could call register_kretprobe() and initialize maxactive (see example in samples/kprobes/kretprobe_example.c). But that is not exposed to userspace and it is currently not possible to choose maxactive when writing to /sys/kernel/debug/tracing/kprobe_events The default maxactive can be as low as 1 on single-core with a non-preemptive kernel. This is too low and we need to increase it not only for recursive functions, but for functions that sleep or resched. This patch updates the format of the command that can be written to kprobe_events so that maxactive can be optionally specified. I need this for a bpf program attached to the kretprobe of inet_csk_accept, which can sleep for a long time. BugLink: iovisor/bcc#1072 Signed-off-by: Alban Crequy <alban@kinvolk.io>

When a kretprobe is installed on a kernel function, there is a maximum limit of how many calls in parallel it can catch (aka "maxactive"). A kernel module could call register_kretprobe() and initialize maxactive (see example in samples/kprobes/kretprobe_example.c). But that is not exposed to userspace and it is currently not possible to choose maxactive when writing to /sys/kernel/debug/tracing/kprobe_events The default maxactive can be as low as 1 on single-core with a non-preemptive kernel. This is too low and we need to increase it not only for recursive functions, but for functions that sleep or resched. This patch updates the format of the command that can be written to kprobe_events so that maxactive can be optionally specified. I need this for a bpf program attached to the kretprobe of inet_csk_accept, which can sleep for a long time. This patch includes a basic selftest: > # ./ftracetest -v test.d/kprobe/ > === Ftrace unit tests === > [1] Kprobe dynamic event - adding and removing [PASS] > [2] Kprobe dynamic event - busy event check [PASS] > [3] Kprobe dynamic event with arguments [PASS] > [4] Kprobes event arguments with types [PASS] > [5] Kprobe dynamic event with function tracer [PASS] > [6] Kretprobe dynamic event with arguments [PASS] > [7] Kretprobe dynamic event with maxactive [PASS] > > # of passed: 7 > # of failed: 0 > # of unresolved: 0 > # of untested: 0 > # of unsupported: 0 > # of xfailed: 0 > # of undefined(test bug): 0 BugLink: iovisor/bcc#1072 Signed-off-by: Alban Crequy <alban@kinvolk.io> --- Changes since v1: - Remove "(*)" from documentation. (Review from Masami Hiramatsu) - Fix support for "r100" without the event name (Review from Masami Hiramatsu) - Get rid of magic numbers within the code. (Review from Steven Rostedt) Note that I didn't use KRETPROBE_MAXACTIVE_ALLOC since that patch is not merged. - Return -E2BIG when maxactive is too big. - Add basic selftest

We are discussing adding a way to specify the maxactive field of a kretprobe in the kernel: iovisor/bcc#1072 (comment)

When a kretprobe is installed on a kernel function, there is a maximum limit of how many calls in parallel it can catch (aka "maxactive"). A kernel module could call register_kretprobe() and initialize maxactive (see example in samples/kprobes/kretprobe_example.c). But that is not exposed to userspace and it is currently not possible to choose maxactive when writing to /sys/kernel/debug/tracing/kprobe_events The default maxactive can be as low as 1 on single-core with a non-preemptive kernel. This is too low and we need to increase it not only for recursive functions, but for functions that sleep or resched. This patch updates the format of the command that can be written to kprobe_events so that maxactive can be optionally specified. I need this for a bpf program attached to the kretprobe of inet_csk_accept, which can sleep for a long time. This patch includes a basic selftest: > # ./ftracetest -v test.d/kprobe/ > === Ftrace unit tests === > [1] Kprobe dynamic event - adding and removing [PASS] > [2] Kprobe dynamic event - busy event check [PASS] > [3] Kprobe dynamic event with arguments [PASS] > [4] Kprobes event arguments with types [PASS] > [5] Kprobe dynamic event with function tracer [PASS] > [6] Kretprobe dynamic event with arguments [PASS] > [7] Kretprobe dynamic event with maxactive [PASS] > > # of passed: 7 > # of failed: 0 > # of unresolved: 0 > # of untested: 0 > # of unsupported: 0 > # of xfailed: 0 > # of undefined(test bug): 0 BugLink: iovisor/bcc#1072 Signed-off-by: Alban Crequy <alban@kinvolk.io>

When a kretprobe is installed on a kernel function, there is a maximum limit of how many calls in parallel it can catch (aka "maxactive"). A kernel module could call register_kretprobe() and initialize maxactive (see example in samples/kprobes/kretprobe_example.c). But that is not exposed to userspace and it is currently not possible to choose maxactive when writing to /sys/kernel/debug/tracing/kprobe_events The default maxactive can be as low as 1 on single-core with a non-preemptive kernel. This is too low and we need to increase it not only for recursive functions, but for functions that sleep or resched. This patch updates the format of the command that can be written to kprobe_events so that maxactive can be optionally specified. I need this for a bpf program attached to the kretprobe of inet_csk_accept, which can sleep for a long time. This patch includes a basic selftest: > # ./ftracetest -v test.d/kprobe/ > === Ftrace unit tests === > [1] Kprobe dynamic event - adding and removing [PASS] > [2] Kprobe dynamic event - busy event check [PASS] > [3] Kprobe dynamic event with arguments [PASS] > [4] Kprobes event arguments with types [PASS] > [5] Kprobe dynamic event with function tracer [PASS] > [6] Kretprobe dynamic event with arguments [PASS] > [7] Kretprobe dynamic event with maxactive [PASS] > > # of passed: 7 > # of failed: 0 > # of unresolved: 0 > # of untested: 0 > # of unsupported: 0 > # of xfailed: 0 > # of undefined(test bug): 0 BugLink: iovisor/bcc#1072 Signed-off-by: Alban Crequy <alban@kinvolk.io> --- Changes since v2: - Explain the default maxactive value in the documentation. (Review from Steven Rostedt) Changes since v1: - Remove "(*)" from documentation. (Review from Masami Hiramatsu) - Fix support for "r100" without the event name (Review from Masami Hiramatsu) - Get rid of magic numbers within the code. (Review from Steven Rostedt) Note that I didn't use KRETPROBE_MAXACTIVE_ALLOC since that patch is not merged. - Return -E2BIG when maxactive is too big. - Add basic selftest

When a kretprobe is installed on a kernel function, there is a maximum limit of how many calls in parallel it can catch (aka "maxactive"). A kernel module could call register_kretprobe() and initialize maxactive (see example in samples/kprobes/kretprobe_example.c). But that is not exposed to userspace and it is currently not possible to choose maxactive when writing to /sys/kernel/debug/tracing/kprobe_events The default maxactive can be as low as 1 on single-core with a non-preemptive kernel. This is too low and we need to increase it not only for recursive functions, but for functions that sleep or resched. This patch updates the format of the command that can be written to kprobe_events so that maxactive can be optionally specified. I need this for a bpf program attached to the kretprobe of inet_csk_accept, which can sleep for a long time. This patch includes a basic selftest: > # ./ftracetest -v test.d/kprobe/ > === Ftrace unit tests === > [1] Kprobe dynamic event - adding and removing [PASS] > [2] Kprobe dynamic event - busy event check [PASS] > [3] Kprobe dynamic event with arguments [PASS] > [4] Kprobes event arguments with types [PASS] > [5] Kprobe dynamic event with function tracer [PASS] > [6] Kretprobe dynamic event with arguments [PASS] > [7] Kretprobe dynamic event with maxactive [PASS] > > # of passed: 7 > # of failed: 0 > # of unresolved: 0 > # of untested: 0 > # of unsupported: 0 > # of xfailed: 0 > # of undefined(test bug): 0 BugLink: iovisor/bcc#1072 Signed-off-by: Alban Crequy <alban@kinvolk.io>

It will help to get missing kretprobe events during the ebpf tracker initialization and missing connect & accept events, specially visible on single-cores: - weaveworks/tcptracer-bpf#36 - iovisor/gobpf#39 It makes use of a Linux patch that should be in Linux >= 4.12 (unreleased yet): iovisor/bcc#1072 (comment) It falls back gracefully on current/older Linux kernels. The new gobpf vendoring also includes other small fixes such as error handling.

When a kretprobe is installed on a kernel function, there is a maximum limit of how many calls in parallel it can catch (aka "maxactive"). A kernel module could call register_kretprobe() and initialize maxactive (see example in samples/kprobes/kretprobe_example.c). But that is not exposed to userspace and it is currently not possible to choose maxactive when writing to /sys/kernel/debug/tracing/kprobe_events The default maxactive can be as low as 1 on single-core with a non-preemptive kernel. This is too low and we need to increase it not only for recursive functions, but for functions that sleep or resched. This patch updates the format of the command that can be written to kprobe_events so that maxactive can be optionally specified. I need this for a bpf program attached to the kretprobe of inet_csk_accept, which can sleep for a long time. This patch includes a basic selftest: > # ./ftracetest -v test.d/kprobe/ > === Ftrace unit tests === > [1] Kprobe dynamic event - adding and removing [PASS] > [2] Kprobe dynamic event - busy event check [PASS] > [3] Kprobe dynamic event with arguments [PASS] > [4] Kprobes event arguments with types [PASS] > [5] Kprobe dynamic event with function tracer [PASS] > [6] Kretprobe dynamic event with arguments [PASS] > [7] Kretprobe dynamic event with maxactive [PASS] > > # of passed: 7 > # of failed: 0 > # of unresolved: 0 > # of untested: 0 > # of unsupported: 0 > # of xfailed: 0 > # of undefined(test bug): 0 BugLink: iovisor/bcc#1072 Link: http://lkml.kernel.org/r/1491215782-15490-1-git-send-email-alban@kinvolk.io Acked-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Alban Crequy <alban@kinvolk.io> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>

alban · 2017-04-13T09:38:56Z

We make use of this kernel patch via:

bcc could do something similar for tools/tcpaccept.py

When a kretprobe is installed on a kernel function, there is a maximum limit of how many calls in parallel it can catch (aka "maxactive"). A kernel module could call register_kretprobe() and initialize maxactive (see example in samples/kprobes/kretprobe_example.c). But that is not exposed to userspace and it is currently not possible to choose maxactive when writing to /sys/kernel/debug/tracing/kprobe_events The default maxactive can be as low as 1 on single-core with a non-preemptive kernel. This is too low and we need to increase it not only for recursive functions, but for functions that sleep or resched. This patch updates the format of the command that can be written to kprobe_events so that maxactive can be optionally specified. I need this for a bpf program attached to the kretprobe of inet_csk_accept, which can sleep for a long time. This patch includes a basic selftest: > # ./ftracetest -v test.d/kprobe/ > === Ftrace unit tests === > [1] Kprobe dynamic event - adding and removing [PASS] > [2] Kprobe dynamic event - busy event check [PASS] > [3] Kprobe dynamic event with arguments [PASS] > [4] Kprobes event arguments with types [PASS] > [5] Kprobe dynamic event with function tracer [PASS] > [6] Kretprobe dynamic event with arguments [PASS] > [7] Kretprobe dynamic event with maxactive [PASS] > > # of passed: 7 > # of failed: 0 > # of unresolved: 0 > # of untested: 0 > # of unsupported: 0 > # of xfailed: 0 > # of undefined(test bug): 0 BugLink: iovisor/bcc#1072 Link: http://lkml.kernel.org/r/1491215782-15490-1-git-send-email-alban@kinvolk.io Acked-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Alban Crequy <alban@kinvolk.io> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>

alban mentioned this issue Mar 27, 2017

missing "accept" events in some scenario: kretprobe maxactive too low for accept() weaveworks/tcptracer-bpf#34

Closed

alban mentioned this issue Mar 27, 2017

offsetguess: workaround missing kretprobes weaveworks/tcptracer-bpf#33

Merged

1 task

alban added a commit to kinvolk/gobpf that referenced this issue Mar 31, 2017

kretprobes: set maxactive

5f20e27

We are discussing adding a way to specify the maxactive field of a kretprobe in the kernel: iovisor/bcc#1072 (comment)

alban mentioned this issue Mar 31, 2017

kretprobes: set maxactive iovisor/gobpf#39

Merged

1 task

alban mentioned this issue Apr 6, 2017

vendor: update gobpf and tcptracer-bpf weaveworks/scope#2428

Merged

iaguis mentioned this issue Jul 11, 2017

Execsnoop can miss some events in parallel environments #1250

Open

Birch-san mentioned this issue Oct 2, 2018

userspace program attached to eBPF SOCKET_FILTER does not receive every skb #1920

Open

pchaigno mentioned this issue Feb 21, 2019

Expose kprobe's maxactive parameter in bcc #2224

Merged

yonghong-song closed this as completed in #2224 Feb 21, 2019

mmat11 mentioned this issue Aug 4, 2022

link: add maxactive for kretprobe cilium/ebpf#755

Merged

alban mentioned this issue Oct 31, 2024

trace_tcpconnect: Fix missing src and dst ip and port on error inspektor-gadget/inspektor-gadget#3517

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools/tcpaccept.py's kretprobes not triggered due to low maxactive #1072

tools/tcpaccept.py's kretprobes not triggered due to low maxactive #1072

alban commented Mar 27, 2017 •

edited

Loading

goldshtn commented Mar 27, 2017

alban commented Mar 27, 2017

alban commented Mar 27, 2017

4ast commented Mar 28, 2017

alban commented Mar 28, 2017 •

edited

Loading

alban commented Apr 13, 2017

tools/tcpaccept.py's kretprobes not triggered due to low maxactive #1072

tools/tcpaccept.py's kretprobes not triggered due to low maxactive #1072

Comments

alban commented Mar 27, 2017 • edited Loading

goldshtn commented Mar 27, 2017

alban commented Mar 27, 2017

alban commented Mar 27, 2017

4ast commented Mar 28, 2017

alban commented Mar 28, 2017 • edited Loading

alban commented Apr 13, 2017

alban commented Mar 27, 2017 •

edited

Loading

alban commented Mar 28, 2017 •

edited

Loading