systrace and syscount #786
Comments
|
In terms of implementation, both of these tools would have to use |
|
Just for note: I'm going to release something similar but with accent on performance. It is written in pure C. |
|
Guys, what do you think about idea of samples/bpf/tracex5_*.c in linux kernel sources? They did something similar, but by kprobing seccomp_phase1(). It allow them to load much much faster @4ast, @brendangregg |
|
Can't say I love the idea of installing a dummy seccomp program and then probing an internal seccomp function. This may be quite brittle. What I'd really want is syscall tracepoints. But if this approach works more or less reliably on all kernels from, say, 4.1 -- then maybe ... |
|
@vitalyvch I haven't hacked on this much, but how do we even enable seccomp for the entire system? The tracex5 example seems to only enable seccomp for child processes. Enabling seccomp just to trace syscalls does seem an odd way to do this. |
|
Thanks. I'm not very familiar with seccomp so your answers are really useful for me. I already have added preliminary tracepoint support to my tool, and think about adding that seccomp trick. |
|
@vitalyvch Is your tool something that you plan to release publicly? Is there a likely timeframe? I ask because systrace/syscount can already be immensely useful, and if you think there's a long way to go, we can implement something temporary in the interim. |
|
@vitalyvch Have you been able to make progress on releasing your tools? If not, I think we should consider implementing simple versions of those in Python. @brendangregg |
|
@goldshtn, I'm sorry for delayed reply. So, I do not stop developing. Yes, there is a plan to release, but I should discuss details with team. Currently I'm working on imitating follow-fork option. |
|
I'm working with Vitalii on this project. It's basically a BSD-licensed strace replacement written in C using bcc/ebpf with both binary and text output. I can't give you the exact timeline, but the plan is to release it in a few months. Initially it will be a part of pmem/nvml, but since it will not depend on anything in nvml (except maybe on testing framework), we might push it to separate repository. We'll let you know here when the tool will be ready in at least experimental state. |
|
@vitalyvch |
|
I didn't bisect. I just install night builds every morning, and do manual testing every few days. On Friday tests failed. Most likely something unexpected was introduced on Wednesday or Thursday. |
|
Confirmed, I see this on the latest master. Trying to bisect. |
|
OK, so looks like it was broken by this merge: b77915d |
|
Going to file this on #940. |
This new tool attaches to the raw_syscalls:sys_enter and sys_exit tracepoints, and collects frequency counts and optionally latencies of syscalls (aggregated by syscall or by process). It is a fairly natural and efficient extension of Brendan Gregg's syscount from perf-tools. Options include latency tracing, interval printouts, process filtering, summarizing only failed syscalls, and more. NOTE: The translation of syscall numbers to names is performed using a static list, borrowed from strace sources. It is accurate up to syscall 313, and does not include the bpf() syscall, for example. Also, it is only relevant for x86_64. Basic example: ``` $ syscount -P Tracing syscalls, printing top 10... Ctrl+C to quit. [10:13:21] PID COMM COUNT 30216 sshd 533 31391 vi 494 25188 screen 134 25776 mysqld 24 31394 python 10 494 systemd-journal 5 ^C $ syscount -L Tracing syscalls, printing top 10... Ctrl+C to quit. [10:13:34] SYSCALL COUNT TIME (us) select 132 1902458.009 nanosleep 166 11136.305 write 89 41.308 ftruncate 1 33.217 stat 1 22.117 fstat 1 6.134 [unknown: 321] 28 4.553 ioctl 7 4.544 wait4 166 3.962 timerfd_settime 1 3.786 ^C ``` Related: iovisor#786
This new tool attaches to the raw_syscalls:sys_enter and sys_exit tracepoints, and collects frequency counts and optionally latencies of syscalls (aggregated by syscall or by process). It is a fairly natural and efficient extension of Brendan Gregg's syscount from perf-tools. Options include latency tracing, interval printouts, process filtering, summarizing only failed syscalls, and more. NOTE: The translation of syscall numbers to names is performed using a static list, borrowed from strace sources. It is accurate up to syscall 313, and does not include the bpf() syscall, for example. Also, it is only relevant for x86_64. Basic example: ``` $ syscount -P Tracing syscalls, printing top 10... Ctrl+C to quit. [10:13:21] PID COMM COUNT 30216 sshd 533 31391 vi 494 25188 screen 134 25776 mysqld 24 31394 python 10 494 systemd-journal 5 ^C $ syscount -L Tracing syscalls, printing top 10... Ctrl+C to quit. [10:13:34] SYSCALL COUNT TIME (us) select 132 1902458.009 nanosleep 166 11136.305 write 89 41.308 ftruncate 1 33.217 stat 1 22.117 fstat 1 6.134 [unknown: 321] 28 4.553 ioctl 7 4.544 wait4 166 3.962 timerfd_settime 1 3.786 ^C ``` Related: iovisor#786
|
Some update: Our current code is available here: https://github.com/vitalyvch/strace.ebpf Tools is not released, but could be interesting for you. |
systrace would be an strace-like tool (with limited parsing of arguments). It would provide a general overview of process syscall activity, which can then be examined in more detail with argdist/trace or dedicated tools.
syscount would provide syscall statistics on the whole system or a specific process. Should probably mimic syscount from Brendan Gregg's perf-tools.
The text was updated successfully, but these errors were encountered: