Skip to content
Tracing system calls in cgroups using BPF and overwritable ring buffers
Go C Makefile Other
Branch: master
Clone or download
alban Merge pull request #27 from kinvolk/kai/keep-log-for-some-hours
Keep trace for 3 hours instead of 1 minute
Latest commit 2b2bcfb Jan 15, 2020
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/workflows ci: use GitHub actions instead of CircleCI Nov 27, 2019
bpf Add forgotten file straceback-tailcall-caps.h Nov 27, 2019
pkg Keep trace for 3 hours instead of 1 minute Jan 15, 2020
scripts Fix the relation of syscall names to sys_enter_NAME Dec 9, 2019
tools build with a default container image based on the branch Oct 24, 2019
.gitignore Rename project to traceloop May 17, 2019
LICENSE Initial commit Apr 12, 2019
Makefile Print git version at startup Nov 27, 2019
README.md Support both interactive dumping and dumping on exit Dec 9, 2019
bpf.mk pidns guesswork Oct 6, 2019
go.mod go mod: update Nov 27, 2019
go.sum
traceloop-builder.Dockerfile Rename project to traceloop May 17, 2019
traceloop.Dockerfile docker image: run traceloop k8s by default Oct 6, 2019
traceloop.go Support both interactive dumping and dumping on exit Dec 9, 2019

README.md

traceloop

traceloop is a command line tool to trace system calls in a similar way to strace but with some differences:

  • traceloop uses BPF instead of ptrace
  • traceloop's tracing granularity is the control group (cgroup) instead of a process
  • traceloop's traces are recorded in a fast, in-memory, overwritable ring buffer like a flight recorder. The tracing could be permanently enabled and inspected in case of crash.

traceloop can be used directly on the command line or via an HTTP interface.

traceloop has been written to trace Kubernetes Pods with Inspektor Gadget, but it can as easily be used with systemd services that are in their own control groups (look for .service and .scope directories inside /sys/fs/cgroup/unified/).

On the command line

Example with an existing systemd service:

sudo -E ./traceloop cgroups /sys/fs/cgroup/unified/system.slice/sshd.service

Example with a custom command:

sudo systemd-run -t  --unit=test42.service  /bin/sh -c 'for i in $(seq 1 1000) ; do sleep 4 ; echo 2*3*7 | bc > /dev/null ; echo Multiplication $i done. ; done'
...
sudo -E ./traceloop cgroups /sys/fs/cgroup/unified/system.slice/test42.service
...
00:04.022260640 cpu#0 pid 23981 [bc] brk(brk=0) = 94045092683776
00:04.022346588 cpu#0 pid 23981 [bc] ioctl(fd=0, cmd=21505, arg=140721805741680) = 18446744073709551591
00:04.022361201 cpu#0 pid 23981 [bc] read(fd=0, buf=94045092586128 "2*3*7\n", count=8192) = 6
00:04.022401517 cpu#0 pid 23981 [bc] fstat() = 0
00:04.022414650 cpu#0 pid 23981 [bc] ioctl(fd=1, cmd=21505, arg=140721805741312) = 18446744073709551591
00:04.022440173 cpu#0 pid 23981 [bc] write(fd=1, buf=94045092602832 "42\n", count=3) = 3
00:04.022460351 cpu#0 pid 23981 [bc] read(fd=0, buf=94045092586128 "", count=8192) = 0
00:04.022475888 cpu#0 pid 23981 [bc] ioctl(fd=0, cmd=21505, arg=140721805741616) = 18446744073709551591
00:04.022525326 cpu#0 pid 23981 [bc] exit_group(error_code=0)...
00:04.022833827 cpu#2 pid 23961 [sh] ...wait4() = 23981

With Docker

docker run --rm -v /sys/kernel/debug:/sys/kernel/debug -v /sys/fs/cgroup:/sys/fs/cgroup -v /sys/fs/bpf:/sys/fs/bpf -v /run:/run --privileged kinvolk/traceloop

With HTTP interface

sudo -E ./traceloop serve
...

$ sudo curl --unix-socket /run/traceloop.socket 'http://localhost/add?name=sshd&cgrouppath=/sys/fs/cgroup/unified/system.slice/sshd.service'
added as id 0
$ sudo curl --unix-socket /run/traceloop.socket 'http://localhost/list'
0: [sshd] /sys/fs/cgroup/unified/system.slice/sshd.service
$ sudo curl --unix-socket /run/traceloop.socket 'http://localhost/dump-by-cgroup?cgroup=/sys/fs/cgroup/unified/system.slice/sshd.service'
...

You can’t perform that action at this time.