Skip to content
eBPF based syscalls, files and network events tracing framework
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
battery
bpf
cli
documentation
examples
generator
metagenerator
metrics
probe
tests
tracer
vendor
.gitignore
Gopkg.lock
Gopkg.toml
LICENSE
Makefile
README.md
builder.Dockerfile
traceleft-block.png

README.md

TraceLeft

TraceLeft is a library to trace applications as well as a small CLI tool (traceleft) which acts as a reference implementation of the framework. It uses Linux eBPF and Kprobes to install probes on Linux function calls (both APIs and other internal functions) in order to receive callbacks for syscalls, file and network events of a traced process. TraceLeft is built using gobpf and takes inspiration from the BCC toolkit. TraceLeft has been designed as a framework to build configuration driven system auditing tools as well as application tracing tools used for network and syscall monitoring. TraceLeft has been tested on kernel versions v4.4.0 till v4.16.0 with eBPF support for Kprobes and Kretprobes. Though eBPF support for static tracepoints has landed in recent kernels, one of the early goals of TraceLeft was to have it run on older kernels with early eBPF support. Tracepoint support is in the works.

The following diagram shows how a set of syscalls and other events from an application can be hooked onto using TraceLeft and then eventually tracked through the lifecycle of the traced application

block-diagram

Decisions on what process to track and what data to collect per-event can be configured to a very fine granularity using Proto/JSON configs. Targeted eBPF handlers are generated based on a pre-defined config.json. Such a config eventually generates a battery of compiled eBPF programs that handle each syscall or a network event as the configuration desires. All the eBPF handlers are controlled via a main eBPF program. When each handler fires as the tracked application executes, it generates an Event which is transmitted via the perf map to userspace. And event can then be aggregated via a reference event aggregator implementation that allows setting filtering rules on each collected event and provides specifications for aggregating events and transferring them over the wire in proto format or to a local file.

Detailed documentation can be found in documentation directory.

Quickstart

Building the traceleft binary requires Docker. Details of builds steps are outlined in the build process documentation.

make
sudo build/bin/traceleft trace $PID1,$PID2:battery/out/handle_syscall_read.bpf $PID3:battery/out/handle_syscall_chown.bpf

The $PID is optional and can be skipped to load a handler as default handler and trace all processes instead. For example, to trace all read syscalls from vim

[root@gondor traceleft]# build/bin/traceleft trace $(pidof vim):battery/out/handle_syscall_read.bpf 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "i" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "W" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "o" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "l" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "o" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "l" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "o" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "\x1b" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf ":" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "w" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "q" Count 4096 
name read pid 22305 program id 0 return value 1 hash 7914472926735816156 Fd 0<unknown> Buf "\r" Count 4096 

Tests

Test can be run using the testing script provided in tests directory:

sudo -E tests/run.sh

Expected output

Using outfile /tmp/traceleft-test-cli-out-Ecw373
Using outdir /tmp/traceleft-trace-out
Running test_sys_chmod with PID: 7996               [PASSED]
Running test_sys_chown with PID: 8045               [PASSED]
Running test_sys_close with PID: 8099               [PASSED]
...

Related Projects

  • BCC Toolkit: It is a collection of cutting-edge tools that provide advanced tracing capabilities using eBPF. TraceLeft differs from BCC by focusing on maintaining a balance of being configurable as well as being easily composed and deployed with minimum dependencies.
  • bpfd: This project focuses on the same goals as TraceLeft - of providing filtered trace support on certain system events, it has an additional feature of taking actions based on events and provides a daemon to create and remove rules dynamically. In contrast, TraceLeft aims to build single self- contained binaries to provide build-time configured tracing support - while allowing much more fine grained control over data obtained from kernel events. Unlike bpfd, TraceLeft does not reply on BCC but does require direct compilation of BPF code
  • BPFtrace: This is a high-level language which allows a easy tracing session control - much like Dtrace and SystemTap and relies on BCC. It compiles BPF code for the scripts directly using LLVM backend APIs while TraceLeft's BPF battery compilation is done using clang itself.

Contributors

©2018 Shiftleft Inc.

You can’t perform that action at this time.