Skip to content

mk-fg/linux-ebpf-connection-overseer

Repository files navigation

Linux eBPF Connection Overseer

Network monitoring tool to display info about new/active app connections on linux desktop, with enough associated process/cgroup information to easily tell what those might be about and some traffic counters, displayed in a way similar to conky tool (i.e. over desktop background or in a transparent overlay window).

Focus is to be useful in the same way conky is - be able to check what's going on at a glance, easily spotting something new or unusual, with enough configurability to remove or filter-out any system-specific irrelevant noise.

Uses system-wide eBPF hooks to gather connection/traffic data in-kernel efficiently, and also presents it via SDL3 in a fairly direct way, without any extra abstractions.

Table of Contents:

Repository URLs:

Roughly how it looks on a second screen here (as of 2025-07-28 version):

leco-sdl-widget.mp4

Technical details

Consists of several components:

  • Self-contained leco-ebpf-load binary (mostly generated by bpftool gen skeleton), that only loads eBPF blob into kernel and either stores file descriptors of data maps in systemd service File Descriptor Store, pins those in /sys/fs/bpf, or no-op exits if it's done already.

    Requires elevated privileges, intended to be run as ExecStartPre=+... command.
    Without pinning, when service stops, fds get closed, which cleans up eBPFs as well.

  • Unprivileged leco-event-pipe system-daemon python script that uses eBPF map file descriptors (passed to it) to monitor network connection events, augment, filter, string-format and export those to some fifo socket/pipe as json lines.

    leco@.service + leco-fifo@.service systemd unit files can be used to run loader binary and start this script, with desktop user uid as an instance name (to use the right XDG_RUNTIME_DIR for fifo, e.g. systemctl start leco@1000).

  • Desktop session leco-sdl-widget to read event info and visualize it in a relatively simple way.

It's kinda similar to OpenSnitch and netatop-bpf projects (and is partly based on those), with simple non-interactive read-only scope, conky-like transparent window/overlay graphical frontend, and privileged access limited to loading persistent data-collection eBPFs once.

Build / Requirements

In short:

  • Build-only dependencies/tools: cmake, make, llvm, clang, Nim.

  • Runtime dependencies (for all components): libc (tested with glibc), libelf, zlib, python, libbpf, SDL3, SDL3_ttf, linux 6.3+ kernel with eBPF, kprobes and tracepoints enabled.

  • Outputs: leco-ebpf-load, leco-event-pipe, leco-sdl-widget, leco-sdl-widget.ini

  • Extra files to start from systemd: leco@.service, leco-fifo@.service

Run git submodule init && git submodule update && make to build all output files as-needed.

More specifically:

  • Basic system dependencies like libc (e.g. glibc), libelf and zlib are needed for build and at runtime. Usually present in any regular linux distro/setup, and are also runtime dependencies of libbpf.

  • leco-ebpf-load loader-binary requires make, clang and llvm tools, as well as kernel headers (e.g. kernel-devel package) to build it.

    Running resulting eBPF-binary on a system needs linux 6.3+, with following config options:

    % bsdcat /proc/config.* | grep -E 'CONFIG_(BPF|TRACEPOINTS|KPROBES)='
    ## Alternatively: grep -E 'CONFIG_(BPF|TRACEPOINTS|KPROBES)=' /boot/config-$(uname -r)
    ## Or also: cat /sys/kernel/debug/tracing/events/sock/sock_send_length/format
    CONFIG_BPF=y
    CONFIG_TRACEPOINTS=y
    CONFIG_KPROBES=y

    (which can be enabled via BPF_SYSCALL + FTRACE + KPROBE_EVENTS)

    Run git submodule init && git submodule update first to fetch libbpf/bpftool/sd-daemon build-dependencies. make leco-ebpf-load command can be used to build only this leco-ebpf-load eBPF-loader binary (~400K).

  • leco-event-pipe script requires python (with ctypes) and libbpf installed, uses fds/pins created by leco-ebpf-load, that should be run before it.

  • leco-sdl-widget requires Nim, make and cmake (to build tinyspline submodule), and only SDL3 + SDL3_ttf (Simple DirectMedia Layer 3.x) libraries installed on the system in order to run.

    Will not work with older SDL 1.x and 2.x library branches (different API).

    git submodule init && git submodule update needs to be run first to fetch tinyspline build-time dependency library. Can be built separately using make leco-sdl-widget command, ~400K binary.

Running make without parameters includes building all these components.

Usage

After building all the stuff using make (see Build / Requirements section above), there should be following output files for two general parts:

  • Data collection part: leco-ebpf-load + leco-event-pipe + leco@.service + leco-fifo@.service

    Install binaries into PATH somewhere, and .service files to /etc/systemd/system, then enable/start it with desktop user's uid as an instance name (uid as in id -u, often 1000): systemctl enable --now leco@1000

    Run leco-event-pipe with -h/--help for a list of options, which can be customized using environment variables, added via e.g. systemctl edit leco@ command and Environment= or EnvironmentFile= lines there (see man systemd.exec).

    Or, alternatively, e.g. for manual testing, these can be (re-)started separately from terminal via following commands:

    ./leco-ebpf-load -p /sys/fs/bpf/leco -v
    ./leco-event-pipe -p /sys/fs/bpf/leco -f events.fifo
    

    Both should require root access when started this way - use leco@.service with systemd FDStore= mechanism to avoid that, or maybe chown on persistent /sys/fs/bpf/leco bpf-pins path that loader binary creates.

    rm -rf /sys/fs/bpf/leco can be used to cleanup those loaded eBPFs afterwards.
    cat events.fifo to see all captured connection information/events.
    leco-event-pipe -p /sys/fs/bpf/leco -EC - dump recent event data anytime.
    bpftool map dump pinned /sys/fs/bpf/leco/maps/... to check raw eBPF data.

  • Visualization part: leco-sdl-widget + leco-sdl-widget.ini

    Tweak leco-sdl-widget.ini configuration file as-needed, though it should work fine with all defaults.

    Run ./leco-sdl-widget, or leco-sdl-widget leco-sdl-widget.ini to use edited config file - by default this should create a transparent borderless window (kinda similar to well-known conky desktop monitoring tool), either displaying network connection info lines, or blank until it can connect to specified fifo pipe (created by leco-event-pipe component above).

Both parts can be started/stopped separately anytime, widget uses inotify to wait for fifo pipe to be created if there isn't one already.

There's nothing interactive here - widget just stays where it's configured to be and displays whatever is captured by ebpf hooks, filtering/grouping connection lines as per [rx-proc] and [rx-group] sections in the ini file.

Should use modest cpu/mem amounts (~0.05% or ~6 cpu-ms/s + 60M mem for sdl app on R5 5600X, less than conky), not add any significant kernel overhead (very basic "put value into LRU cache" hooks), and intended to run indefinitely as a desktop overlay or background widget.

More specific features and configuration options are described in more detail below.

Regular expressions in [rx-*] sections

Those regexps process "line" keys which can be seen if you e.g. run cat /run/user/1000/leco.fifo or widget with -d/--debug option (or debug = yes set in the ini config file).

leco-sdl-widget.ini produced by make or its source widget.ini file should have a bunch of examples of such regexp replacement/grouping rules and more description in the comments above.
They can look like this (with # ... inline comments at the end):

[rx-proc]
^(\s*\S+ ::) myuser :: = $1 # only keep usernames other than default local user
\ 53/udp :: = \ DNS :: # nicer name for a common port
\ \[tmux-spawn-\S+\] :: = \ [tmux] :: # shorten long cgroup names

So, for example, connection-info line from fifo like this:

18:32 :: myuser :: drill [tmux-spawn-822f3728+] :: 10.2.0.1 53/udp :: v 58B / 26B ^

Will be changed by those regexps to this: 18:32 :: drill [tmux] :: 10.2.0.1 DNS :: v 58B / 26B ^

Regular expression syntax is PCRE. Put that line above as "test string" into e.g. regex101.com, along with regexp keys like ^(\s*\S+ ::) myuser :: (as "regular expression" above) to see how it all works and make new rules like that.

Regexps configured in the ini file can be tested against any line using tool's -r/--rx-test option - e.g. ./leco-sdl-widget leco-sdl-widget.ini -r '18:32 :: myuser :: ...' - which will print how that exact line should be replaced or grouped, if anything matches it.

Often apps like web browsers tend to make a lot of connections, which might not be that interesting or relevant, and using g`... replacements in [rx-proc] section or [rx-group] matches can be used to group all those into one output slot/line.

Regexps can also be messy to match multiple things anywhere on the line, or to say "replace if this doesn't match", so config file also supports stringing multiple ones together (with AND logic), for example: /match this/ && /and that/ && !/but not this/.

/.../ wrapper around regexps can also be used with flags, e.g. I/.../ for case-sensitive matches (default is case-insensitive), and in addition to g`... "group under replacement key" substitution values can also have index of regexp in the chain to replace, e.g. 1`with this to replace /match this/ in example above (instead of last matched non-negative regexp, which is the default).

There's usually no need to bother with all this complicated extra syntax though - just <regexp> = <replacement> PCREs should be enough for most cases.

line-fade-curve spline editor

Connection info lines can be configured to fade-in and fade-out, and generally change opacity over time since last activity, for a period configured by line-fade-time.

line-fade-curve parameter allows to set any number of (relative-time, opacity) points, which are then connected by a smooth cardinal cubic curve, and transparency of info text follows that curve over line-fade-time number of seconds.

spline-editor.html file in this repository is a simple editor for such curves, which allows to visually add any number of points, move them around and see connecting curve with values interpolated in-between those points. Open in a browser (as in Ctrl-O) to use it.

E.g. a curve like this from widget.ini file example:

line-fade-curve = range=0:100 0,41 2,62 4,80 6,94 18,98 58,100 97,84 161,67
  270,63 340,60 395,48 560,46 645,45 715,43 774,31 834,27 902,24 953,17 994,11

Will quickly fade-in new connection infos, and then fade those out in roughly 3 steps, down to 11% opacity at the end, so that it'd be apparent at a glance which connections are new, which are slightly older, which haven't seen activity for a couple minutes, and which have faded into irrelevance by now.

leco-sdl-widget.ini (widget.ini) has more description of how that value works, but it's just a list of "x,y" points to build a curve in-between, with optional "range=" prefix to set min/max for y, so that e.g. last "994,11" point in example above is read as 11% instead of 0% opacity, with range (and 0%) set from highest/lowest values by default.

Traffic counters aren't always displayed

Done for efficiency reasons - traffic update events are rate-limited to one in 3s (see CONN_TRX_UPD_NS constant in ebpf.c), so new or quick one-off connections that close within seconds won't really get counted properly.

It doesn't seem to be useful to count couple KiB on those anyway, so traffic counters just aren't displayed there to have less visual noise, until rx+tx traffic crosses basic 5K threshold (configurable via conn-traffic-min -c/--conf option or env-vars in leco-event-pipe script).

Which traffic gets detected/counted

Tracepoints used in ebpf.c (sock_send_length and sock_recv_length) currently only detect bytes that pass through all variations of send() and recv() kernel syscalls on IPv4/IPv6 sockets.

That is, they detect all userspace apps calling e.g. socket.send(data), even if resulting packet is immediately blocked by firewall, or even if send() is called on a closed/broken socket, and same for recv - anything returned to userspace is counted, regardless of whether it's from actual network or generated/injected somewhere else.

Conversely, network traffic generated by TCP connect() calls in userspace is not counted/displayed, as it doesn't pass through those send/recv syscalls, which means that new/idle connections are ignored, until app sends/receives some data there.

All network traffic generated or forwarded by linux kernel internally is ignored as well - there're no send/recv syscalls involved there. This includes traffic of in-kernel network filesystems like NFS, CIFS, Ceph, or e.g. encrypted UDP packets of WireGuard tunnels. If host works as a router moving traffic between interfaces, that's not detected/displayed either, use network flow monitoring tools for that.

Traffic of address families other than AF_INET / AF_INET6 might be passing through these tracepoints, but will be ignored by eBPF code as not relevant for this tool. Regular loopback traffic over 127.0.0.1, ::1 and such localhost IPs is same AF_INET(6), so picked-up as well. Local unix sockets (AF_UNIX, such as /run/dbus/system_bus_socket) are not intended to be displayed by this tool and are not handled by eBPF tracepoints.

All application-visible IP protocols other than TCP/UDP (like ICMP or SCTP) are relayed and displayed in the widget as "x" (e.g. 10.83.1.4 189/x) - not ignored, but somewhat rare to see, so kinda bundled together as "others" like that.

IPv4/IPv6 addresses displayed by the tool are copied from kernel inet socket data structures, and should match what application opening the socket connects to, but can be different from the actual bytes in sent/received packets, if for example network address translation (NAT) is setup in a local firewall. Arbitrary traffic constructed on type=SOCK_RAW sockets doesn't have any "connection" information recorded in kernel, so shows up with remote addr=0.0.0.0 port=0 atm.

To summarize: eBPF hooks here detect and count userspace applications attempting to send or receiving data via any IPv4/IPv6 sockets, which might not be same as actual network/interface traffic, and don't count kernel-initiated or forwarded traffic at all.

Known limitations and things to improve later

It's working fine for me as it is, but there's always plenty of room for improvement.

  • Unfinished and minor stuff.

    • widget: +1 fade curve for lingering connections, with configurable rgba-mod/timespan.
    • widget: effects for new/faded conns added to the list (glow/blur, slide, etc).
    • widget: custom colors/curves selectable by regexp matches.
    • ebpf: add kprobes to update icmp destination addresses.
    • Some distinction for in/out conns (accept/recvmsg vs connect/sendmsg).
  • Fetch DNS info for addresses via separate API, from e.g. local resolver.

    Idea is to reverse-query local unbound cache, to return last name that resulted in it returning specified IP, which should work for all local apps, incl. ones using such resolver via DoH protocol.

  • Show RIPE/ASN network info from local databases.

    nft-set-to-ranges script has implementation for some of this.

  • Add option to use icons from a separate emoji-font using regexps, instead of UIDs.

    Should track same known things better than UID hashes on the left, be more consistent, and more distinctive at a glance due to different shapes.

  • Send events from pipe to multiple receivers.

    Coupled with widget-side filtering, should allow to have different windows for different data, or maybe just round-robin events between those based on key hash.

  • Add sending of eBPF event info over network sockets (unix, tcp/udp).

    Should allow for easily opening desktop widget to monitor remote host. Currently can be done by running socat from fifo to socket and back to fifo, but should be easy enough to support in the python script and widget too.

  • Maybe track wireguard tunnel connections.

    Kprobes for those were in earlier eBPF code, but removed in 724adb5 in favor of much simpler tracepoints for everything. Doesn't seem very useful to track those though, as it's mostly noise - app connections over those tunnels are interesting and are tracked already.

  • Add eBPF hooks to detect/mark whether connections are blocked/firewalled.

    Can be done for send() by matching data going out from same socket in some egress hook (maybe mangled in some way, but linked to same socket), or maybe checking connection state (for TCP) or whatever similar counters/stats.

    With current tracking, connect() should fail and TCP ones won't be displayed, while UDP sockets might show (mostly) one-sided traffic (if recv works, blocked send attempts will still be counted too).

Links

About

Desktop network monitoring widget to display connections with process/cgroup info in an overlay window

Topics

Resources

License

Stars

3 stars

Watchers

1 watching

Forks

Contributors