Repository navigation
Two new stages for live feeds, lighter file processing, and simpler, more predictable directions. Built on bgpfix v0.28.0.
Highlights
combine: put UPDATEs back together
combine merges UPDATEs from the same peer, with the same attributes, into one UPDATE, eg. on a live stream:
bgpipe -o -- rv-live -- combineOnly the packing changes: every announcement and withdrawal of every prefix is kept, in order. A RouteViews stream split into one prefix per UPDATE (251k messages) comes back as 83k UPDATEs, slightly fewer than the original archive.
mrai: drop the path hunting
mrai sends prefix announcements at most once per window (5s by default) for each peer, like the BGP MRAI timer: only the last announcement of each prefix, withdrawals at once. On 60 seconds of RIS Live it removed 60% of the path hunting (2+ paths, then a withdrawal), with the same last state of every prefix:
bgpipe -o -- ris-live -- mrai -- combineFaster and lighter on files
On a RouteViews 15-minute archive (updates.20240212.0745.bz2: 13 MB, 734k messages), median of 5 runs:
| task | v0.22.1 | v0.23.0 | v0.25.0 | v0.26.0 | v0.27.0 | v0.28.0 |
|---|---|---|---|---|---|---|
| MRT to JSON | 2.33s | 2.01s | 2.00s | 1.98s | 1.29s | 1.25s |
| MRT to MRT | 2.16s | 1.99s | 1.95s | 1.94s | 1.29s | 1.25s |
| MRT to grep to JSON | 2.01s | 1.79s | 1.79s | 1.78s | 1.26s | 1.22s |
| MRT to JSON: CPU time | 3.50s | 2.62s | 2.60s | 2.58s | 2.71s | 2.36s |
| MRT to JSON: memory allocated | 111 MB | 90 MB | 89 MB | 89 MB | 186 MB | 49 MB |
Decompression now takes most of the time: compared to v0.27.0, bgpipe uses 13% less CPU and allocates 4x less memory. UPDATEs are parsed while reading, in parallel to the rest of the pipeline; compressed input re-uses its buffers; write makes 8x fewer write syscalls; message tags cost no allocations. Measured with test/perf/bench.sh.
Simpler directions
-L and -R now choose only what a stage reads: the messages flowing left or right. By default, a stage reads the messages flowing right, and writes new messages to the right. The ends of the pipeline are the exceptions: the first stage reads the messages flowing left, and the last stage writes to the left. With -LR, each message keeps its own direction.
Before, the defaults depended on the type of stage, and some stages ignored -L/-R at the ends of the pipeline. All pipelines in the docs work as before (their --explain output is identical), and the defaults are now predictable: mrai as the last stage no longer passes everything through, and write -L at the end warns instead of being ignored. See Direction.
Router filtering
grep/drop --withdraw: a filtered UPDATE withdraws its prefixes instead of disappearing ("treat-as-withdraw", RFC 7606). On a live session, a dropped UPDATE may leave the router with an older path.- New filters
bogon_asnandbogon_prefix; the edge firewall example adds bogon filtering and "peerlock-lite".
Fixes
- Ctrl-C lost data: all pipe inputs were closed at once, so whatever a stage held was lost, and a
writecould be left as.tmp. Stages now stop in order, each draining into the next. ris-live,rv-livetook 3 seconds to stop.execin the middle of a pipeline never ended:read file.mrt -- exec ./filter.py -- write out.jsonnow finishes when the file is done, and the last output of the command is not lost.- ADD-PATH Path Ids were lost when a message from MRT or BMP was edited (eg. with
update).
Changes to note
- Tags: the well-known tags are ALL-CAPS (
PEER_IP,PEER_AS,PEER_ID,LOCAL_IP,LOCAL_AS,COLLECTOR,ROUTER,SRC); all other tags are lowercase, by conventionstage/name. Renamed:RIS_ID,RIS_HOST->ris-live/id,ris-live/host;RV_COLLECTOR->rv-live/collector;BMP_RIB,PEER_RD,PEER_TYPE->bmp/rib,bmp/rd,bmp/type;INTERFACE->mrt/interface. See the new Message Tags page. exec:--keep-stdinand--keep-stdoutare gone; the stage runs until the command exits.-L/-Ron the first or last stage choose only what the stage reads: it always writes away from its end (unless-LR).combineandmraidon't take-LR: use one stage per direction.grep --withdrawnever drops withdrawals.- Message JSON: tags are written in a fixed order, and
ACTIONis no longer written.
Install
Binaries for Linux, macOS, FreeBSD, OpenBSD, Android and Windows are attached below. Docker: ghcr.io/bgpfix/bgpipe:v0.28.0. Go: go install github.com/bgpfix/bgpipe@v0.28.0.