Skip to content

v0.28.0

Latest

Choose a tag to compare

@github-actions github-actions released this 18 Nov 11:46
· 2 commits to main since this release
aa24f95

Two new stages for live feeds, lighter file processing, and simpler, more predictable directions. Built on bgpfix v0.28.0.

Highlights

combine: put UPDATEs back together

combine merges UPDATEs from the same peer, with the same attributes, into one UPDATE, eg. on a live stream:

bgpipe -o -- rv-live -- combine

Only the packing changes: every announcement and withdrawal of every prefix is kept, in order. A RouteViews stream split into one prefix per UPDATE (251k messages) comes back as 83k UPDATEs, slightly fewer than the original archive.

mrai: drop the path hunting

mrai sends prefix announcements at most once per window (5s by default) for each peer, like the BGP MRAI timer: only the last announcement of each prefix, withdrawals at once. On 60 seconds of RIS Live it removed 60% of the path hunting (2+ paths, then a withdrawal), with the same last state of every prefix:

bgpipe -o -- ris-live -- mrai -- combine

Faster and lighter on files

On a RouteViews 15-minute archive (updates.20240212.0745.bz2: 13 MB, 734k messages), median of 5 runs:

task v0.22.1 v0.23.0 v0.25.0 v0.26.0 v0.27.0 v0.28.0
MRT to JSON 2.33s 2.01s 2.00s 1.98s 1.29s 1.25s
MRT to MRT 2.16s 1.99s 1.95s 1.94s 1.29s 1.25s
MRT to grep to JSON 2.01s 1.79s 1.79s 1.78s 1.26s 1.22s
MRT to JSON: CPU time 3.50s 2.62s 2.60s 2.58s 2.71s 2.36s
MRT to JSON: memory allocated 111 MB 90 MB 89 MB 89 MB 186 MB 49 MB

Decompression now takes most of the time: compared to v0.27.0, bgpipe uses 13% less CPU and allocates 4x less memory. UPDATEs are parsed while reading, in parallel to the rest of the pipeline; compressed input re-uses its buffers; write makes 8x fewer write syscalls; message tags cost no allocations. Measured with test/perf/bench.sh.

Simpler directions

-L and -R now choose only what a stage reads: the messages flowing left or right. By default, a stage reads the messages flowing right, and writes new messages to the right. The ends of the pipeline are the exceptions: the first stage reads the messages flowing left, and the last stage writes to the left. With -LR, each message keeps its own direction.

Before, the defaults depended on the type of stage, and some stages ignored -L/-R at the ends of the pipeline. All pipelines in the docs work as before (their --explain output is identical), and the defaults are now predictable: mrai as the last stage no longer passes everything through, and write -L at the end warns instead of being ignored. See Direction.

Router filtering

  • grep / drop --withdraw: a filtered UPDATE withdraws its prefixes instead of disappearing ("treat-as-withdraw", RFC 7606). On a live session, a dropped UPDATE may leave the router with an older path.
  • New filters bogon_asn and bogon_prefix; the edge firewall example adds bogon filtering and "peerlock-lite".

Fixes

  • Ctrl-C lost data: all pipe inputs were closed at once, so whatever a stage held was lost, and a write could be left as .tmp. Stages now stop in order, each draining into the next.
  • ris-live, rv-live took 3 seconds to stop.
  • exec in the middle of a pipeline never ended: read file.mrt -- exec ./filter.py -- write out.json now finishes when the file is done, and the last output of the command is not lost.
  • ADD-PATH Path Ids were lost when a message from MRT or BMP was edited (eg. with update).

Changes to note

  • Tags: the well-known tags are ALL-CAPS (PEER_IP, PEER_AS, PEER_ID, LOCAL_IP, LOCAL_AS, COLLECTOR, ROUTER, SRC); all other tags are lowercase, by convention stage/name. Renamed: RIS_ID, RIS_HOST -> ris-live/id, ris-live/host; RV_COLLECTOR -> rv-live/collector; BMP_RIB, PEER_RD, PEER_TYPE -> bmp/rib, bmp/rd, bmp/type; INTERFACE -> mrt/interface. See the new Message Tags page.
  • exec: --keep-stdin and --keep-stdout are gone; the stage runs until the command exits.
  • -L/-R on the first or last stage choose only what the stage reads: it always writes away from its end (unless -LR). combine and mrai don't take -LR: use one stage per direction.
  • grep --withdraw never drops withdrawals.
  • Message JSON: tags are written in a fixed order, and ACTION is no longer written.

Install

Binaries for Linux, macOS, FreeBSD, OpenBSD, Android and Windows are attached below. Docker: ghcr.io/bgpfix/bgpipe:v0.28.0. Go: go install github.com/bgpfix/bgpipe@v0.28.0.