Skip to content
Fast tcp-info collector in Go
Go Dockerfile
Branch: master
Clone or download
gfr10598 Handle connections that drop snapshot in final states (#103)
* use HasInetDiag

* fix negative delta bug

* log only 10K closes, to limit log spam

* fix assignment bug

* fix broken test

* partially implemented test of FIN_WAIT2 behavior

* friendly test filename

* tweak timestamp handling and TODO
Latest commit 8af213a Aug 27, 2019

README.md

tcp-info

GoDoc Build Status Go Report Card Coverage Status

The tcp-info tool executes a polling loop that tracks the measurement statistics of every open TCP socket on a system. Data is written, in JSONL format (refered to internally as ArchivedRecord), to files compressed using zstd. This tool forms the basis of a lot of measurements on the Kubernetes-based Measurement Lab platform.

We expect most people will run this tool using a docker container. To invoke, with data written to ~/data, and prometheus metrics published on port 7070:

docker run --network=host -v ~/data:/home/ -it measurementlab/tcp-info -prom=7070

Fast tcp-info collector in Go

This repository uses the netlink API to collect inet_diag messages, partially parses them, and caches the intermediate representation. It then detects differences from one scan to the next, and queues connections that have changed for logging. It logs the intermediate representation through external zstd processes to one file per connection.

The previous version uses protobufs, but we have discontinued that largely because of the increased maintenance overhead, and risk of losing unparsed data. Instead, we are now using ArchivedRecord which is partially parsed netlink messages, mostly in base64 encoded blobs, marshaled to JSONL format, with one JSON object per line.

To run the tests or the collection tool, you will also require zstd, which can be installed with:

bash <(curl -fsSL https://raw.githubusercontent.com/horta/zstd.install/master/install)

OR

sudo apt-get update && sudo apt-get install -y zstd

Parse library and command line tools

CSV tool

The cmd/csvtool directory contains a tool for parsing ArchivedRecord and producing CSV files. Currently reads netlink-jSONL from stdin and writes CSV to stdout.

Code Layout

  • inetdiag - code related to include/uapi/linux/inet_diag.h. All structs will be in structs.go
  • tcp - Should include ONLY the code related to include/uapi/linux/tcp.h
  • parse - code related to parsing the messages in inetdiag and tcp.
  • zstd - zstd reader and writer.
  • saver - code related to writing ParsedMessages to files.
  • cache - code to cache netlink messages and detect changes.
  • collector - code related to collecting netlink messages from the kernel.

Dependencies (as of March 2019)

  • saver: inetdiag, cache, parse, tcp, zstd
  • collector: parse, saver, inetdiag, tcp
  • main.go: collector, saver, parse (just for sanity check)
  • cache: parse
  • parse: inetdiag

And (almost) all package use metrics.

Layers for main.go (each layer depends only on items to right, or lower layers)

  1. main.go
  2. collector > saver > cache
  3. netlink > inetdiag
  4. tcp, zstd, metrics

Layers for parse package:

  1. parse (used by command line tools, etl)
  2. netlink > inetdiag
  3. tcp, zstd, metrics
You can’t perform that action at this time.