Stenographer is a packet capture solution which aims to quickly spool all packets to disk, then provide simple, fast access to subsets of those packets. Discussion/announcements at stenographer@googlegroups.com
Go C++ Shell Yacc Makefile
Latest commit 66a8e7e Dec 6, 2016 Reid Price committed with gconnell blockfile: Correct order of strings in debug log
The formatting arguments were reversed, this corrects messages like:
  Blockfile "host 1.2.3.4" looking up query "/tmp/stenographer0/PKT0/0"
to be:
  Blockfile "/tmp/stenographer0/PKT0/0" looking up query "host 1.2.3.4"
Permalink
Failed to load latest commit information.
base Fix base_test after gopacket change. May 2, 2016
blockfile blockfile: Correct order of strings in debug log Dec 6, 2016
certs Modify cert usage; allow users to manage certs. Mar 4, 2016
config Store open files in an LRU cache. May 3, 2016
configs Merge pull request #115 from hosom/master Aug 10, 2015
env Store open files in an LRU cache. May 3, 2016
filecache Store open files in an LRU cache. May 3, 2016
httputil Context work, byte/packet limits. Feb 10, 2016
indexfile Fix tests to use filecache. May 4, 2016
integration_test Store open files in an LRU cache. May 3, 2016
query query: Swap the time adjustment for time query lookup. Nov 17, 2016
stats Fix calls to NanoTimer to call twice, due to defer. Feb 10, 2015
stenotype Wrap mmap in ifdef check. Oct 24, 2016
testdata Move test data, add blockfile tests. Feb 14, 2015
thread Fix tests to use filecache. May 4, 2016
.travis.yml Specify use of Go 1.6. Mar 4, 2016
CONTRIBUTING.md Make CONTRIBUTING a proper markdown file. Feb 18, 2015
DESIGN.md Modify cert usage; allow users to manage certs. Mar 4, 2016
INSTALL.md INSTALL.md: Mark "--preallocate_file_mb" as important when using Jan 6, 2016
LICENSE Initial commit of stenographer code. Oct 13, 2014
README.md Fix UTC timestamp example Jun 8, 2015
format.sh Move util functions into util.cc. Feb 15, 2015
install.sh Get dir name from config, name certs by CN. Mar 4, 2016
install_el7.sh Sternographer needs at least Go 1.6 Jul 21, 2016
lib.sh Add an integration test. Jan 27, 2015
rpmbuild-steno-centos Fix #145 RPM build on centos. Oct 25, 2016
stenocurl Modify cert usage; allow users to manage certs. Mar 4, 2016
stenographer.go Better logging of config with %+v Mar 8, 2015
stenographer.spec Added new file, correct spec file May 20, 2016
stenokeys.sh Move stuff around stenokeys a bit. Mar 11, 2016
stenoread Fix limit headers and instrument stenoread. Feb 10, 2016

README.md

Stenographer

Overview

Stenographer is a full-packet-capture utility for buffering packets to disk for intrusion detection and incident response purposes. It provides a high-performance implementation of NIC-to-disk packet writing, handles deleting those files as disk fills up, and provides methods for reading back specific sets of packets quickly and easily.

It is designed to:

  • Write packets to disk, very quickly (~10Gbps on multi-core, multi-disk machines)
  • Store as much history as it can (managing disk usage, storing longer durations when traffic slows, then deleting the oldest packets when it hits disk limits)
  • Read a very small percentage (<1%) of packets from disk based on analyst needs

It is NOT designed for:

  • Complex packet processing (TCP stream reassembly, etc)
    • It’s fast because it doesn’t do this.  Even with the very minimal, single-pass processing of packets we do, processing ~1Gbps for indexing alone can take >75% of a single core.
    • Processing the data by reading it back from disk also doesn’t work:  see next bullet point.
  • Reading back large amounts of packets (> 1% of packets written)
    • The key concept here is that disk reads compete with disk writes… you can write at 90% of disk speed, but that only gives you 10% of your disk’s time for reading.  Also, we’re writing highly sequential data, which disks are very good at doing quickly, and generally reading back sparse data with lots of seeks, which disks do slowly.

For further reading, check out DESIGN.md for a discussion of stenographer's design, or read INSTALL.md for how to install stenographer on a machine.

Querying

Query Language

A user requests packets from stenographer by specifying them with a very simple query language. This language is a simple subset of BPF, and includes the primitives:

host 8.8.8.8          # Single IP address (hostnames not allowed)
net 1.0.0.0/8         # Network with CIDR
net 1.0.0.0 mask 255.255.255.0  # Network with mask
port 80               # Port number (UDP or TCP)
ip proto 6            # IP protocol number 6
icmp                  # equivalent to 'ip proto 1'
tcp                   # equivalent to 'ip proto 6'
udp                   # equivalent to 'ip proto 17'

# Stenographer-specific time additions:
before 2012-11-03T11:05:00Z      # Packets before a specific time (UTC)
after 2012-11-03T11:05:00-0700   # Packets after a specific time (with TZ)
before 45m ago        # Packets before a relative time
before 3h ago         # Packets after a relative time

Primitives can be combined with and/&& and with or/||, which have equal precendence and evaluate left-to-right. Parens can also be used to group.

(udp and port 514) or (tcp and port 8080)

Stenoread CLI

The stenoread command line script automates pulling packets from Stenographer and presenting them in a usable format to analysts. It requests raw packets from stenographer, then runs them through tcpdump to provide a more full-featured formatting/filtering experience. The first argument to stenoread is a stenographer query (see 'Query Language' above). All other arguments are passed to tcpdump. For example:

# Request all packets from IP 1.2.3.4 port 6543, then do extra filtering by
# TCP flag, which typical stenographer does not support.
$ stenoread 'host 1.2.3.4 and port 6543' 'tcp[tcpflags] & tcp-push != 0'

# Request packets on port 8765, disabling IP resolution (-n) and showing
# link-level headers (-e) when printing them out.
$ stenoread 'port 8765' -n -e

# Request packets for any IPs in the range 1.1.1.0-1.1.1.255, writing them
# out to a local PCAP file so they can be opened in Wireshark.
$ stenoread 'net 1.1.1.0/24' -w /tmp/output_for_wireshark.pcap

Downloading

To download the source code, install Go locally, then run:

$ go get github.com/google/stenographer

Go will handle downloading and installing all Go libraries that stenographer depends on. To build stenotype, go into the stenotype directory and run make. You may need to install the following Ubuntu packages (or their equivalents on other Linux distros):

  • libaio-dev
  • libleveldb-dev
  • libsnappy-dev
  • g++
  • libcap2-bin
  • libseccomp-dev

Obligatory Fine Print

This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.

This code is not intended (or used) to watch Google's users. Its purpose is to increase security on our networks by augmenting our internal monitoring capabilities.