Recording counters

Jon Watte edited this page Sep 24, 2012 · 1 revision

Home - Recording counters

istatd can record counters reflecting the performance of the host it is running on. However, that is not its only function, or even its main function.

TCP data stream format

istatd listens for incoming connections on TCP port 8111 (default -- change with --stat-port.) On those connections, individual lines (separated with CRLF -- \r\n) of statistics are accepted. A line has one of three formats:

  • value
  • timestamp value
  • timestamp sum sumSquared min max count

The value, sum, sumSquared, min and max are floating-point values. timestamp and count are integer values, where timestamp is in UNIX epoch UTC time. The "value" of a bucket of data for the last format is sum divided by count. The reason you may want to break these apart is that, with sum, sum squared, and count, you can calculate the standard deviation of all data points that contribute to a particular counter for a particular time.

Counter names

Names are formulated roughly as C-style identifiers separated by periods. Each period defines a new "branch" in a tree of counters. Some additional characters are allowed in counter names, that are not allowed in C identifiers, such as dashes.

Names can also contain carets. Carets are the same as periods, except the next caret in a name will "rewind" the name to the position of the previous caret, and substitute the next substring. This will generate many counters from a single sample. For example, if the counter name is cpu.load^host.jupiter^role.database^everything then the data on that line will be merged into three counters:, cpu.load.role.database, and cpu.load.everything.

Types of counters

See the wiki entry for Counter Types: events or gauges. Hint: you have to get this right the first time!

UDP data format

Additionally, istatd accepts the same kind of data in UDP packets that arrive on the same port as TCP. Each UDP packet can contain multiple lines/counter values, but must end with a proper linefeed.

Security and topology

There is no security against forged packets, other than whatever firewalls you have to protect your machines in general. To aid in this configuration, you can tell istatd to only bind to a certain IP address (such as a 10.x internal address, or the localhost address) using --listen-address.

At IMVU, we run one istatd process on each host, in "agent" mode, where it doesn't store data on disk, but instead just forwards data to a master database. Any process on a host can then fling any counter value to localhost:8111 in a UDP packet (or TCP stream) and be reasonably certain that the counter event will be taken care of by the system.