A data vault for metrics

“I may not agree with your methodology, but I'll store to capacity the data you apply it to.”

  —Vaultaire, 1770 (apocryphal)

Basis for analytics

Most systems that store server metrics are lossy: they average data out as they compact them in order to save space. If you're planning on doing serious analytics or predictive modelling, however, you can't be throwing away data points. We wanted a place to put metrics that wasn't lossy, allows arbitrary range querying, and would scale out without bothering anybody.

Vaultaire is a data vault for system metrics, backed onto Ceph. We use it internally to store systems metrics from Nagios, OpenStack and pmacct for problem diagnosis, abnormality detection, metering/billing, forecasting and capacity planning. We're looking forward to letting clients write to it as well.

Design

Vaultaire is a fault-tolerant, distributed, scalable system. A few key architectural decisions enable this:

• Data points are immutable. Once you've written a metric you don't go changing it. If a business decision is made to value that data differently, then that's a job for analytics later.

• Writes operations are idempotent. Under the hood we append every time we write, and do de-duplication when we read. If you aren't trying to to update or sort in time order when saving points, the on disk data structures become very simple indeed. And with idempotent operations, it means that should a client not receive an acknowledgment of write it can re-send that point.

• No state is held by daemons. Vaultaire has quite a number of moving parts, but all of them can be multiply provisioned and none of them carry any state, so a single failure does not take down the system. Scaling under load can be done by adding more daemons horizontally. The only place state is held is in the Ceph cluster (which, by definition, is consistent and redundant).

Deployment

System dependencies

• librados
• zeromq 4+
• GHC 7.8.3+
• cabal

Common packages

• vaultaire-common
• marquise

Build

cabal sandbox init
cabal sandbox add-source ../vaultaire-common
cabal sandbox add-source ../marquise
cabal install

You'll need to add the sandbox's bin directory to your $PATH, or else replace the vault command with a fully-qualified path.

Running

You need to run four components: the broker, plus one or more of each of the writer, reader and contents daemons. These can be on the same or different hosts, but they must all point to the same broker (if you want to make the broker highly-available, use a TCP load-balancer).

The defaults assume your Rados pool is called vaultaire, and you're connecting with the username vaultaire; if this isn't the case, override with -p and -u respectively.

Additionally, if your Ceph keyring is not at the default path used by librados (/etc/ceph/keyring as of Feburary 2015) you'll need to override it with -k /path/to/your/keyring/file.

vault -d broker # Start broker in debug mode
vault -d writer -b localhost # Start writer in debug mode
vault -d reader -b localhost # Start reader in debug mode
vault -d contents -b localhost # Start contents daemon in debug mode

Ecosystem

Vaultaire on its own only provides a binary interface over ZeroMQ. To fill this gap, there's a standard ecosystem of tools to provide various interfaces:

• Chevalier - datasource search engine backed to Elasticsearch.
• Sieste - RESTful web interface to datapoints and metadata (talks to Vaultaire and Chevalier).
• Marquise - client library for reading and writing to Vaultaire, and the marquised daemon for transmitting queued datapoints.
• libmarquise - C client library for writing datapoints, for situations where using the Haskell package is unsuitable.
• vaultaire-query - analytics-focused query DSL.
• Machiavelli - graphing engine with support for using Sieste as a backend.

Implementation

This is the second major release of Vaultaire. "v2" writes directly to our production Ceph cluster (via librados). The server daemons (vaultaire) are written in Haskell, with client libraries available in Haskell (marquise) and C (libmarquise). There's a search index (chevalier) backed by ElasticSearch, and a visualization pipeline where RESTful access to interpolated data points (sieste) is provided by Haskell to a beautiful Ruby/JavaScript graphing front-end (machiavelli) that allows you to correlate and view data streams. The vaultaire-query library provides a Haskell-based DSL for analytics applications.