graphite protocol extension
Standard graphite protocol (
string.based.metric[...] value unix_timestamp
"string", "based", "metric" being the nodes of the metric, and metric names usually being unorganized, unstandardized and lacking information.
Carbon-tagger implements metrics 2.0 which aims to make metrics entirely self describing, structured, and standardized.
It does this by using an axtended graphite protocol (backwards compatible):
- nodes can be old-style values, or new-style
key_is_valtag pairs. (Some versions of graphite-web/graphite-api struggle with equals signs, so the latter format is recommended)
- if there's a "=" or "is" in one or more of the nodes, we'll try to parse as
proto2and add it to the index if below conditions are met.
- there must be a tag pair with
unitas tag key.
- there must be at least one other tag.
- you can freely choose the order of the nodes for every metric, but when you change the order, you change the metric key.
- old-style nodes (i.e. not "key=val" or
key_is_valformat) within a proto2 metric implicitly get an "nX" tag key where X is the node position in the string, starting from 1.
So this defines a metric as a set of tags, and while we're at it, it also specifies the metric_id that will be used by the current carbon and related tools. carbon-tagger will maintain a database of metrics and their tags, and pass it on (unaltered) to a daemon like carbon-cache or carbon-relay. So the protocol is fully backwards compatible.
This is metrics 2.0 but
- using dots as delimiters until we can fix graphite.
- you can use units like "Mbps" or "Errps" to mean "Mb/s" and "Err/s". Graphite treats slashes as delimiters. Carbon-tagger will set the proper unit tag.
- Indexes metrics 2.0 full (_id and tag)
- legacy metrics, just the _id, so you can search for it. (empty tags property) it's up to a tool like graph-explorer to create or update documents for legacy metrics with tags enabled.
how does this affect the rest of my stack?
- carbon-relay, carbon-cache: unaffected, they receive the same data as usual, the identifiers just look a little different.
- graphite-web: unaffected. metrics 2.0 still show up in the tree of the graphite composer, although that's an inferior UI model that I want to phase out. ideally, dashboards leverage the tag database, like graph-explorer does.
- aggregators like statsd need
proto2support. statsdaemon is a drop-in statsd replacement that for metrics in metrics2.0 format correctly represents its aggregations and statistical summaries by updating the key/value pairs of metric names, as opposed to the traditional prefix/suffix "features" which leave metrics even vaguer than they were.
why do you process the same metrics every time they are submitted?
to have a realtime database. you could get all metricnames later and process them offline, which lowers resource usage but has higher delays
are in proto2 format and are submitted to a carbon endpoint (typically your relay) they are also available on the http address at /debug/vars2
currently, not very optimized at all! but it's probably speedy enough, and there's a big buffer that smoothens the effect of new metrics
I reach about 15k metrics/s processing speed, even when the temp buffer is full and it's syncing to ES. in fact, i don't see a discernable difference between buffer full (unblocked) and buffer full (blocked) probably carbon-cache (whisper) was being the bottleneck?
space used: 176B/metric (21M for 125k metrics, twice that if we'd enable indexing/analyzing)
- make sure the bulk thing uses 'create' and doesn't update/replace the doc every time
- it seems like ES doesn't contain all metrics (on 2M unique inserts, ES' count is 1889300)
- better mapping, _source, type analyzing?
- don't store legacy metrics with empty tags, if already exists. now it will temporarily break graph-explorer's structured metrics (until the latter indexes again)
- populate an ES cache from disk on program start
- forward_lines channel buffering, so that runtime doesn't have to switch Gs all the time?
if/when ES performance becomes an issue, consider:
- edge-ngrams/reverse edge-ngrams to do it at index time
- use prefix match with regular/reverse fields
- query_string maybe
if you already have a working Go setup, adjust accordingly:
mkdir -p ~/go/ export GOPATH=~/go go get github.com/Vimeo/carbon-tagger go get github.com/mjibson/party go build github.com/Vimeo/carbon-tagger
- just copy the carbon-tagger binary and run it (TODO: initscripts)
- install elasticsearch and run it (super easy, see http://www.elasticsearch.org/guide/reference/setup/installation/, just set a unique cluster name)