Cassandra-backed, metrics2.0 based, multi-tenant timeseries database for Graphite and friends
Go Shell Other
Latest commit ba401cb Jan 18, 2017 @Dieterbe Dieterbe committed on GitHub Merge pull request #482 from raintank/472-cleanup
472 cleanup
Permalink
Failed to load latest commit information.
api fix primary parameter name Jan 18, 2017
batch package docs! Jul 23, 2016
cassandra update metrics descriptions Jan 3, 2017
cluster rename peers to members Jan 17, 2017
cmd partition typo Jan 18, 2017
consolidation refactor HTTP into its own package Nov 15, 2016
contrib added es_list function, made graphite wildcards match exactly Nov 4, 2016
docker update docker-cluster config Jan 18, 2017
docs partition typo Jan 18, 2017
idx Merge pull request #459 from raintank/peerDiscovery Jan 17, 2017
input Make kafka-cluster chunk save messages partition aware. Jan 17, 2017
kafka Make kafka-cluster chunk save messages partition aware. Jan 17, 2017
mdata consistent messages Jan 18, 2017
scripts consistent messages Jan 18, 2017
stats do chunk cache stats the new way Jan 4, 2017
usage implement most of dieters comments and integrate cache results in dat… Jan 4, 2017
util refactor HTTP into its own package Nov 15, 2016
vendor Merge pull request #459 from raintank/peerDiscovery Jan 17, 2017
.gitignore fix gitignore bug, add MT docker stuff Aug 4, 2016
LICENSE switch to a license that may make more sense Aug 8, 2016
Makefile add make target for builds with -race Jan 4, 2017
NOTICE switch to a license that may make more sense Aug 8, 2016
README.md update documentation/configs for clustering + default primary true Dec 12, 2016
circle.yml build tools on circle Jan 3, 2017
dashboard.json metrics updates Jan 16, 2017
metrictank-sample.ini consistent messages Jan 18, 2017
metrictank.go Make kafka-cluster chunk save messages partition aware. Jan 17, 2017

README.md

Circle CI Go Report Card GoDoc

introduction

Metrictank is a cassandra-backed, metrics2.0 based, multi-tenant timeseries database for Graphite and friends

metrictank is a timeseries database, inspired by the Facebook gorilla paper. Most notably, it employs the compression mechanism described in the paper to dramatically lower storage overhead, as well as data chunking to lower the load on cassandra. Graphite users are first-class citizens.

status

While raintank has been running metrictank in production since december 2015, there are still plenty of kinks to work out and bugs to fix. It should be considered an alpha project.

limitations

  • no performance/availability isolation between tenants per instance. (only data isolation)
  • clustering is basic: statically defined peers, master promotions are manual, etc. See clustering for more.
  • no computation locality: we move the data from storage to processing code, which is both metrictank and graphite-api.
  • the datastructures can use performance engineering. A Go GC issue may occassionally inflate response times.
  • the native input protocol is inefficient. Should not send all metadata with each point.
  • we use metrics2.0 in native input protocol and indexes, but barely do anything with it yet.
  • for any series you can't write points that are earlier than previously written points. (unless you restart MT)

interesting design characteristics (feature or limitation.. up to you)

  • upgrades / process restarts requires running multiple instances (potentially only for the duration of the maintenance) and re-assigning the primary role. Otherwise data loss of current chunks will be incurred. See operations guide
  • only float64 values. Ints and bools currently stored as floats (works quite well due to the gorilla compression), No text support.
  • only uint32 unix timestamps in second resolution. For higher resolution, consider streaming directly to grafana
  • no data locality: doesn't seem needed yet to put related series together.

main features

  • 100% open source
  • graphite is a first class citizen (note: currently requires a fork of graphite-api and the graphite-metrictank plugin)
  • accurate, flexible rollups by storing min/max/sum/count (which also gives us average). So we can do consolidation (combined runtime+archived) accurately and correctly, unlike most other graphite backends like whisper
  • metrictank acts as a writeback RAM cache for recent data.
  • flexible tenancy: can be used as single tenant or multi tenant. Selected data can be shared across all tenants.
  • input options: carbon, metrics2.0, kafka. (soon: json or msgpack over http)
  • guards against excessive data requests
  • efficient data compression and efficient use of Cassandra.

Docs

installation, configuration and operation.

features in-depth

Other

Releases and versioning

  • releases and changelog
  • we aim to keep master stable and vet code before merging to master
  • We're pre-1.0 but adopt semver for our 0.MAJOR.MINOR format. The rules are simple:

    • MAJOR version for incompatible API or functionality changes
    • MINOR version when you add functionality in a backwards-compatible manner, and

    We don't do patch level releases since minor releases are frequent enough.

License

Copyright 2016 Dieter Plaetinck, Anthony Woods, Jeremy Bingham, Damian Gryski, raintank inc

This software is distributed under the terms of the GNU Affero General Public License.