Cassandra-backed, metrics2.0 based, multi-tenant timeseries database for Graphite and friends
Dieterbe Merge pull request #1050 from grafana/no-need-to-go-list-vendor
no need for $(go list ./... | grep -v /vendor/) ...
Latest commit 4b0b8c1 Sep 19, 2018
Permalink
Failed to load latest commit information.
.circleci build using golang 1.11 Sep 13, 2018
api Merge pull request #1043 from grafana/named-status-codes-everywhere Sep 12, 2018
batch just use dep instead of gopkg.in to pin raintank/schema Aug 20, 2018
cassandra rebrand: raintank -> GrafanaLabs Sep 29, 2017
clock copy clock code so we don't have to pull in carbon-relay-ng Dec 1, 2017
cluster Merge pull request #1000 from bloomberg/fixSingleMode Aug 21, 2018
cmd also convert "omit read timeout" to duration Sep 13, 2018
conf just use dep instead of gopkg.in to pin raintank/schema Aug 20, 2018
consolidation just use dep instead of gopkg.in to pin raintank/schema Aug 20, 2018
dashboards fix a few dashboards, set proper datasource Sep 10, 2018
devdocs intro to dev instructions should also go to docs, not devdocs Sep 14, 2018
docker sync configs Sep 14, 2018
docs Merge pull request #1017 from grafana/cassandra_timeout_format Sep 14, 2018
errors cleaner way to work with errors in non-http related code Jan 22, 2018
expr move expr/NOTES to devdocs Sep 7, 2018
idx update metric descriptions and docs Sep 11, 2018
input Merge pull request #1043 from grafana/named-status-codes-everywhere Sep 12, 2018
kafka typo's Mar 6, 2017
mdata update metric descriptions and docs Sep 11, 2018
scripts no need for $(go list ./... | grep -v /vendor/) ... Sep 17, 2018
stacktest show version & fix support for older docker-compose Sep 10, 2018
stats PR feedback: move process memory stats to process reporter Aug 30, 2018
store Merge pull request #1045 from grafana/go1.11 Sep 14, 2018
test just use dep instead of gopkg.in to pin raintank/schema Aug 20, 2018
tracing better way to report errors Aug 24, 2017
util simpler May 2, 2018
vendor fix too-eager 'build' ignore rule Sep 6, 2018
.gitignore sort Sep 10, 2018
Gopkg.lock add prometheus metrics instrumentation endpoint and active series met… Sep 4, 2018
Gopkg.toml pin procfs to the version that happened to be already vendored Aug 30, 2018
LICENSE switch to a license that may make more sense Aug 8, 2016
Makefile no need for $(go list ./... | grep -v /vendor/) ... Sep 17, 2018
NOTICE update copyright years Jan 2, 2018
README.md docker docs linking Sep 14, 2018
metrictank-sample.ini sync configs Sep 14, 2018

README.md

Metrictank

Circle CI Go Report Card GoDoc

Introduction

Metrictank is a multi-tenant timeseries engine for Graphite and friends. It provides long term storage, high availability, efficient storage, retrieval and processing for large scale environments.

GrafanaLabs has been running metrictank in production since December 2015. It currently requires an external datastore like Cassandra, and we highly recommend using Kafka to support clustering, as well as a clustering manager like Kubernetes. This makes it non-trivial to operate, though GrafanaLabs has an on-premise product that makes this process much easier.

Features

  • 100% open source
  • Inspired by the Facebook gorilla paper. Most notably, the heavily compressed chunks dramatically lower cpu, memory and storage requirements.
  • Writeback RAM cache, serving most data out of memory.
  • Graphite is a first class citizen. As of graphite-1.0.1, metrictank can be used as a graphite CLUSTER_SERVER.
  • Can also act as a Graphite server itself, though the functions processing library is only partially implemented, metrictank proxies requests to Graphite if it can't handle the required processing (for those requests it will degrade to just being the backend storage)
  • Can also be used with Prometheus (but the experience won't be as good as something built just for prometheus, which we're also working on)
  • Accurate, flexible rollups by storing min/max/sum/count (which also gives us average). So we can do consolidation (combined runtime+archived) accurately and correctly, unlike most other graphite backends like whisper
  • Flexible tenancy: can be used as single tenant or multi tenant. Selected data can be shared across all tenants.
  • Input options: carbon, metrics2.0, kafka, Prometheus (soon: json or msgpack over http)
  • Guards against excessive data requests

Limitations

  • No performance/availability isolation between tenants per instance. (only data isolation)
  • Minimum computation locality: we move the data from storage to processing code, which is both metrictank and graphite.
  • Backlog replaying and queries can be made faster. A Go GC issue may occasionally inflate response times.
  • We use metrics2.0 in native input protocol and indexes, but barely do anything with it yet.
  • can't overwrite old data. We support reordering the most recent time window but that's it. (unless you restart MT)

Interesting design characteristics (feature or limitation... up to you)

  • Upgrades / process restarts requires running multiple instances (potentially only for the duration of the maintenance) and possibly re-assigning the primary role. Otherwise data loss of current chunks will be incurred. See operations guide
  • clustering works best with an orchestrator like kubernetes. MT itself does not automate master promotions. See clustering for more.
  • Only float64 values. Ints and bools currently stored as floats (works quite well due to the gorilla compression), No text support.
  • Only uint32 unix timestamps in second resolution. For higher resolution, consider streaming directly to grafana
  • No data locality: doesn't seem needed yet to put related series together.

Docs

installation, configuration and operation.

features in-depth

Other

Releases and versioning

  • releases and changelog

  • we aim to keep master stable and vet code before merging to master

  • We're pre-1.0 but adopt semver for our 0.MAJOR.MINOR format. The rules are simple:

    • MAJOR version for incompatible API or functionality changes
    • MINOR version when you add functionality in a backwards-compatible manner, and

    We don't do patch level releases since minor releases are frequent enough.

License

Copyright 2016-2018 Dieter Plaetinck, Anthony Woods, Jeremy Bingham, Damian Gryski, raintank inc

This software is distributed under the terms of the GNU Affero General Public License.