Skip to content
metrics2.0 based, multi-tenant timeseries store for Graphite and friends.
Go Shell Other
Branch: master
Clone or download
Dieterbe Merge pull request #1470 from D3luxee/patch-1
Don't validate MaxChunkSpan if BigTable store is disabled
Latest commit 3006629 Sep 19, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Revert "add initial idx interning logic" Aug 7, 2019
api better fix: clean up old code Sep 18, 2019
batch move schema into metrictank Aug 15, 2019
benchmarks Update benchmarks/simulate-lock-congestion/README.md Sep 13, 2019
cassandra rebrand: raintank -> GrafanaLabs Sep 29, 2017
clock copy clock code so we don't have to pull in carbon-relay-ng Dec 1, 2017
cluster support bySeriesWithTagsFnv, move str->method logic to schema Aug 27, 2019
cmd-dev/mt-simulate-memory-idx-lock-contention remove duplicate code Sep 13, 2019
cmd move tool to cmd-dev + better name Sep 5, 2019
conf move schema into metrictank Aug 15, 2019
consolidation Consolidate extra comment Sep 11, 2019
dashboards separate cluster events from status to clarify viz Jul 17, 2019
devdocs better documentation Nov 27, 2018
docker make enrichment cache size configurable Aug 23, 2019
docs solve series aggregation and merging mismatches Sep 11, 2019
errors cleaner way to work with errors in non-http related code Jan 22, 2018
expr Merge pull request #1452 from grafana/fix-uneven-length-panics Sep 11, 2019
idx Merge pull request #1450 from grafana/simulate-lock-congestion Sep 13, 2019
input move schema into metrictank Aug 15, 2019
jaeger implement tags according to spec Jun 7, 2019
kafka remove kafka offset tracker Oct 24, 2018
logger change log-level config option to use string Oct 3, 2018
mdata Added unit test for getSeriesFixed with archInterval different from o… Sep 11, 2019
schema make schema apache2 Aug 28, 2019
scripts make enrichment cache size configurable Aug 23, 2019
stacktest move schema into metrictank Aug 15, 2019
stats fix race condition with backoff mode out of sync Apr 2, 2019
store Don't validate MaxChunkSpan if BigTable store is disabled Sep 19, 2019
test move schema into metrictank Aug 15, 2019
tracing better way to report errors Aug 24, 2017
util use fnv hash that supports string writing Aug 23, 2019
vendor patch and update macaron/binding middleware Sep 19, 2019
.codelingoignore Add .codelingoignore file at project root. Mar 13, 2019
.gitignore move tool to cmd-dev + better name Sep 5, 2019
CHANGELOG.md prep for v0.13.0 Sep 17, 2019
Gopkg.lock patch has been merged upstream Sep 19, 2019
Gopkg.toml patch has been merged upstream Sep 19, 2019
LICENSE switch to a license that may make more sense Aug 8, 2016
Makefile add test target for the stack tests Aug 13, 2019
NOTICE update copyright years Jan 2, 2018
README.md update license info in readme Aug 28, 2019
codelingo.yaml Add codelingo.yaml file at project root. Mar 13, 2019
metrictank-sample.ini make enrichment cache size configurable Aug 23, 2019

README.md

Metrictank

Circle CI Go Report Card GoDoc

Introduction

Metrictank is a multi-tenant timeseries engine for Graphite and friends. It provides long term storage, high availability, efficient storage, retrieval and processing for large scale environments.

GrafanaLabs has been running metrictank in production since December 2015. It currently requires an external datastore like Cassandra or Bigtable, and we highly recommend using Kafka to support clustering, as well as a clustering manager like Kubernetes. This makes it non-trivial to operate, though GrafanaLabs has an on-premise product that makes this process much easier.

Features

  • 100% open source
  • Inspired by the Facebook gorilla paper. Most notably, the heavily compressed chunks dramatically lower cpu, memory and storage requirements.
  • Writeback RAM cache, serving most data out of memory.
  • Graphite is a first class citizen. As of graphite-1.0.1, metrictank can be used as a graphite CLUSTER_SERVER.
  • Can also act as a Graphite server itself, though the functions processing library is only partially implemented, metrictank proxies requests to Graphite if it can't handle the required processing (for those requests it will degrade to just being the backend storage)
  • Can also be used with Prometheus (but the experience won't be as good as something built just for prometheus, which we're also working on)
  • Accurate, flexible rollups by storing min/max/sum/count (which also gives us average). So we can do consolidation (combined runtime+archived) accurately and correctly, unlike most other graphite backends like whisper
  • Flexible tenancy: can be used as single tenant or multi tenant. Selected data can be shared across all tenants.
  • Input options: carbon, metrics2.0, kafka, Prometheus (soon: json or msgpack over http)
  • Guards against excessive data requests

Limitations

  • No performance/availability isolation between tenants per instance. (only data isolation)
  • Minimum computation locality: we move the data from storage to processing code, which is both metrictank and graphite.
  • Backlog replaying and queries can be made faster. A Go GC issue may occasionally inflate response times.
  • We use metrics2.0 in native input protocol and indexes, but barely do anything with it yet.
  • can't overwrite old data. We support reordering the most recent time window but that's it. (unless you restart MT)

Interesting design characteristics (feature or limitation... up to you)

  • Upgrades / process restarts requires running multiple instances (potentially only for the duration of the maintenance) and possibly re-assigning the primary role. Otherwise data loss of current chunks will be incurred. See operations guide
  • clustering works best with an orchestrator like kubernetes. MT itself does not automate master promotions. See clustering for more.
  • Only float64 values. Ints and bools currently stored as floats (works quite well due to the gorilla compression), No text support.
  • Only uint32 unix timestamps in second resolution. For higher resolution, consider streaming directly to grafana
  • No data locality: doesn't seem needed yet to put related series together.

Docs

installation, configuration and operation.

features in-depth

Other

Releases and versioning

  • releases and changelog

  • we aim to keep master stable and vet code before merging to master

  • We're pre-1.0 but adopt semver for our 0.MAJOR.MINOR format. The rules are simple:

    • MAJOR version for incompatible API or functionality changes
    • MINOR version when you add functionality in a backwards-compatible manner, and

    We don't do patch level releases since minor releases are frequent enough.

License

Copyright 2016-2019 Grafana Labs

This software is distributed under the terms of the GNU Affero General Public License.

Some specific packages have a different license:

You can’t perform that action at this time.