Skip to content
Prometheus push federation
Go Jsonnet Shell Makefile Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cmd pkg/authorize: abstract out authorization Aug 27, 2019
dockerfiles Adds build-deploy.sh script for CI integration Oct 29, 2018
docs Add metric gathering for node_role_os_version_machine:cpu_capacity_co… Sep 13, 2019
jsonnet Add metric gathering for node_role_os_version_machine:cpu_capacity_co… Sep 13, 2019
manifests Add metric gathering for node_role_os_version_machine:cpu_capacity_co… Sep 13, 2019
pkg Use requestBodyLimit constant to limit request body to 32MiB Aug 27, 2019
test deploy: remove deploy dir Aug 21, 2019
vendor vendor dependencies Jul 31, 2019
.gitignore
Dockerfile *: enable usage of go 1.12 and go modules Jun 4, 2019
LICENSE Create LICENSE Oct 16, 2018
Makefile Makefile: only benchmark one value Jul 22, 2019
OWNERS Remove riuvshin from owners Sep 16, 2019
README.md Update deployment configuration with latest settings for local auth Jul 16, 2018
build_deploy.sh Fix typos and bugs Oct 30, 2018
go.mod Update dependencies Jun 26, 2019
go.sum Update dependencies Jun 26, 2019
metrics.json Add comments to each of the metrics in telemeter to explain there use Dec 7, 2018

README.md

Telemeter

Telemeter implements a Prometheus federation push client and server to allow isolated Prometheus instances that cannot be scraped from a central Prometheus to instead perform push federation to a central location.

  1. The local client scrapes /federate on a given Prometheus instance.
  2. The local client performs cleanup and anonymization and then pushes the metrics to the server.
  3. The server authenticates the client, validates and verifies that the metrics are "safe", and then ensures they have a label uniquely identifying the source client.
  4. The server holds the metrics in a local disk store until scraped.
  5. A centralized Prometheus scrapes each server instance and aggregates all the metrics.

Since that push is across security boundaries, the server must perform authentication, authorization, and data integrity checks as well as being resilient to denial of service.

Each client is uniquely identified by a cluster ID and all metrics federated are labelled with that ID.

Since Telemeter is dependent on Prometheus federation, each server instance must ensure that all metrics for a given cluster ID are routed to the same instance, otherwise Prometheus will mark those metrics series as stale. To do this, the server instances form a cluster using a secure gossip transport and build a consistent hash ring so that pushed client metrics are routed internally to the same server.

For resiliency, each server instance stores the received metrics on disk hashed by cluster ID until they are accessed by a federation endpoint.

note: Telemeter is alpha and may change significantly

Get started

To see this in action, run

make
./test/integration.sh http://localhost:9005

The command launches a two instance telemeter-server cluster and a single telemeter-client to talk to that server, along with a Prometheus instance running on http://localhost:9005 that shows the federated metrics. The client will scrape metrics from the local prometheus, then send those to the telemeter server cluster, which will then be scraped by that instance.

To run this test against another Prometheus server, change the URL (and if necessary, specify the bearer token necessary to talk to that server as the second argument).

To build binaries, run

make

To execute the unit test suite, run

make check

To launch a self contained integration test, run:

make test-integration
You can’t perform that action at this time.