Service to replicate InfluxDB data for high availability.
Clone or download
Pull request Compare This branch is 71 commits ahead of influxdata:master.
Marc BENSLAHDINE
Marc BENSLAHDINE Merge branch 'develop' into 'master'
Bugfixing, new configuration, health, hot flushing, rate limiting, filtering

See merge request noc/influxdb/influxdb-relay!38
Latest commit 98b8c44 Dec 27, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
dev Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
docs Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
examples Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
relay Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
relayservice Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
scripts Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
vendor Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
.gitignore Formatting issues Nov 15, 2018
.gitlab-ci.yml Fixed a data race & added more debug logs Dec 5, 2018
.mdlrc Update README Oct 12, 2018
CHANGELOG.md Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
CONTRIBUTING.md Add CI tests & fix errors Jun 30, 2018
Dockerfile Update README.md, Dockerfile files Oct 13, 2018
Gopkg.lock Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
Gopkg.toml Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
LICENSE Initial commit Mar 8, 2016
PACKAGING.md Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
README.md Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018
main.go Bugfixing, new configuration, health, hot flushing, rate limiting, fi… Dec 27, 2018

README.md

influxdb-relay

License

  1. Overview
  2. Description
  3. Requirements
  4. Setup
  5. Usage
  6. Limitations
  7. Development
  8. Miscellaneous

Overview

Maintained fork of influxdb-relay originally developed by InfluxData. Replicate InfluxDB data for high availability.

Description

This project adds a basic high availability layer to InfluxDB. With the right architecture and disaster recovery processes, this achieves a highly available setup.

Tested on

Other versions will probably work but are untested.

Setup

Go

Download the daemon into your $GOPATH and install it in /sur/sbin.

go get -u github.com/vente-privee/influxdb-relay
cp ${GOPATH}/bin/influxdb-relay /usr/bin/influxdb-relay
chmod 755 /usr/bin/influxdb-relay

Create the configuration file in /etc/influxdb-relay.

mkdir -p /etc/influxdb-relay
cp ${GOPATH}/src/github.com/vente-privee/influxdb-relay/examples/sample.conf \
   /etc/influxdb-relay/influxdb-relay.conf

Docker

Build your own image.

git clone git@github.com:vente-privee/influxdb-relay
cd influxdb-relay
docker build \
       --file Dockerfile \
       --rm \
       --tag influxdb-relay:latest \
       .
docker run \
       --volume /path/to/influxdb-relay.conf:/etc/influxdb-relay/influxdb-relay.conf
       --rm
       influxdb-relay:latest

or

Docker pull our image.

docker pull vptech/influxdb-relay:latest
docker run \
       --volume /path/to/influxdb-relay.conf:/etc/influxdb-relay/influxdb-relay.conf
       --rm
       vptech/influxdb-relay:latest

Usage

You can find more documentation in docs folder.

You can find some configurations in examples folder.

Configuration

[[http]]
# Name of the HTTP server, used for display purposes only.
name = "example-http"

# TCP address to bind to, for HTTP server.
bind-addr = "127.0.0.1:9096"

# Timeout for /health route
# After this time, the host may be considered down
health-timeout-ms = 10000

# Request limiting (Applied to all backend)
rate-limit = 5
burst-limit = 10

# Ping response code, default is 204
default-ping-response = 200

# Enable HTTPS requests.
ssl-combined-pem = "/path/to/influxdb-relay.pem"

# InfluxDB instances to use as backend for Relay
[[http.output]]
# name: name of the backend, used for display purposes only.
name="local-influxdb01"

# location: full URL of the /write endpoint of the backend
location="http://127.0.0.1:8086/"

# endpoints: Routes to use on Relay
# write: Route for standard InfluxDB request
# write_prom: Route for Prometheus request
# ping: Route for ping request
# query: Route fot querying InfluxDB backends
endpoints = {write="/write", write_prom="/api/v1/prom/write", ping="/ping", query="/query"}

# timeout: Go-parseable time duration. Fail writes if incomplete in this time.
timeout="10s"

# skip-tls-verification: skip verification for HTTPS location. WARNING: it's insecure. Don't use in production.
skip-tls-verification = false

# InfluxDB
[[http.output]]
name="local-influxdb02"
location="http://127.0.0.1:7086/"
endpoints = {write="/write", ping="/ping", query="/query"}
timeout="10s"

# Prometheus
[[http.output]]
name="local-influxdb03"
location="http://127.0.0.1:6086/"
endpoints = {write="/write", write_prom="/api/v1/prom/write", ping="/ping", query="/query"}
timeout="10s"

[[http.output]]
name="local-influxdb05"
location="http://127.0.0.1:5086/"
endpoints = {write="/write", write_prom="/api/v1/prom/write", ping="/ping", query="/query"}
timeout="10s"

[[udp]]
# Name of the UDP server, used for display purposes only.
name = "example-udp"

# UDP address to bind to.
bind-addr = "127.0.0.1:9096"

# Socket buffer size for incoming connections.
read-buffer = 0 # default

# Precision to use for timestamps
precision = "n" # Can be n, u, ms, s, m, h

# InfluxDB instance to use as backend for Relay.
[[udp.output]]
# name: name of the backend, used for display purposes only.
name="local1"

# location: host and port of backend.
location="127.0.0.1:8089"

# mtu: maximum output payload size
mtu=512

[[udp.output]]
name="local2"
location="127.0.0.1:7089"
mtu=1024

InfluxDB Relay is able to forward from a variety of input sources, including:

  • influxdb
  • prometheus

Administrative tasks

/admin endpoint

Whereas data manipulation relies on the /write endpoint, some other features such as database or user management are based on the /query endpoint. As InfluxDB Relay does not send back a response body to the client(s), we are not able to forward all of the features this endpoint provides. Still, we decided to expose it through the /admin route.

It is now possible to query the /admin endpoint. Its usage is the same as the standard /query Influx DB enpoint:

curl -X POST "http://127.0.0.1:9096/admin" --data-urlencode 'q=CREATE DATABASE some_database'

Errors will be logged just like regular /write queries. The HTTP response bodies will not be forwarded back to the clients.

/health endpoint

This endpoint provides a quick way to check the state of all the backends. It will return a JSON object detailing the status of the backends like this :

{
  "status": "problem",
  "problem": {
    "local-influxdb01": "KO. Get http://influxdb/ping: dial tcp: lookup influxdb on 0.0.0.0:8086: no such host"
  },
  "healthy": {
    "local-influxdb02": "OK. Time taken 3ms"
  }
}

If the relay encounters an error while checking a backend, this backend will be reported with the associated error in the problems object. The backends wich the relay was able to communicate with will be reported in the healthy object.

The status field is a summary of the general state of the backends, the defined states are as follows:

  • healthy: no errors were encountered
  • problem: some backends, but no all of them, returned errors
  • critical: every backend returned an error

Filters

We allow tags and measurements filtering through regular expressions. Please, take a look at this document for more information.

Limitations

So far, this is compatible with Debian, RedHat, and other derivatives.

Development

Please read carefully CONTRIBUTING.md before making a merge request.

Clone repository into your $GOPATH.

mkdir -p ${GOPATH}/src/github.com/vente-privee
cd ${GOPATH}/src/github.com/vente-privee
git clone git@github.com:vente-privee/influxdb-relay

Enter the directory and build the daemon.

cd ${GOPATH}/src/github.com/vente-privee/influxdb-relay
go build -a -ldflags '-extldflags "-static"' -o influxdb-relay

Miscellaneous

    ╚⊙ ⊙╝
  ╚═(███)═╝
 ╚═(███)═╝
╚═(███)═╝
 ╚═(███)═╝
  ╚═(███)═╝
   ╚═(███)═╝