A set of OpenTSDB plugins for consuming from Apache Kafka
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src Fix javadocs. Jan 1, 2019
.gitignore Initiali commit. Jul 11, 2017
.travis.yml Missing `fi`. Jan 2, 2019
LICENSE Change to The Apache 2.0 license. Jan 1, 2019
README.md Add a note about the shaded flag. Jan 1, 2019
pom.xml Set version to 2.4.1-SNAPSHOT Jan 2, 2019
settings.xml Add a settings and try fixing the yaml. Jan 2, 2019

README.md

   ___                 _____ ____  ____  ____
  / _ \ _ __   ___ _ _|_   _/ ___||  _ \| __ )
 | | | | '_ \ / _ \ '_ \| | \___ \| | | |  _ \
 | |_| | |_) |  __/ | | | |  ___) | |_| | |_) |
  \___/| .__/ \___|_| |_|_| |____/|____/|____/
       |_|    The modern time series database.

Build Status Coverage Status

Kafka RPC Plugin

This plugin allows OpenTSDB to consume messages from a Kafka cluster and write them directly to storage, bypassing the Telnet style or HTTP APIs. It includes a Storage Exception Handler plugin that will post messages back to a Kafka queue if writing to storage fails.

NOTE: This branch is compatible with OpenTSDB 2.4.x. For 2.3.x please use the 2.3 branch from Git.

Installation

  1. Download the source code and run mvn package -Pshaded to create the shaded jar in the target/ directory. Copy this file to your OpenTSDB plugin directory as defined in the opentsdb config via tsd.core.plugin_path (Note that without the -P flag it will simply build a basic jar without the Kafka libraries. Those will need to be on your class path.).
  2. Setup the appropriate Kafka topics and partitions. More on that later.
  3. Add configuration settings to your opentsdb.conf file as described later on.
  4. Restart the TSD and make sure the plugin was loaded and associated with the proper ID. E.g. look in the logs for lines like:
2017-07-10 23:08:58,264 INFO  [main] RpcManager: Successfully initialized plugin [net.opentsdb.tsd.KafkaRpcPlugin] version: 2.4.0
2017-07-10 23:08:57,790 INFO  [main] TSDB: Successfully initialized storage exception handler plugin [net.opentsdb.tsd.KafkaStorageExceptionHandler] version: 2.4.0

Usage

The plugin can accept various message formats using classes implementing the net.opentsdb.data.deserializers.Deserializer interface. A Deserializer must be defined per consumer group and all messages in the topics must be of the same format.

For default JSON formatted messages, use the net.opentsdb.data.deserializers.JSONDeserializer. Each JSON message must include a type field with the value being one of the following:

  • Metric A single numeric measurement.
  • Aggregate A single numeric measurement that may be a pre-aggregate, a rolled up data point or both.
  • Histogram A single histogram measurement.

Each message is similar to the HTTP JSON messages in the OpenTSDB API with the addition of the type field so that the JSON deserializer can figure out what the message contains.

The deserialization class must return a list of typed objects as defined below (with JSON provided as an example).

###Metric

The metric message appears as follows:

{
	"type": "Metric",
	"metric": "sys.cpu.user",
	"tags": {
		"host": "web01"
	},
	"timestamp": 1492641000,
	"value": 42
}

For field information, see [/api/put] (http://opentsdb.net/docs/build/html/api_http/put.html).

###Aggregate

Aggregate messages are the same as those documented in [/api/rollup] (http://opentsdb.net/docs/build/html/api_http/rollup.html).

###Histogram

Histogram messages are documented at [/api/histogram] (http://opentsdb.net/docs/build/html/api_http/histogram.html).

Configuration

The following properties can be stored in the opentsdb.conf file:

Property Type Required Description Default Example
tsd.rpc.plugins String Required The full class name of the plugin. This must be net.opentsdb.tsd.KafkaRpcPlugin net.opentsdb.tsd.KafkaRpcPlugin
KafkaRpcPlugin.kafka.zookeeper.connect String Required The comma separated list of zookeeper hosts and ports used by the Kafka cluster. localhost:2181
KafkaRpcPlugin.groups String Required A comma separated list of one or more consumer group names. TsdbConsumer,TsdbRequeueConsumer
KafkaRpcPlugin.<GROUPNAME>.topics String Required A comma separated list of one or more topics for the <GROUPNAME> to consume from. TSDB_1,TSDB_2
KafkaRpcPlugin.<GROUPNAME>.consumerType String Required The type of messages written to the queue. TODO. For now, leave it as raw raw
KafkaRpcPlugin.<GROUPNAME>.deserializer String Required The deserialization class to use for parsing messages from the Kafka topic. net.opentsdb.data.deserializers.JSONDeserializer
KafkaRpcPlugin.<GROUPNAME>.rate Integer Required How many messages per second to throttle the total of consumer threads at for the consumer group 250000
KafkaRpcPlugin.<GROUPNAME>.threads Integer Required The number of consumer threads to create per group 4
tsd.http.rpc.plugins String Optional A comma separated list of HTTP RPC plugins to load. Included with this package is a plugin that allows for fetching stats from the Kafka plugin as well as viewing or modifying the write rate during runtime. net.opentsdb.tsd.KafkaHttpRpcPlugin
tsd.core.storage_exception_handler.enable Boolean Optional Whether or not to enable the storage exception handler plugin. false true
tsd.core.storage_exception_handler.plugin String Optional The full class of the storage exception handler plugin. net.opentsdb.tsd.KafkaStorageExceptionHandler
KafkaRpcPlugin.kafka.metadata.broker.list String Optional The comma separated list of Kafka brokers and ports used to write messages to for the storage exception handler plugin localhost:9092
KafkaRpcPlugin.seh.topic.default String Optional The topic used to write messages to for the storage exception handler. TSDB_Requeue

Note the KafkaRpcPlugin.groups and <GROUP_NAME> entries above. Kafka consumers belong to a particular group. The Kafka RPC plugin can launch multiple groups consuming from multiple topics so that OpenTSDB messages can be organized by type or source for more efficient control over rate limits and priorities. When setting the KafkaRpcPlugin.groups value, make sure you have a complete set of KafkaRpcPlugin.<GROUP_NAME>.* parameters per group or initialization will fail.