Skip to content
This repository has been archived by the owner on Feb 16, 2022. It is now read-only.

eBay/hadoop-tsdb-connector

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 

Repository files navigation

The aim of the TSDB Connector for Hadoop project is

Developing a solution to push metrics to TSDB which is highly reliable and a built-in hadoop solution.

Every machine in hadoop cluster has a set of metrics associated with it which convey the state of the Hadoop process / system at that specific time. Currently there are external agents which are used to push these "metrics" to the database (TSDB ) .

We propose a solution to push these metrics from within the hadoop process itself providing more reliability inc ase the external agent stops working and

Compatible Versions:-

This system was tested with Hadoop versions :-

  1. Hadoop 1.2.1
  2. Hadoop 2.4.1

and OpenTSDB 2.0

The 2 files here are TSDBSink-1.2.1.java and TSDBSink-2.4.1.java for Hadoop 1.2.1 and Hadoop 2.4.1 Version respectively

Instructions :-

To use this Sink you need to place the files in the following folder

For Hadoop 1.2.1 <hadoop-1.2.1 source folder>/src/core/org/apache/hadoop/metrics2/sink/TSDBSink.java

For Hadoop 2.4.1 <hadoop-2.4.1 source folder >/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/TSDBSink.java

and compile hadoop.

Change the hadoop-metrics2.properties file and add the following in the sink

*.sink.tsdb.class=org.apache.hadoop.metrics2.sink.TSDBSink

*.period=10

namenode.sink.tsdb.servers="TSDB Server hostname":4242

datanode.sink.tsdb.servers="TSDB Server hostname":4242

Limitations:-

There are some limitations to this project. In case the TSDB server is restarted , the metrics are stopped .

Roadmap:-

  1. We are currently working on adding code to reset the connection in such a case upto a certain number of retires

  2. We are also adding the option for resetting connection in the hadoop dfsadmin utility.

About

No description, website, or topics provided.

Resources

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published