No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.gitignore
.travis.yml
README.md

README.md

Moved to aliyun/aliyun-maxcompute-data-collectors: https://github.com/aliyun/aliyun-maxcompute-data-collectors

Odps Sink Plugin for Apache Flume

Build Status


The Odps sink streams events data into a datahub topic. It has the following features:

  • Map structured data to corresponding columns in the datahub topic.
  • Currently only support delimited text data.
  • Highly customizable.
  • Compatible with all the features of Flume, such as fan-in and fan-out flows, contextual routing and backup routes (fail-over) for failed hops.

NOTE: For non-developer users, please goto (click here).

Getting Started


Requirements

To get started using this plugin, you will need three things:

  1. JDK 1.6 or later (JDK 1.7 recommended)
  2. Apache Maven 3.x
  3. Flume-NG 1.x (Home Page)

Clone and Build the Package

Clone the project from gitlab:

$ git clone git@github.com:aliyun/aliyun-odps-flume-plugin.git

Use maven to build the package:

$ cd aliyun-odps-flume-plugin/
$ mvn clean package -DskipTests

Wait until building success, the plugin will be aliyun-odps-flume-plugin/target/flume-datahub-sink-x.x.x.tar.gz.

Use the Sink in Flume

$ tar zxvf flume-datahub-sink-x.x.x.tar.gz
$ ls flume-datahub-sink
lib    libext

Move the plugin flume-datahub-sink into the plugin directory of Flume (i.e., the folder plugins.d/ under the Flume installation directory). If the plugin directory does not exist, create it at first:

$ mkdir {YOUR_FLUME_DIRECTORY}/plugins.d
$ mv flume-datahub-sink {YOUR_FLUME_DIRECTORY}/plugins.d/

Optionally, you can check if the plugin is already in the directory:

$ ls {YOUR_FLUME_DIRECTORY}/plugins.d
flume-datahub-sink

The ODPS sink should be available for Flume now. You can use this sink by set the type of the Flume sink to com.aliyun.datahub.flume.sink.DatahubSink. Details about the configure paramters of the Datahub sink are listed in Sink Paramters.

Tutorial


Following are some examples that use Flume along with this plugin to upload structured log data to Datahub topic. This tutorial is designed for the ODPS users who had not used Flume before, and will illustrate the basics about how to use Flume to upload streaming data into Datahub.

Useful Links


Authors && Contributors


License


licensed under the Apache License 2.0