A logstash plugin to store events via webhdfs.
Ruby
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
lib/logstash/outputs
spec/outputs
.gitignore
Gemfile
LICENSE
README.md
Rakefile
logstash-output-webhdfs.gemspec

README.md

This repository is discontinued. The plugin can now be found in the logstash repository @https://github.com/logstash-plugins/logstash-output-webhdfs

logstash-webhdfs

A logstash plugin to store events via webhdfs.

Tested with v1.3.3, v1.4.0 and 1.5.0.

It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

This plugin only has a mandatory dependency on the webhdfs gem from Kazuki Ohta and TAGOMORI Satoshi (@see: https://github.com/kzk/webhdfs). Optional dependencies are zlib and snappy gem.

No jars from hadoop are needed, thus reducing configuration and compatibility problems.

Installation

Change into your logstash install directory and execute:

bin/plugin install logstash-output-webhdfs

Documentation

Example configuration:

output {
    webhdfs {
        workers => 2
        host => "your.nameno.de"
        port => 14000
        user => "flume"
        path => "/user/flume/logstash/dt=%{+Y}-%{+M}-%{+d}/logstash-%{+H}.log"
        flush_size => 500
        compression => "snappy"
        idle_flush_time => 10
        retry_interval => 0.5
    }
}

For a complete list of options, see config section in source code.

This plugin has dependencies on: