Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Flume input and output plugin for Fluentd data collector
Ruby
branch: master
Failed to load latest commit information.
bin first commit
lib/fluent/plugin Support log_level option
test/plugin first commit
.gitignore first commit
AUTHORS first commit
ChangeLog
README.rdoc Update README.rdoc
Rakefile updated thrift to 0.8.0
VERSION prepared releasing 0.1.1
example-in-flume.conf first commit
example-out-flume.conf first commit
fluent-plugin-flume.gemspec Regenerate gemspec for version 0.1.1
sample_client.rb
sample_server.rb first commit

README.rdoc

Flume input/output plugin for Fluentd

Overview

This is a plugin for fluentd data collector. This plugin adds the Flume compatible interface to fluentd.

What's Flume?

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.

It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.

What's Flume plugin for fluent?

The Flume plugin for fluentd, which enables fluentd to talk the Flume protocol. Flume protocol is defined as follows, in Thrift-IDL format:

typedef i64 Timestamp

enum Priority {
  FATAL = 0,
  ERROR = 1,
  WARN = 2,
  INFO = 3,
  DEBUG = 4,
  TRACE = 5
}

enum EventStatus {
  ACK = 0,
  COMMITED = 1,
  ERR = 2
}

struct ThriftFlumeEvent {
  1: Timestamp timestamp,
  2: Priority priority,
  3: binary body,
  4: i64 nanos,
  5: string host,
  6: map<string,binary> fields
}

# Instead of using thrift's serialization, we just assume the contents are serialized already.
struct RawEvent {
  1: binary raw
}

service ThriftFlumeEventServer {
  oneway void append( 1:ThriftFlumeEvent evt ),
  oneway void rawAppend( 1:RawEvent evt),
  EventStatus ackedAppend( 1: ThriftFlumeEvent evt ),

  void close(),
}

A value that is stored in the ThriftFlumeEvent.fields map is used as fluentd 'tag'. A key of the value enables be specified by users as configuration parameter.

How to use?

fluent-plugin-flume contains both input and output.

Flume Input

Please add the following configurations to fluent.conf.

# Flume input
<source>
  type flume
  port 56789
</source>

These options are supported.

  • port: port number (default: 56789)

  • bind: bind address (default: 0.0.0.0)

  • server_type: server architecture either in 'simple', 'threaded', 'thread_pool', (default: simple)

  • is_framed: use framed protocol or not (default: false)

  • tag_field: key name of fluentd 'tag' that is stored in ThriftFlumeEvent.fields (default: nil)

  • default_tag: default fluentd 'tag' (default: 'category')

  • add_prefix: prefix string, added to the tag (default: nil)

Flume Output

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Flume server. Note that fluentd conveys semi-structured data while Flume conveys unstructured data. Thus the plugin translates semi-structured data into JSON data and conveys it to Flume.

# Flume output
<match *>
  type flume
  host flume-host.local
  port 56789
</match>

These options are supported.

  • host: host name or address (default: localhost)

  • port: port number (default: 35863)

  • timeout: thrift protocol timeout (default: 30)

  • remove_prefix: prefix string, removed from the tag (default: nil)

Contributors

Copyright

Copyright

Copyright © 2012 - 2013 Treasure Data, Inc.

License

Apache License, Version 2.0

Something went wrong with that request. Please try again.