Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Flume input and output plugin for Fluentd event collector
branch: master

This branch is 2 commits ahead, 13 commits behind fluent:master

Merge pull request #1 from kiyoto/patch-1

Update README.rdoc
latest commit f3bc90c5ee
Muga Nishizawa authored
Failed to load latest commit information.
bin first commit
lib/fluent/plugin first commit
test/plugin first commit
.gitignore first commit
AUTHORS first commit
ChangeLog first commit
README.rdoc Update README.rdoc
Rakefile first commit
VERSION
example-in-flume.conf first commit
example-out-flume.conf first commit
fluent-plugin-flume.gemspec first commit
sample_client.rb first commit
sample_server.rb first commit

README.rdoc

Flume input/output plugin for Fluentd

Overview

This is a plugin for fluentd event collector. This plugin adds the Flume compatible interface to fluentd.

What's Flume?

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.

It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.

What's Flume plugin for fluent?

The Flume plugin for fluentd, which enables fluentd to talk the Flume protocol. Flume protocol is defined as follows, in Thrift-IDL format:

typedef i64 Timestamp

enum Priority {
  FATAL = 0,
  ERROR = 1,
  WARN = 2,
  INFO = 3,
  DEBUG = 4,
  TRACE = 5
}

enum EventStatus {
  ACK = 0,
  COMMITED = 1,
  ERR = 2
}

struct ThriftFlumeEvent {
  1: Timestamp timestamp,
  2: Priority priority,
  3: binary body,
  4: i64 nanos,
  5: string host,
  6: map<string,binary> fields
}

# Instead of using thrift's serialization, we just assume the contents are serialized already.
struct RawEvent {
  1: binary raw
}

service ThriftFlumeEventServer {
  oneway void append( 1:ThriftFlumeEvent evt ),
  oneway void rawAppend( 1:RawEvent evt),
  EventStatus ackedAppend( 1: ThriftFlumeEvent evt ),

  void close(),
}

A value that is stored in the ThriftFlumeEvent.fields map is used as fluentd 'tag'. A key of the value enables be specified by users as configuration parameter.

How to use?

fluent-plugin-flume contains both input and output.

Flume Input

Please add the following configurations to fluent.conf.

# Flume input
<source>
  type flume
  port 56789
</source>

These options are supported.

  • port: port number (default: 56789)

  • bind: bind address (default: 0.0.0.0)

  • server_type: server architecture either in 'simple', 'threaded', 'thread_pool', (default: simple)

  • is_framed: use framed protocol or not (default: false)

  • tag_field: key name of fluentd 'tag' that is stored in ThriftFlumeEvent.fields (default: nil)

  • default_tag: default fluentd 'tag' (default: 'category')

  • add_prefix: prefix string, added to the tag (default: nil)

Flume Output

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Flume server. Note that fluentd conveys semi-structured data while Flume conveys unstructured data. Thus the plugin translates semi-structured data into JSON data and conveys it to Flume.

# Flume output
<match *>
  type flume
  host flume-host.local
  port 56789
</match>

These options are supported.

  • host: host name or address (default: localhost)

  • port: port number (default: 35863)

  • timeout: thrift protocol timeout (default: 30)

  • remove_prefix: prefix string, removed from the tag (default: nil)

Contributors

Copyright

Copyright

Copyright © 2012 Treasure Data, Inc.

License

Apache License, Version 2.0

Something went wrong with that request. Please try again.