Scribe input and output plugin for Fluentd event collector
Ruby
Pull request Compare This branch is 39 commits behind fluent:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
lib/fluent/plugin
test/plugin
.gitignore
AUTHORS
ChangeLog
README.rdoc
Rakefile
VERSION
example.conf
fluent-plugin-scribe.gemspec

README.rdoc

Scribe input plugin for Fluent

Overview

This is a plugin for fluentd event collector. This plugin adds the Scribe compatible interface to fluentd.

What's Scribe?

Scribe is a server for aggregating log data streamed in real time from a large number of servers, developed at Facebook.

It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.

What's Scribe plugin for fluent?

The Scribe plugin for fluentd, which enables fluentd to talk the Scribe protocol. Scribe protocol is defined as follows, in Thrift-IDL format:

enum ResultCode
{
  OK,
  TRY_LATER
}

struct LogEntry
{
  1:  string category,
  2:  string message
}

service scribe extends fb303.FacebookService
{
  ResultCode Log(1: list<LogEntry> messages);
}

The category field is used as fluentd 'tag'.

How to use?

fluent-plugin-scribe contains both input and output.

Scribe Input

Please add the following configurations to fluent.conf. This allows your Scribe client to import logs through port 1463.

# Scribe input
<source>
  type scribe
  port 1463
</source>

These options are supported.

  • port: port number (default: 1463)

  • bind: bind address (default: 0.0.0.0)

  • server_type: server architecture either in 'simple', 'threaded', 'thread_pool', 'nonblocking' (default: nonblocking)

  • is_framed: use framed protocol or not (default: true)

  • add_prefix: prefix string, added to the tag (default: nil)

  • msg_format: format of the messages either in 'text', 'json', 'url_param' (default: text)

Scribe Output

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Scribe server. Note that fluentd conveys semi-structured data while Scribe conveys unstructured data, thus, 'field_ref' parameter is prepared to specify which field will be transferred.

# Scribe output
<match *>
  type scribe
  host scribed-host.local
  port 1463
  field_ref message
</match>

These options are supported.

  • host: host name or address (default: localhost)

  • port: port number (default: 1463)

  • field_ref: field name which sent as scribe log message (default: message)

  • timeout: thrift protocol timeout (default: 30)

  • body_format: parse a body message as a specified format (default: 'text', option: 'json')

  • remove_prefix: prefix string, removed from the tag (default: nil)

For Developers

To test this plugin, please execute the following command in your 'fluent' project directory, not scribe input plugin directory.

$ export SCRIBE_PLUGIN_DIR=../fluent-plugin-scribe                                                                   
$ FLUENT_PLUGIN=$SCRIBE_PLUGIN_DIR/lib/fluent/plugin/ ./bin/fluentd --config $SCRIBE_PLUGIN_DIR/example.conf -v -v -v

Then please execute the sample client.

$ ./bin/fluent-scribe-remote

Contributors

Copyright

Copyright

Copyright © 2011 Treasure Data, Inc.

License

Apache License, Version 2.0