Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Scribe input/output plugin for Fluentd data collector
Failed to load latest commit information.
bin add body_format param, for whom uses Scribe to transfer JSON
lib/fluent/plugin Fix Log handler to reduce call times of Engine.emit
test/plugin Improve invalid record handling
.gitignore remove jeweler, move to bundler
.travis.yml remove unsupported versions on fluentd
AUTHORS first commit of fluent-scribe-plugin
ChangeLog v0.10.14
Gemfile remove jeweler, move to bundler
README.rdoc
Rakefile remove jeweler, move to bundler
VERSION Version bump to 0.10.10
example.conf use scribe category as fluent tag
fluent-plugin-scribe.gemspec v0.10.14

README.rdoc

Scribe input plugin for Fluentd

Overview

This is a plugin for fluentd data collector. This plugin adds the Scribe compatible interface to fluentd.

What's Scribe?

Scribe is a server for aggregating log data streamed in real time from a large number of servers, developed at Facebook.

It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.

What's Scribe plugin for fluent?

The Scribe plugin for fluentd, which enables fluentd to talk the Scribe protocol. Scribe protocol is defined as follows, in Thrift-IDL format:

enum ResultCode
{
  OK,
  TRY_LATER
}

struct LogEntry
{
  1:  string category,
  2:  string message
}

service scribe extends fb303.FacebookService
{
  ResultCode Log(1: list<LogEntry> messages);
}

The category field is used as fluentd 'tag'.

How to use?

fluent-plugin-scribe contains both input and output.

Scribe Input

Please add the following configurations to fluent.conf. This allows your Scribe client to import logs through port 1463.

# Scribe input
<source>
  type scribe
  port 1463
</source>

These options are supported.

  • port: port number (default: 1463)

  • bind: bind address (default: 0.0.0.0)

  • server_type: server architecture either in 'simple', 'threaded', 'thread_pool', 'nonblocking' (default: nonblocking)

  • is_framed: use framed protocol or not (default: true)

  • add_prefix: prefix string, added to the tag (default: nil)

  • msg_format: format of the messages either in 'text', 'json', 'url_param' (default: text)

Scribe Output

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Scribe server. Note that fluentd conveys semi-structured data while Scribe conveys unstructured data, thus, 'field_ref' parameter is prepared to specify which field will be transferred.

# Scribe output
<match *>
  type scribe
  host scribed-host.local
  port 1463
  field_ref message
</match>

These options are supported.

  • host: host name or address (default: localhost)

  • port: port number (default: 1463)

  • field_ref: field name which sent as scribe log message (default: message)

  • timeout: thrift protocol timeout (default: 30)

  • format_to_json: if true/yes, format entire record as json, and send as message (default: false)

  • remove_prefix: prefix string, removed from the tag (default: nil)

For Developers

To run fluentd with this plugin on chaging,

$ bundle # (or 'bundle update')
$ bundle exec fluentd -v -v -v -c example.conf

Then please execute the sample client.

$ bundle exec bin/fluent-scribe-remote

Contributors

Copyright

Copyright

Copyright © 2011 Treasure Data, Inc.

License

Apache License, Version 2.0

Something went wrong with that request. Please try again.