Scribe input/output plugin for Fluentd data collector
Ruby Thrift
Latest commit f207012 Dec 19, 2014 @repeatedly repeatedly v0.10.14


Scribe input plugin for Fluentd


This is a plugin for fluentd data collector. This plugin adds the Scribe compatible interface to fluentd.

What's Scribe?

Scribe is a server for aggregating log data streamed in real time from a large number of servers, developed at Facebook.

It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.

What's Scribe plugin for fluent?

The Scribe plugin for fluentd, which enables fluentd to talk the Scribe protocol. Scribe protocol is defined as follows, in Thrift-IDL format:

enum ResultCode

struct LogEntry
  1:  string category,
  2:  string message

service scribe extends fb303.FacebookService
  ResultCode Log(1: list<LogEntry> messages);

The category field is used as fluentd 'tag'.

How to use?

fluent-plugin-scribe contains both input and output.

Scribe Input

Please add the following configurations to fluent.conf. This allows your Scribe client to import logs through port 1463.

# Scribe input
  type scribe
  port 1463

These options are supported.

  • port: port number (default: 1463)

  • bind: bind address (default:

  • server_type: server architecture either in 'simple', 'threaded', 'thread_pool', 'nonblocking' (default: nonblocking)

  • is_framed: use framed protocol or not (default: true)

  • add_prefix: prefix string, added to the tag (default: nil)

  • msg_format: format of the messages either in 'text', 'json', 'url_param' (default: text)

Scribe Output

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Scribe server. Note that fluentd conveys semi-structured data while Scribe conveys unstructured data, thus, 'field_ref' parameter is prepared to specify which field will be transferred.

# Scribe output
<match *>
  type scribe
  host scribed-host.local
  port 1463
  field_ref message

These options are supported.

  • host: host name or address (default: localhost)

  • port: port number (default: 1463)

  • field_ref: field name which sent as scribe log message (default: message)

  • timeout: thrift protocol timeout (default: 30)

  • format_to_json: if true/yes, format entire record as json, and send as message (default: false)

  • remove_prefix: prefix string, removed from the tag (default: nil)

For Developers

To run fluentd with this plugin on chaging,

$ bundle # (or 'bundle update')
$ bundle exec fluentd -v -v -v -c example.conf

Then please execute the sample client.

$ bundle exec bin/fluent-scribe-remote




Copyright © 2011 Treasure Data, Inc.


Apache License, Version 2.0