Fluentd plugin to add or replace fields of a event record
Ruby
Clone or download

README.md

fluent-plugin-record-reformer

Build Status

Fluentd plugin to add or replace fields of a event record

Requirements

See .travis.yml

Note that fluent-plugin-record-reformer supports both v0.14 API and v0.12 API in one gem.

Installation

Use RubyGems:

gem install fluent-plugin-record-reformer

Configuration

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  
  tag reformed.${tag_prefix[-2]}
  <record>
    hostname ${hostname}
    input_tag ${tag}
    last_tag ${tag_parts[-1]}
    message ${record['message']}, yay!
  </record>
</match>

Assume following input is coming (indented):

foo.bar {
  "remove_me":"bar",
  "not_remove_me":"bar",
  "message":"Hello world!"
}

then output becomes as below (indented):

reformed.foo {
  "not_remove_me":"bar",
  "hostname":"YOUR_HOSTNAME",
  "input_tag":"foo.bar",
  "last_tag":"bar",
  "message":"Hello world!, yay!",
}

Configuration (Classic Style)

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  tag reformed.${tag_prefix[-2]}
  
  hostname ${hostname}
  input_tag ${tag}
  last_tag ${tag_parts[-1]}
  message ${record['message']}, yay!
</match>

This results in same, but please note that following option parameters are reserved, so can not be used as a record key.

Option Parameters

  • output_tag (obsolete)

    The output tag name. This option is deprecated. Use tag option instead

  • tag

    The output tag name.

  • remove_keys

    Specify record keys to be removed by a string separated by , (comma) like

      remove_keys message,foo
    
  • renew_record bool

    renew_record true creates an output record newly without extending (merging) the input record fields. Default is false.

  • renew_time_key string

    renew_time_key foo overwrites the time of events with a value of the record field foo if exists. The value of foo must be a unix time.

  • keep_keys

    You may want to remain some record fields although you specify renew_record true. Then, specify record keys to be kept by a string separated by , (comma) like

      keep_keys message,foo
    
  • enable_ruby bool

    Enable to use ruby codes in placeholders. See Placeholders section. Default is true (just for lower version compatibility).

  • auto_typecast bool

    Automatically cast the field types. Default is false. NOTE: This option is effective only for field values comprised of a single placeholder.

    Effective Examples:

      foo ${foo}
    

    Non-Effective Examples:

      foo ${foo}${bar}
      foo ${foo}bar
      foo 1
    

    Internally, this keeps the type of value if the value text is comprised of a single placeholder, otherwise, values are treated as strings.

    When you need to cast field types manually, out_typecast and filter_typecast are available.

Placeholders

Following placeholders are available:

  • ${record["key"]} Record value of key such as ${record["message"]} in the above example (available from v0.8.0).

    • Originally, record placeholders were available as ${key} such as ${message}. This is still kept for the backward compatibility, but would be removed in the future.
  • ${hostname} Hostname of the running machine

  • ${tag} Input tag

  • ${time} Time of the event

  • ${tags[N]} (Obsolete. Use tag_parts) Input tag splitted by '.'

  • ${tag_parts[N]} Input tag splitted by '.' indexed with N such as ${tag_parts[0]}, ${tag_parts[-1]}.

  • ${tag_prefix[N]} Tag parts before and on the index N. For example,

      Input tag: prefix.test.tag.suffix
      
      ${tag_prefix[0]}  => prefix
      ${tag_prefix[1]}  => prefix.test
      ${tag_prefix[-2]} => prefix.test.tag
      ${tag_prefix[-1]} => prefix.test.tag.suffix
    
  • ${tag_suffix[N]} Tag parts after and on the index N. For example,

      Input tag: prefix.test.tag.suffix
    
      ${tag_suffix[0]}  => prefix.test.tag.suffix
      ${tag_suffix[1]}  => test.tag.suffix
      ${tag_suffix[-2]} => tag.suffix
      ${tag_suffix[-1]} => suffix
    

It is also possible to write a ruby code in placeholders if you set enable_ruby true option, so you may write some codes as

  • ${time.strftime('%Y-%m-%dT%H:%M:%S%z')}
  • ${tag_parts.last}

but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance.

Relatives

Following plugins look similar:

ChangeLog

See CHANGELOG.md for details.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Copyright

Copyright (c) 2013 - 2015 Naotoshi Seo. See LICENSE for details.