fluent-plugin-map is the non-buffered plugin that can convert an event log to different event log(s)
Switch branches/tags
Clone or download
Latest commit f6fa660 Feb 27, 2018
Permalink
Failed to load latest commit information.
.github Fix a typo Dec 28, 2017
gemfiles Remove needless gemfiles Mar 16, 2017
lib/fluent/plugin Support multiple workers Apr 25, 2017
test Remove needless if and begin branches Mar 16, 2017
.gitignore
.travis.yml travis: Update Ruby versions Jan 30, 2018
Appraisals Appraisals: Remove needless entries Mar 16, 2017
Gemfile
LICENSE addLicense Jun 21, 2015
README.markdown Add requirement section Mar 16, 2017
Rakefile first commit Jul 9, 2012
fluent-plugin-map.gemspec Bump up version to 0.2.1 Apr 25, 2017

README.markdown

fluent-plugin-map

Build Status

fluent-plugin-map(out_map) is the non-buffered plugin that can convert an event log to different event log(s)

Requirements

fluent-plugin-map fluentd ruby
>= 0.2.0 >= v0.14.0 >= 2.1
< 0.2.0 >= v0.12.0 >= 1.9

MapFilter

Example

This sample config filter code file and time file.

<source>
  @type tail
  format apache
  path /var/log/httpd-access.log
  tag tag
  @label @raw
</source>
<label @raw>
  <match **>
    @type copy
    <store>
      @type relabel
      @label @code
    </store>
    <store>
      @type relabel
      @label @time
    </store>
  </match>
</label>
<label @code>
  <filter **>
    @type map
    map ([time, {"code" => record["code"].to_i}])
  </filter>
  <match **>
    @type file
    path code.log
  </match>
</label>
<label @time>
  <filter **>
    @type map
    map ([time, {"time" => record["time"].to_i}])
  </filter>
  <match **>
    @type file
    path time.log
  </match>
</label>

The parameter "map" can use 2 variables in event log; time, record. The format of time is an integer number of seconds since the Epoch. The format of record is hash. The config file parses # as the begin of comment. So the "map" value cannot use #{tag} operation. This plugin can output multi logs by seting multi to true.

If you don't use multi option, you can use time, record parameter. The 2 following filter directive is same:

<filter tag>
  @type map
  map ([time, {"code" => record["code"].to_i}])
</filter>
<filter tag>
  @type map
  time time
  record ({"code" => record["code"].to_i})
</filter>

MapOutput

Example

This sample config output code file and time file.

<source>
  @type tail
  format apache
  path /var/log/httpd-access.log
  tag tag
</source>
<match tag>
  @type map
  map ([["code." + tag, time, {"code" => record["code"].to_i}], ["time." + tag, time, {"time" => record["time"].to_i}]])
  multi true
</match>
<match code.tag>
  @type file
  path code.log
</match>
<match time.tag>
  @type file
  path time.log
</match>

The parameter "map" can use 3 variables in event log; tag, time, record. The format of time is an integer number of seconds since the Epoch. The format of record is hash. The config file parses # as the begin of comment. So the "map" value cannot use #{tag} operation. This plugin can output multi logs by seting multi to true.

If you don't use multi option, you can use key, time, record parameter. The 2 following match directive is same:

<match tag>
  @type map
  map (["code." + tag, time, {"code" => record["code"].to_i}])
</match>
<match tag>
  @type map
  tag ("code." + tag)
  time time
  record ({"code" => record["code"].to_i})
</match>

Note

you have to wrap some configuration values with parenthesis like ("code." + tag), to avoid parsing by Fluentd itself. See also: Fluentd | Configuration File | Format tips

Copyright

  • Copyright (c) 2015- Tomita Kohei
  • Apache License, Version 2.0