Fluentd plugin to count the number of matched messages, and emit if exeeds the threshold
Ruby
Clone or download
Latest commit 661f801 Jan 23, 2017

README.md

fluent-plugin-grepcounter

Build Status Coverage Status

Fluentd plugin to count the number of matched messages, and emit if exeeds the threshold.

Configuration

Assume inputs from another plugin are as belows:

syslog.host1: {"message":"20.4.01/13T07:02:11.124202 INFO GET /ping" }
syslog.host1: {"message":"20.4.01/13T07:02:13.232645 WARN POST /auth" }
syslog.host1: {"message":"20.4.01/13T07:02:21.542145 WARN GET /favicon.ico" }
syslog.host1: {"message":"20.4.01/13T07:02:43.632145 WARN POST /login" }

An example of grepcounter configuration:

<match syslog.**>
  type grepcounter
  count_interval 60
  input_key message
  regexp WARN
  exclude favicon.ico
  threshold 1
  add_tag_prefix warn.count
</match>

Then, output bocomes as belows (indented):

warn.count.syslog.host1: {
  "count":2,
  "message":["20.4.01/13T07:02:13.232645 WARN POST /auth","20.4.01/13T07:02:43.632145 WARN POST /login"],
  "input_tag":"syslog.host1",
  "input_tag_last":"host1",
}

Output message by joining with a delimiter

As default, the grepcounter plugin outputs matched message as an array as shown above. You may want to output message as a string, then use delimiter option like:

<match syslog.**>
  type grepcounter
  count_interval 60
  input_key message
  regexp WARN
  exclude favicon.ico
  threshold 1
  add_tag_prefix warn.count
  delimiter \n
</match>

Then, output bocomes as belows (indented). You can see the message field is joined with \n.

warn.count.syslog.host1: {
  "count":2,
  "message":"20.4.01/13T07:02:13.232645 WARN POST /auth\n20.4.01/13T07:02:43.632145 WARN POST /login",
  "input_tag":"syslog.host1",
  "input_tag_last":"host1",
}

Parameters

  • count_interval

    The interval time to count in seconds. Default is 60.

  • input_key field_key

    The target field key to grep out. Use with regexp or exclude.

  • regexp regexp

    The filtering regular expression

  • exclude regexp

    The excluding regular expression like grep -v

  • regexp[1-20] field_key regexp (experimental)

    The target field key and the filtering regular expression to grep out. No message is outputted in this case.

  • exclude[1-20] field_key regexp (experimental)

    The target field key and the excluding regular expression like grep -v. No message is outputted in this case.

  • threshold

    The threshold number to emit. Emit if count value >= specified value.

  • greater_equal

    This is same with threshold option. Emit if count value is greater than or equal to (>=) specified value.

  • greater_than

    Emit if count value is greater than (>) specified value.

  • less_than

    Emit if count value is less than (<) specified value.

  • less_equal

    Emit if count value is less than or equal to (<=) specified value.

  • tag

    The output tag. Required for aggregate all.

  • add_tag_prefix

    Add tag prefix for output message

  • remove_tag_prefix

    Remove tag prefix for output message

  • add_tag_suffix

    Add tag suffix for output message

  • remove_tag_suffix

    Remove tag suffix for output message

  • remove_tag_slice min..max

    Remove tag parts by slice function. FYI: This option behaves like tag.split('.').slice(min..max).

    For example,

      remove_tag_slice 0..-2
    

    changes an input tag foo.bar.host1 to foo.bar.

  • aggregate

    Aggregation unit. One of all, in_tag, out_tag can be specified. Default is all.

    • all counts summation for all input messages and emit one message in each interval.
    • in_tag counts summation for each input tag seperately.
    • out_tag counts summation for each tag modified by add_tag_prefix, remove_tag_prefix, or remove_tag_slice.
  • delimiter

    Output matched messages after joined with the specified delimiter.

  • replace_invalid_sequence

    Replace invalid byte sequence in UTF-8 with '?' character if true

  • store_file

    Store internal count data into a file of the given path on shutdown, and load on statring.

ChangeLog

See CHANGELOG.md for details.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Copyright

Copyright (c) 2013 Naotoshi SEO. See LICENSE for details.