fluent-plugin-stats

Fluentd plugin to calculate statistics such as sum, max, min, avg.

Configuration

Example 1

Get sum for xxx_count, max for xxx_max, min for xxx_min, avg for xxx_avg

<match foo.**>
  type stats
  interval 5s
  add_tag_prefix stats

  sum .*_count$
  max .*_max$
  min .*_min$
  avg .*_avg$
</match>

Assuming following inputs are coming:

foo.bar: {"4xx_count":1,"5xx_count":2","reqtime_max":12083,"reqtime_min":10,"reqtime_avg":240.46}
foo.bar: {"4xx_count":4,"5xx_count":2","reqtime_max":24831,"reqtime_min":82,"reqtime_avg":300.46}

then output bocomes as belows:

stats.foo.bar: {"4xx_count":5,"5xx_count":4","reqtime_max":24831,"reqtime_min":10,"reqtime_avg":270.46}

Example 2

Get sum, max, min, avg for the same key

<match foo.**>
  type stats
  interval 5s
  add_tag_prefix stats

  sum ^reqtime$
  max ^reqtime$
  min ^reqtime$
  avg ^reqtime$
  sum_suffix _sum
  max_suffix _max
  min_suffix _min
  avg_suffix _avg
</match>

Assuming following inputs are coming:

foo.bar: {"reqtime":1.000}
foo.bar: {"reqtime":2.000}

then output bocomes as belows:

stats.foo.bar: {"reqtime_sum":3.000,"reqtime_max":2.000,"reqtime_min":1.000,"reqtime_avg":1.500}

Parameters

• sum, min, max, avg

  Target of calculation. Specify input keys by a regular expression

• sum_keys, min_keys, max_keys, avg_keys

  Target of calculation. Specify input keys by a string separated by , (comma) such as

    sum_keys 4xx_count,5xx_count
  

• sum_suffix, min_suffix, max_suffix, avg_suffix

  Add a suffix to keys of the output record

• interval

  The interval to calculate in seconds. Default is 5s.

• tag

  The output tag name. Required for aggregate all.

• add_tag_prefix

  Add tag prefix for output message. Default: 'stats'

• remove_tag_prefix

  Remove tag prefix for output message.

• add_tag_suffix

  Add tag suffix for output message.

• remove_tag_suffix

  Remove tag suffix for output message.

• remove_tag_slice min..max

  Remove tag parts by slice function. FYI: This option behaves like tag.split('.').slice(min..max).

  For example,

    remove_tag_slice 0..-2
  

  changes an input tag foo.bar.host1 to foo.bar.

• aggregate

  Aggregation unit. One of all, in_tag, out_tag can be specified. Default is in_tag.

  • all calculate stats for all input messages and emit one message in each interval.
  • in_tag calculate stats for each input tag seperately.
  • out_tag calculate stats for for each tag modified by add_tag_prefix, remove_tag_prefix, or remove_tag_slice.

• store_file

  Store internal data into a file of the given path on shutdown, and load on starting.

• zero_emit

  Emit 0 on the next interval. This is useful for some software which requires to reset data such as GrowthForecast .

    stats.foo.bar: {"4xx_count":5,"5xx_count":4","reqtime_max":24831,"reqtime_min":10,"reqtime_avg":270.46}
    # after @interval later
    stats.foo.bar: {"4xx_count":0,"5xx_count":0","reqtime_max":0,"reqtime_min":0,"reqtime_avg":0}
  

ChangeLog

See CHANGELOG.md for details.

ToDo

Get the number of denominator to calculate avg from input json field.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Copyright

Copyright (c) 2013 Naotoshi Seo. See LICENSE for details.