A sentry pluging to throttle logs. Logs are grouped by a configurable key. When a group exceeds a configuration rate, logs are dropped for this group.
install with gem or td-agent provided command as:
# for fluentd
$ gem install fluent-plugin-throttle<filter **>
@type throttle
group_key kubernetes.container_name
group_bucket_period_s 60
group_bucket_limit 6000
group_reset_rate_s 100
</filter>Default: kubernetes.container_name.
Used to group logs. Groups are rate limited independently.
A dot indicates a key within a sub-object. As an example, in the following log, the group key resolve to "random":
{"level": "error", "msg": "plugin test", "kubernetes": { "container_name": "random" } }
If the group cannot be resolved, am anonymous group is used.
Default: 60 (60 second).
This is the period of of time over which group_bucket_limit applies.
Default: 6000 (logs per group_bucket_period_s).
Maximum number logs allowed per groups over the period of group_bucket_period_s.
This translate to a log rate of group_bucket_limit/group_bucket_period_s.
When a group exceeds this rate, logs from this group are dropped.
For example, the default is 6000/60s, making for a rate of 100 logs per seconds.
Note that this is not expressed as a rate directly because there is a difference between the overall rate and the distribution of logs over a period time. For example, a burst of logs in the middle of a minute bucket might not exceed the average rate of the full minute.
Consider 60/60s, 60 logs over a minute, versus 1/1s, 1 log per second.
Over a minute, both will emit a maximum of 60 logs. Limiting to a rate of 60
logs per minute. However 60/60s will readily emit 60 logs within the first
second then nothing for the remaining 59 seconds. While the 1/1s will only
emit the first log of every second.
Default: group_bucket_limit/group_bucket_period_s (logs per group_bucket_period_s).
Maximum: group_bucket_limit/group_bucket_period_s.
After a group has exceeded its bucket limit, logs are dropped until the rate
per second falls below or equal to group_reset_rate_s.
The default value is group_bucket_limits/group_bucket_period_s. For example
for 3600 logs per hour, the reset will defaults to 3600/3600s = 1/s, one log
per second.
Taking the example 3600 log/hour with the default reset rate of 1 log/s
further:
- Let's say we have a period of 10 hours.
- During the first hour, 2 logs/s are produced. After 30 minutes, the hourly bucket has reached its limit, and logs are dropped. At this point the rate is still 2 logs/s for the remaining 30 minutes.
- Because the last hour finished on 2 logs/s, which is higher that the
1 log/sreset, all logs are still dropped when starting the second hour. The bucket limit is left untouched since nothing is being emitted. - Now, at 2 hours and 30 minutes, the log rate halves to
1 log/s, which is equal to the reset rate. Logs are emitted again, counting toward the bucket limit as normal. Allowing up to 3600 logs for the last 30 minutes of the second hour.
Because this could allow for some instability if the log rate hovers around the
group_bucket_limit/group_bucket_period_s rate, it is possible to set a
different reset rate.
Note that a value of 0 effectively means the plugin will drops logs forever
after a single breach of the limit until the next restart of fluentd.
A value of -1 disables the feature.
Default: 0.1 (10 seconds).
When a group reaches its limit and as long as it is not reset, a warning message with the current log rate for the group is logged at this frequency.
Copyright © 2018 (Rubrik Inc.)
Apache License, Version 2.0