From eda6f1b1ac7979d4fa2434b91b071a5e824eb39a Mon Sep 17 00:00:00 2001 From: wangzhuzhen Date: Mon, 25 Apr 2022 18:24:53 +0800 Subject: [PATCH 1/2] throttle: support not drop messages Signed-off-by: wangzhuzhen --- pipeline/filters/throttle.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/pipeline/filters/throttle.md b/pipeline/filters/throttle.md index 9c54c256d..0a86197cb 100644 --- a/pipeline/filters/throttle.md +++ b/pipeline/filters/throttle.md @@ -12,6 +12,7 @@ The plugin supports the following configuration parameters: | Window | Integer | Amount of intervals to calculate average over. Default 5. | | Interval | String | Time interval, expressed in "sleep" format. e.g 3s, 1.5m, 0.5h etc | | Print\_Status | Bool | Whether to print status messages with current rate and the limits to information logs | +| Retain | Bool | Whether to whether or not to drop logs if rate limit is exceeded | ## Functional description @@ -64,6 +65,14 @@ will become: ``` As you can see, last pane of the window was overwritten and 1 message was dropped. +If you can accept the cost of latency for collector messages, you can retain all the logs without dropped use parameter 'Retain'. + +### Do not drop messages +The default value for 'Retain' is 'false', +For 'Retain' not set or set as 'false', if rate limit is exceeded, throttle will drop the messages. +In case that, if before fluent-bit first running, there is a input with huge messages which exceeded the throttle's (window * rate * interval), then only the first (window * rate * interval) records will be collected, the others before fluent-bit running will be dropped. + +If 'Retain' set as 'true', all messages will be collected without dropped, but at the cost of some latency for collecting all messages, which depends on the account of collected target input. ### Interval vs Window size @@ -128,6 +137,7 @@ $ bin/fluent-bit -i tail -p 'path=lines.txt' -F throttle -p 'rate=1' -m '*' -o s Rate 1000 Window 300 Interval 1s + Retain false [OUTPUT] Name stdout From f03bc8fe332c4eb3aa51841b4b47fde78d800fd8 Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Mon, 27 Oct 2025 11:52:04 -0700 Subject: [PATCH 2/2] Update pipeline/filters/throttle.md Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/filters/throttle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/throttle.md b/pipeline/filters/throttle.md index 5149790d5..65aace039 100644 --- a/pipeline/filters/throttle.md +++ b/pipeline/filters/throttle.md @@ -69,7 +69,7 @@ The last pane of the window was overwritten and one message was dropped. If you ### Do not drop messages -When `Retain` isn't set or is set to `false`, if therate limit is exceeded, throttle drops the messages. +When `Retain` isn't set or is set to `false`, if the rate limit is exceeded, throttle drops the messages. In cases when Fluent Bit first runs, and there is a input with huge messages which exceeded the throttle's (`window * rate * interval`), then only the first (`window * rate * interval`) records will be collected. Other records collected before Fluent Bit runs will be dropped.