5
5
It's possible for logs or data to be ingested or created faster than the ability to
6
6
flush it to some destinations. A common scenario is when reading from big log files,
7
7
especially with a large backlog, and dispatching the logs to a backend over the
8
- network, which takes time to respond. This generates backpressure leading to high
8
+ network, which takes time to respond. This generates _ backpressure _ , leading to high
9
9
memory consumption in the service.
10
10
11
11
To avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts
12
12
the amount of data an input plugin can ingest. Restriction is done through the
13
13
configuration parameters ` Mem_Buf_Limit ` and ` storage.Max_Chunks_Up ` .
14
14
15
15
As described in the [ Buffering] ( ../concepts/buffering.md ) concepts section, Fluent
16
- Bit offers two modes for data handling: in-memory only (default) and in-memory +
16
+ Bit offers two modes for data handling: in-memory only (default) and in-memory and
17
17
filesystem (optional).
18
18
19
19
The default ` storage.type memory ` buffer can be restricted with ` Mem_Buf_Limit ` . If
@@ -23,19 +23,19 @@ input pauses and Fluent Bit
23
23
[ emits] ( https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1334 )
24
24
a ` [warn] [input] {input name or alias} paused (mem buf overlimit) ` log message.
25
25
26
- Depending on the input plugin in use, this might lead to discard incoming data (for
27
- example, TCP input plugin). The tail plugin can handle pause without data loss; it
28
- stores its current file offset and resumes reading later. When buffer memory is
29
- available, the input resumes accepting logs. Fluent Bit
26
+ Depending on the input plugin in use, this might cause incoming data to be discarded
27
+ (for example, TCP input plugin). The tail plugin can handle pauses without data
28
+ ingloss, storing its current file offset and resuming reading later. When buffer
29
+ memory is available, the input resumes accepting logs. Fluent Bit
30
30
[ emits] ( https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1277 )
31
31
a ` [info] [input] {input name or alias} resume (mem buf overlimit) ` message.
32
32
33
33
Mitigate the risk of data loss by configuring secondary storage on the filesystem
34
34
using the ` storage.type ` of ` filesystem ` (as described in [ Buffering &
35
35
Storage] ( buffering-and-storage.md ) ). Initially, logs will be buffered to both memory
36
36
and the filesystem. When the ` storage.max_chunks_up ` limit is reached, all new data
37
- will be stored safely in the filesystem. Fluent Bit stops queueing new data in memory
38
- and will only buffers to the filesystem. When ` storage.type filesystem ` is set, the
37
+ will be stored in the filesystem. Fluent Bit stops queueing new data in memory and
38
+ buffers only to the filesystem. When ` storage.type filesystem ` is set, the
39
39
` Mem_Buf_Limit ` setting no longer has any effect. Instead, the ` [SERVICE] ` level
40
40
` storage.max_chunks_up ` setting controls the size of the memory buffer.
41
41
@@ -47,15 +47,15 @@ disabled by default and can be applied to all input plugins.
47
47
As an example situation:
48
48
49
49
- ` Mem_Buf_Limit ` is set to ` 1MB ` .
50
- - The input plugin tries to append 700KB .
50
+ - The input plugin tries to append 700 & nbsp ; KB .
51
51
- The engine routes the data to an output plugin.
52
52
- The output plugin backend (HTTP Server) is down.
53
53
- Engine scheduler retries the flush after 10 seconds.
54
54
- The input plugin tries to append 500KB.
55
55
56
- In this situation, the engine allows appending those 500KB of data into the memory,
57
- with a total of 1.2 MB of data buffered. The limit is permissive and will
58
- allow a single write past the limit. Once the limit is exceeded, the following
56
+ In this situation, the engine allows appending those 500 & nbsp ; KB of data into the memory,
57
+ with a total of 1.2& nbsp ; MB of data buffered. The limit is permissive and will
58
+ allow a single write past the limit. When the limit is exceeded, the following
59
59
actions are taken:
60
60
61
61
- Block local buffers for the input plugin (can't append more data).
@@ -65,13 +65,13 @@ The engine protects itself and won't append more data coming from the input plug
65
65
question. It's the responsibility of the plugin to keep state and decide what to do
66
66
in a ` paused ` state.
67
67
68
- In a few seconds, if the scheduler was able to flush the initial 700KB of data or it
68
+ In a few seconds, if the scheduler was able to flush the initial 700 & nbsp ; KB of data or it
69
69
has given up after retrying, that amount of memory is released and the following
70
70
actions occur:
71
71
72
- - Upon data buffer release (700KB ), the internal counters get updated.
73
- - Counters now are set at 500KB .
74
- - Since 500KB is & lt ; 1 MB it checks the input plugin state.
72
+ - Upon data buffer release (700 & nbsp ; KB ), the internal counters get updated.
73
+ - Counters now are set at 500 & nbsp ; KB .
74
+ - Because 500 & nbsp ; KB isless than 1 & nbsp ; MB, it checks the input plugin state.
75
75
- If the plugin is paused, it invokes a ` resume ` callback.
76
76
- The input plugin can continue appending more data.
77
77
@@ -85,7 +85,7 @@ The setting behaves similar to the `Mem_Buf_Limit` scenario when the non-default
85
85
` storage.pause_on_chunks_overlimit ` is enabled.
86
86
87
87
When (default) ` storage.pause_on_chunks_overlimit ` is disabled, the input won't pause
88
- when the memory limit is reached. Instead, it switches to only buffering logs in
88
+ when the memory limit is reached. Instead, it switches to buffering logs only in
89
89
the filesystem. Limit the disk spaced used for filesystem buffering with
90
90
` storage.total_limit_size ` .
91
91
0 commit comments