Skip to content

Commit def2c25

Browse files
esmerelcnorris-cs
andauthored
Apply suggestions from code review
Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: esmerel <6818907+esmerel@users.noreply.github.com>
1 parent c31dc9a commit def2c25

File tree

1 file changed

+17
-17
lines changed

1 file changed

+17
-17
lines changed

administration/backpressure.md

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,15 @@
55
It's possible for logs or data to be ingested or created faster than the ability to
66
flush it to some destinations. A common scenario is when reading from big log files,
77
especially with a large backlog, and dispatching the logs to a backend over the
8-
network, which takes time to respond. This generates backpressure leading to high
8+
network, which takes time to respond. This generates _backpressure_, leading to high
99
memory consumption in the service.
1010

1111
To avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts
1212
the amount of data an input plugin can ingest. Restriction is done through the
1313
configuration parameters `Mem_Buf_Limit` and `storage.Max_Chunks_Up`.
1414

1515
As described in the [Buffering](../concepts/buffering.md) concepts section, Fluent
16-
Bit offers two modes for data handling: in-memory only (default) and in-memory +
16+
Bit offers two modes for data handling: in-memory only (default) and in-memory and
1717
filesystem (optional).
1818

1919
The default `storage.type memory` buffer can be restricted with `Mem_Buf_Limit`. If
@@ -23,19 +23,19 @@ input pauses and Fluent Bit
2323
[emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1334)
2424
a `[warn] [input] {input name or alias} paused (mem buf overlimit)` log message.
2525

26-
Depending on the input plugin in use, this might lead to discard incoming data (for
27-
example, TCP input plugin). The tail plugin can handle pause without data loss; it
28-
stores its current file offset and resumes reading later. When buffer memory is
29-
available, the input resumes accepting logs. Fluent Bit
26+
Depending on the input plugin in use, this might cause incoming data to be discarded
27+
(for example, TCP input plugin). The tail plugin can handle pauses without data
28+
ingloss, storing its current file offset and resuming reading later. When buffer
29+
memory is available, the input resumes accepting logs. Fluent Bit
3030
[emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1277)
3131
a `[info] [input] {input name or alias} resume (mem buf overlimit)` message.
3232

3333
Mitigate the risk of data loss by configuring secondary storage on the filesystem
3434
using the `storage.type` of `filesystem` (as described in [Buffering &
3535
Storage](buffering-and-storage.md)). Initially, logs will be buffered to both memory
3636
and the filesystem. When the `storage.max_chunks_up` limit is reached, all new data
37-
will be stored safely in the filesystem. Fluent Bit stops queueing new data in memory
38-
and will only buffers to the filesystem. When `storage.type filesystem` is set, the
37+
will be stored in the filesystem. Fluent Bit stops queueing new data in memory and
38+
buffers only to the filesystem. When `storage.type filesystem` is set, the
3939
`Mem_Buf_Limit` setting no longer has any effect. Instead, the `[SERVICE]` level
4040
`storage.max_chunks_up` setting controls the size of the memory buffer.
4141

@@ -47,15 +47,15 @@ disabled by default and can be applied to all input plugins.
4747
As an example situation:
4848

4949
- `Mem_Buf_Limit` is set to `1MB`.
50-
- The input plugin tries to append 700KB.
50+
- The input plugin tries to append 700&nbsp;KB.
5151
- The engine routes the data to an output plugin.
5252
- The output plugin backend (HTTP Server) is down.
5353
- Engine scheduler retries the flush after 10 seconds.
5454
- The input plugin tries to append 500KB.
5555

56-
In this situation, the engine allows appending those 500KB of data into the memory,
57-
with a total of 1.2 MB of data buffered. The limit is permissive and will
58-
allow a single write past the limit. Once the limit is exceeded, the following
56+
In this situation, the engine allows appending those 500&nbsp;KB of data into the memory,
57+
with a total of 1.2&nbsp;MB of data buffered. The limit is permissive and will
58+
allow a single write past the limit. When the limit is exceeded, the following
5959
actions are taken:
6060

6161
- Block local buffers for the input plugin (can't append more data).
@@ -65,13 +65,13 @@ The engine protects itself and won't append more data coming from the input plug
6565
question. It's the responsibility of the plugin to keep state and decide what to do
6666
in a `paused` state.
6767

68-
In a few seconds, if the scheduler was able to flush the initial 700KB of data or it
68+
In a few seconds, if the scheduler was able to flush the initial 700&nbsp;KB of data or it
6969
has given up after retrying, that amount of memory is released and the following
7070
actions occur:
7171

72-
- Upon data buffer release (700KB), the internal counters get updated.
73-
- Counters now are set at 500KB.
74-
- Since 500KB is &lt; 1 MB it checks the input plugin state.
72+
- Upon data buffer release (700&nbsp;KB), the internal counters get updated.
73+
- Counters now are set at 500&nbsp;KB.
74+
- Because 500&nbsp;KB isless than 1&nbsp;MB, it checks the input plugin state.
7575
- If the plugin is paused, it invokes a `resume` callback.
7676
- The input plugin can continue appending more data.
7777

@@ -85,7 +85,7 @@ The setting behaves similar to the `Mem_Buf_Limit` scenario when the non-default
8585
`storage.pause_on_chunks_overlimit` is enabled.
8686

8787
When (default) `storage.pause_on_chunks_overlimit` is disabled, the input won't pause
88-
when the memory limit is reached. Instead, it switches to only buffering logs in
88+
when the memory limit is reached. Instead, it switches to buffering logs only in
8989
the filesystem. Limit the disk spaced used for filesystem buffering with
9090
`storage.total_limit_size`.
9191

0 commit comments

Comments
 (0)