Skip to content

Commit 0b594b7

Browse files
authored
administration | concepts Improve grammar (#792)
Fixing minor grammar mistakes in the buffering and storage section Signed-off-by: mickeypash <pashov.m@gmail.com>
1 parent f85755b commit 0b594b7

File tree

2 files changed

+23
-23
lines changed

2 files changed

+23
-23
lines changed

administration/buffering-and-storage.md

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,25 +2,25 @@
22

33
The end-goal of [Fluent Bit](https://fluentbit.io) is to collect, parse, filter and ship logs to a central place. In this workflow there are many phases and one of the critical pieces is the ability to do _buffering_ : a mechanism to place processed data into a temporary location until is ready to be shipped.
44

5-
By default when Fluent Bit process data, it uses Memory as a primary and temporary place to store the records, but there are certain scenarios where would be ideal to have a persistent buffering mechanism based in the filesystem to provide aggregation and data safety capabilities.
5+
By default when Fluent Bit processes data, it uses Memory as a primary and temporary place to store the records, but there are certain scenarios where it would be ideal to have a persistent buffering mechanism based in the filesystem to provide aggregation and data safety capabilities.
66

7-
Choosing the right configuration is critical and the behavior of the service can be conditioned based in the backpressure settings. Before to jump into the configuration properties let's understand the relationship between _Chunks_, _Memory_, _Filesystem_ and _Backpressure_.
7+
Choosing the right configuration is critical and the behavior of the service can be conditioned based in the backpressure settings. Before we jump into the configuration let's make sure we understand the relationship between _Chunks_, _Memory_, _Filesystem_ and _Backpressure_.
88

99
## Chunks, Memory, Filesystem and Backpressure
1010

1111
Understanding the chunks, buffering and backpressure concepts is critical for a proper configuration. Let's do a recap of the meaning of these concepts.
1212

1313
#### Chunks
1414

15-
When an input plugin \(source\) emit records, the engine group the records together in a _Chunk_. A Chunk size usually is around 2MB. By configuration, the engine decide where to place this Chunk, the default is that all chunks are created only in memory.
15+
When an input plugin \(source\) emits records, the engine groups the records together in a _Chunk_. A Chunk size usually is around 2MB. By configuration, the engine decides where to place this Chunk, the default is that all chunks are created only in memory.
1616

1717
#### Buffering and Memory
1818

1919
As mentioned above, the Chunks generated by the engine are placed in memory but this is configurable.
2020

21-
If memory is the only mechanism set for the input plugin, it will just store data as much as it can there \(memory\). This is the fastest mechanism with less system overhead, but if the service is not able to deliver the records fast enough because of a slow network or an unresponsive remote service, Fluent Bit memory usage will increase since it will accumulate more data than it can deliver.
21+
If memory is the only mechanism set for the input plugin, it will just store data as much as it can there \(memory\). This is the fastest mechanism with the least system overhead, but if the service is not able to deliver the records fast enough because of a slow network or an unresponsive remote service, Fluent Bit memory usage will increase since it will accumulate more data than it can deliver.
2222

23-
On a high load environment with backpressure the risks of having high memory usage is the chance to get killed by the Kernel \(OOM Killer\). A workaround for this backpressure scenario is to limit the amount of memory in records that an input plugin can register, this configuration property is called `mem_buf_limit`: if a plugin have enqueued more than `mem_buf_limit`, it won't be able to ingest more until it data can be delivered or flushed properly. On this scenario the input plugin in question is paused.
23+
In a high load environment with backpressure the risks of having high memory usage is the chance of getting killed by the Kernel \(OOM Killer\). A workaround for this backpressure scenario is to limit the amount of memory in records that an input plugin can register, this configuration property is called `mem_buf_limit`: if a plugin has enqueued more than the `mem_buf_limit`, it won't be able to ingest more until that data can be delivered or flushed properly. In this scenario the input plugin in question is paused.
2424

2525
The workaround of `mem_buf_limit` is good for certain scenarios and environments, it helps to control the memory usage of the service, but at the costs that if a file gets rotated while paused, you might lose that data since it won't be able to register new records. This can happen with any input source plugin. The goal of `mem_buf_limit` is memory control and survival of the service.
2626

@@ -30,7 +30,7 @@ For full data safety guarantee, use filesystem buffering.
3030

3131
Filesystem buffering enabled helps with backpressure and overall memory control.
3232

33-
Behind the scenes, Memory and Filesystem buffering mechanisms are **not** mutual exclusive, indeed when enabling filesystem buffering for your input plugin \(source\) you are getting the best of the two worlds: performance and data safety.
33+
Behind the scenes, Memory and Filesystem buffering mechanisms are **not** mutually exclusive, indeed when enabling filesystem buffering for your input plugin \(source\) you are getting the best of the two worlds: performance and data safety.
3434

3535
When the Filesystem buffering is enabled, the behavior of the engine is different, upon Chunk creation, it stores the content in memory but also it maps a copy on disk \(through [mmap\(2\)](https://man7.org/linux/man-pages/man2/mmap.2.html)\), this Chunk is active in memory and backed up in disk is called to be `up` which means "the chunk content is up in memory".
3636

@@ -42,11 +42,11 @@ If the input plugin has enabled `mem_buf_limit` and `storage.type` as `filesyste
4242

4343
**Limiting Filesystem space for Chunks**
4444

45-
Fluent Bit implements the concept of logical queues: a Chunk based on its Tag, can be routed to multiple destinations, so internally we keep a reference from where a Chunk was created and where it needs to go.
45+
Fluent Bit implements the concept of logical queues: based on its Tag a Chunk, can be routed to multiple destinations, so internally we keep a reference from where a Chunk was created and where it needs to go.
4646

47-
It's common to find cases that if we have multiple destinations for a Chunk, one of the destination might be slower than the other, and maybe one of the destinations is generating backpressure and not all of them. On this scenario how do we limit the amount of filesystem Chunks that we are logically queueing ?.
47+
It's common to find cases where if we have multiple destinations for a Chunk, one of the destinations might be slower than the other, or maybe one is generating backpressure and not all of them. In this scenario, how do we limit the amount of filesystem Chunks that we are logically queueing?
4848

49-
Starting from Fluent Bit v1.6, we introduced the new configuration property for output plugins called `storage.total_limit_size` which limits the number of Chunks that exists in the file system for a certain logical output destination. If one destinations reaches the `storage.total_limit_size` limit, the oldest Chunk from it queue for that logical output destination will be discarded.
49+
Starting from Fluent Bit v1.6, we introduced the new configuration property for output plugins called `storage.total_limit_size` which limits the number of Chunks that exist in the file system for a certain logical output destination. If one of destinations reaches the `storage.total_limit_size`, the oldest Chunk from its queue for that logical output destination will be discarded.
5050

5151
## Configuration
5252

@@ -56,7 +56,7 @@ The storage layer configuration takes place in three areas:
5656
* Input Section
5757
* Output Section
5858

59-
The known Service section configure a global environment for the storage layer, the Input sections defines which buffering mechanism to use and the output the limits for the logical queues.
59+
The known Service section configures a global environment for the storage layer, the Input sections define which buffering mechanism to use and the output the limits for the logical queues.
6060

6161
### Service Section Configuration
6262

@@ -68,8 +68,8 @@ The Service section refers to the section defined in the main [configuration fil
6868
| storage.sync | Configure the synchronization mode used to store the data into the file system. It can take the values _normal_ or _full_. | normal |
6969
| storage.checksum | Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. | Off |
7070
| storage.max\_chunks\_up | If the input plugin has enabled `filesystem` storage type, this property sets the maximum number of Chunks that can be `up` in memory. This helps to control memory usage. | 128 |
71-
| storage.backlog.mem\_limit | If _storage.path_ is set, Fluent Bit will look for data chunks that were not delivered and are still in the storage layer, these are called _backlog_ data. This option configure a hint of maximum value of memory to use when processing these records. | 5M |
72-
| storage.metrics | If `http_server` option has been enable in the main `[SERVICE]` section, this option registers a new endpoint where internal metrics of the storage layer can be consumed. For more details refer to the [Monitoring](monitoring.md) section. | off |
71+
| storage.backlog.mem\_limit | If _storage.path_ is set, Fluent Bit will look for data chunks that were not delivered and are still in the storage layer, these are called _backlog_ data. This option configures a hint of maximum value of memory to use when processing these records. | 5M |
72+
| storage.metrics | If `http_server` option has been enabled in the main `[SERVICE]` section, this option registers a new endpoint where internal metrics of the storage layer can be consumed. For more details refer to the [Monitoring](monitoring.md) section. | off |
7373

7474
a Service section will look like this:
7575

@@ -83,18 +83,18 @@ a Service section will look like this:
8383
storage.backlog.mem_limit 5M
8484
```
8585

86-
that configuration configure an optional buffering mechanism where it root for data is _/var/log/flb-storage/_, it will use _normal_ synchronization mode, without checksum and up to a maximum of 5MB of memory when processing backlog data.
86+
that configuration sets an optional buffering mechanism where the route to the data is _/var/log/flb-storage/_, it will use _normal_ synchronization mode, without running a checksum and up to a maximum of 5MB of memory when processing backlog data.
8787

8888
### Input Section Configuration
8989

90-
Optionally, any Input plugin can configure their storage preference, the following table describe the options available:
90+
Optionally, any Input plugin can configure their storage preference, the following table describes the options available:
9191

9292
| Key | Description | Default |
9393
| :--- | :--- | :--- |
94-
| storage.type | Specify the buffering mechanism to use. It can be _memory_ or _filesystem_. | memory |
95-
| storage.max_chunks_pause | Specify if file storage is to be paused when reaching the chunk limit. | off |
94+
| storage.type | Specifies the buffering mechanism to use. It can be _memory_ or _filesystem_. | memory |
95+
| storage.max_chunks_pause | Specifies if file storage is to be paused when reaching the chunk limit. | off |
9696

97-
The following example configure a service that offers filesystem buffering capabilities and two Input plugins being the first based in filesystem and the second with memory only.
97+
The following example configures a service that offers filesystem buffering capabilities and two Input plugins being the first based in filesystem and the second with memory only.
9898

9999
```python
100100
[SERVICE]
@@ -116,7 +116,7 @@ The following example configure a service that offers filesystem buffering capab
116116

117117
### Output Section Configuration
118118

119-
If certain chunks are filesystem storage.type based, it's possible to control the size of the logical queue for an output plugin. The following table describe the options available:
119+
If certain chunks are filesystem _storage.type_ based, it's possible to control the size of the logical queue for an output plugin. The following table describes the options available:
120120

121121
| Key | Description | Default |
122122
| :--- | :--- | :--- |
@@ -143,5 +143,5 @@ The following example create records with CPU usage samples in the filesystem an
143143
storage.total_limit_size 5M
144144
```
145145

146-
If for some reason Fluent Bit gets offline because of a network issue, it will continuing buffering CPU samples but just keeping a maximum of 5M of the newest data.
146+
If for some reason Fluent Bit gets offline because of a network issue, it will continue buffering CPU samples but just keep a maximum of 5M of the newest data.
147147

concepts/buffering.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,19 +4,19 @@ description: Performance and Data Safety
44

55
# Buffering
66

7-
When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory \(heap\) as a primary and temporal place to store the record logs before they get delivered, on this private memory area the records are processed.
7+
When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory \(heap\) as a primary and temporary place to store the record logs before they get delivered, in this private memory area the records are processed.
88

9-
Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. Buffering in memory is the fastest mechanism, but there are certain scenarios where the mechanism requires special strategies to deal with [backpressure](../administration/backpressure.md), data safety or reduce memory consumption by the service in constraint environments.
9+
Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. Buffering in memory is the fastest mechanism, but there are certain scenarios where it requires special strategies to deal with [backpressure](../administration/backpressure.md), data safety or reduce memory consumption by the service in constrained environments.
1010

1111
{% hint style="info" %}
1212
Network failures or latency on third party service is pretty common, and on scenarios where we cannot deliver data fast enough as we receive new data to process, we likely will face backpressure.
1313

1414
Our buffering strategies are designed to solve problems associated with backpressure and general delivery failures.
1515
{% endhint %}
1616

17-
Fluent Bit as buffering strategies, offers a primary buffering mechanism in **memory** and an optional secondary one using the **file system**. With this hybrid solution you can adjust to any use case safety and keep a high performance while processing your data.
17+
Fluent Bit as buffering strategies go, offers a primary buffering mechanism in **memory** and an optional secondary one using the **file system**. With this hybrid solution you can accomodate any use case safely and keep a high performance while processing your data.
1818

19-
Both mechanisms are not exclusive and when the data is ready to be processed or delivered it will be always **in memory**, while other data in the queue might be in the file system until is ready to be processed and moved up to memory.
19+
Both mechanisms are not mutally exclusive and when the data is ready to be processed or delivered it will always be **in memory**, while other data in the queue might be in the file system until is ready to be processed and moved up to memory.
2020

2121
To learn more about the buffering configuration in Fluent Bit, please jump to the [Buffering & Storage](../administration/buffering-and-storage.md) section.
2222

0 commit comments

Comments
 (0)