Skip to content

Commit 103d4fe

Browse files
edsipergitbook-bot
edsiper
authored and
gitbook-bot
committed
GitBook: [1.4] 119 pages modified
1 parent a60effd commit 103d4fe

File tree

9 files changed

+323
-43
lines changed

9 files changed

+323
-43
lines changed

‎SUMMARY.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,18 +44,19 @@
4444
## Administration
4545

4646
* [Configuring Fluent Bit](administration/configuring-fluent-bit/README.md)
47-
* [Files Schema / Structure](administration/configuring-fluent-bit/files-schema-structure.md)
47+
* [Format and Schema](administration/configuring-fluent-bit/format-schema.md)
48+
* [Configuration File](administration/configuring-fluent-bit/configuration-file.md)
4849
* [Variables](administration/configuring-fluent-bit/variables.md)
4950
* [Commands](administration/configuring-fluent-bit/commands.md)
50-
* [Configuration File](administration/configuring-fluent-bit/configuration-file.md)
5151
* [Upstream Servers](administration/configuring-fluent-bit/upstream-servers.md)
5252
* [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md)
5353
* [Security](administration/security.md)
5454
* [Buffering & Storage](administration/buffering-and-storage.md)
5555
* [Backpressure](administration/backpressure.md)
56+
* [Scheduling and Retries](administration/scheduling-and-retries.md)
5657
* [Memory Management](administration/memory-management.md)
5758
* [Monitoring](administration/monitoring.md)
58-
* [Scheduling and Retries](administration/scheduling-and-retries.md)
59+
* [Dump Internals / Signal](administration/dump-internals-signal.md)
5960

6061
## Data Pipeline <a id="pipeline"></a>
6162

‎administration/backpressure.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,16 @@ In certain environments is common to see that logs or data being ingested is fas
44

55
In order to avoid backpressure, Fluent Bit implements a mechanism in the engine that restrict the amount of data than an input plugin can ingest, this is done through the configuration parameter **Mem\_Buf\_Limit**.
66

7+
{% hint style="info" %}
8+
As described in the [Buffering](../concepts/buffering.md) concepts section, Fluent Bit offers an hybrid mode for data handling: in-memory and filesystem \(optional\).
9+
10+
In `memory` is always available and can be restricted with **Mem\_Buf\_Limit**. If your plugin gets restricted because of the configuration and you are under a backpressure scenario, you won't be able to ingest more data until the data chunks that are in memory can flushed.
11+
12+
Depending of the input plugin type in use, this might lead to discard incoming data \(e.g: TCP input plugin\), but you can rely on the secondary filesystem buffering to be safe.
13+
14+
If in addition to Mem\_Buf\_Limit the input plugin defined a `storage.type` of `filesystem` \(as described in [Buffering & Storage](buffering-and-storage.md)\), when the limit is reached, all the new data will be stored safety in the file system.
15+
{% endhint %}
16+
717
## Mem\_Buf\_Limit
818

919
This option is disabled by default and can be applied to all input plugins. Let's explain it behavior using the following scenario:

‎administration/buffering-and-storage.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ The known Service section configure a global environment for the storage layer,
1717

1818
### Service Section Configuration
1919

20+
The Service section refers to the section defined in the main [configuration file](configuring-fluent-bit/configuration-file.md):
21+
2022
| Key | Description | Default |
2123
| :--- | :--- | :--- |
2224
| storage.path | Set an optional location in the file system to store streams and chunks of data. If this parameter is not set, Input plugins can only use in-memory buffering. | |

‎administration/configuring-fluent-bit/configuration-file.md

Lines changed: 103 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,117 @@
1-
# Configuration File
1+
---
2+
description: This page describes the main configuration file used by Fluent Bit
3+
---
24

3-
There are some cases where using the command line to start Fluent Bit is not ideal. When running Fluent Bit as a service, a configuration file is preferred.
5+
# Configuration File
46

5-
Fluent Bit allows to use one configuration file which works at a global scope and uses the [schema](https://github.com/fluent/fluent-bit-docs/tree/ad9d80e5490bd5d79c86955c5689db1cb4cf89db/configuration/configuration_schema.md) defined previously.
7+
One of the ways to configure Fluent Bit is using a main configuration file. Fluent Bit allows to use one configuration file which works at a global scope and uses the [Format and Schema](format-schema.md) defined previously.
68

7-
The configuration file supports four types of sections:
9+
The main configuration file supports four types of sections:
810

9-
* [Service](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/file.md#config_section)
10-
* [Input](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/file.md#config_input)
11-
* [Filter](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/file.md#config_filter)
12-
* [Output](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/file.md#config_output)
11+
* Service
12+
* Input
13+
* Filter
14+
* Output
1315

14-
In addition there is an additional feature to include external files:
16+
In addition, it's also possible to split the main configuration file in multiple files using the feature to include external files:
1517

16-
* [Include File](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/file.md#config_include_file)
18+
* Include File
1719

1820
## Service <a id="config_section"></a>
1921

2022
The _Service_ section defines global properties of the service, the keys available as of this version are described in the following table:
2123

22-
| Key | Description | Default Value |
23-
| :--- | :--- | :--- |
24-
| Flush | Set the flush time in seconds. Everytime it timeouts, the engine will flush the records to the output plugin. | 5 |
25-
| Daemon | Boolean value to set if Fluent Bit should run as a Daemon \(background\) or not. Allowed values are: yes, no, on and off. | Off |
26-
| Log\_File | Absolute path for an optional log file. | |
27-
| Log\_Level | Set the logging verbosity level. Allowed values are: error, info, debug and trace. Values are accumulative, e.g: if 'debug' is set, it will include error, info and debug. Note that _trace_ mode is only available if Fluent Bit was built with the _WITH\_TRACE_ option enabled. | info |
28-
| Parsers\_File | Path for a _parsers_ configuration file. Multiple Parsers\_File entries can be used. | |
29-
| Plugins\_File | Path for a _plugins_ configuration file. A _plugins_ configuration file allows to define paths for external plugins, for an example [see here](https://github.com/fluent/fluent-bit/blob/master/conf/plugins.conf). | |
30-
| Streams\_File | Path for the Stream Processor configuration file. For details about the format of SP configuration file [see here](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/stream_processor.md). | |
31-
| HTTP\_Server | Enable built-in HTTP Server | Off |
32-
| HTTP\_Listen | Set listening interface for HTTP Server when it's enabled | 0.0.0.0 |
33-
| HTTP\_Port | Set TCP Port for the HTTP Server | 2020 |
34-
| Coro\_Stack\_Size | Set the coroutines stack size in bytes. The value must be greater than the page size of the running system. Don't set too small value \(say 4096\), or coroutine threads can overrun the stack buffer. | 24576 |
35-
36-
### Example
24+
<table>
25+
<thead>
26+
<tr>
27+
<th style="text-align:left">Key</th>
28+
<th style="text-align:left">Description</th>
29+
<th style="text-align:left">Default Value</th>
30+
</tr>
31+
</thead>
32+
<tbody>
33+
<tr>
34+
<td style="text-align:left">Flush</td>
35+
<td style="text-align:left">Set the flush time in <code>seconds.nanoseconds</code>. The engine loop
36+
uses a Flush timeout to define when is required to flush the records ingested
37+
by input plugins through the defined output plugins.</td>
38+
<td style="text-align:left">5</td>
39+
</tr>
40+
<tr>
41+
<td style="text-align:left">Daemon</td>
42+
<td style="text-align:left">Boolean value to set if Fluent Bit should run as a Daemon (background)
43+
or not. Allowed values are: yes, no, on and off.
44+
<br />
45+
<br />note: If you are using a Systemd based unit as the one we provide in our
46+
packages, do not turn on this option.</td>
47+
<td style="text-align:left">Off</td>
48+
</tr>
49+
<tr>
50+
<td style="text-align:left">Log_File</td>
51+
<td style="text-align:left">Absolute path for an optional log file. By default all logs are redirected
52+
to the standard output interface (stdout).</td>
53+
<td style="text-align:left"></td>
54+
</tr>
55+
<tr>
56+
<td style="text-align:left">Log_Level</td>
57+
<td style="text-align:left">Set the logging verbosity level. Allowed values are: error, warning, info,
58+
debug and trace. Values are accumulative, e.g: if &apos;debug&apos; is
59+
set, it will include error, warning, info and debug.
60+
<br />
61+
<br />Note that <em>trace</em> mode is only available if Fluent Bit was built
62+
with the <em>WITH_TRACE</em> option enabled.</td>
63+
<td style="text-align:left">info</td>
64+
</tr>
65+
<tr>
66+
<td style="text-align:left">Parsers_File</td>
67+
<td style="text-align:left">Path for a <code>parsers</code> configuration file. Multiple Parsers_File
68+
entries can be defined within the section.</td>
69+
<td style="text-align:left"></td>
70+
</tr>
71+
<tr>
72+
<td style="text-align:left">Plugins_File</td>
73+
<td style="text-align:left">Path for a <code>plugins</code> configuration file. A <em>plugins</em> configuration
74+
file allows to define paths for external plugins, for an example <a href="https://github.com/fluent/fluent-bit/blob/master/conf/plugins.conf">see here</a>.</td>
75+
<td
76+
style="text-align:left"></td>
77+
</tr>
78+
<tr>
79+
<td style="text-align:left">Streams_File</td>
80+
<td style="text-align:left">Path for the Stream Processor configuration file. To learn more about
81+
Stream Processing configuration go <a href="../../stream-processing/stream-processing.md">here</a>.</td>
82+
<td
83+
style="text-align:left"></td>
84+
</tr>
85+
<tr>
86+
<td style="text-align:left">HTTP_Server</td>
87+
<td style="text-align:left">Enable built-in HTTP Server</td>
88+
<td style="text-align:left">Off</td>
89+
</tr>
90+
<tr>
91+
<td style="text-align:left">HTTP_Listen</td>
92+
<td style="text-align:left">Set listening interface for HTTP Server when it&apos;s enabled</td>
93+
<td
94+
style="text-align:left">0.0.0.0</td>
95+
</tr>
96+
<tr>
97+
<td style="text-align:left">HTTP_Port</td>
98+
<td style="text-align:left">Set TCP Port for the HTTP Server</td>
99+
<td style="text-align:left">2020</td>
100+
</tr>
101+
<tr>
102+
<td style="text-align:left">Coro_Stack_Size</td>
103+
<td style="text-align:left">
104+
<p>Set the coroutines stack size in bytes. The value must be greater than
105+
the page size of the running system. Don&apos;t set too small value (say
106+
4096), or coroutine threads can overrun the stack buffer.</p>
107+
<p></p>
108+
<p>Do not change the default value of this parameter unless you know what
109+
you are doing.</p>
110+
</td>
111+
<td style="text-align:left">24576</td>
112+
</tr>
113+
</tbody>
114+
</table>### Example
37115

38116
The following is an example of a _SERVICE_ section:
39117

‎administration/configuring-fluent-bit/format-schema.md

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Format and Schema
2+
3+
Fluent Bit might optionally use a configuration file to define how the service will behave, and before proceeding we need to understand how the configuration schema works.
4+
5+
The schema is defined by three concepts:
6+
7+
* Sections
8+
* Entries: Key/Value
9+
* Indented Configuration Mode
10+
11+
A simple example of a configuration file is as follows:
12+
13+
```python
14+
[SERVICE]
15+
# This is a commented line
16+
Daemon off
17+
log_level debug
18+
```
19+
20+
## Sections <a id="sections"></a>
21+
22+
A section is defined by a name or title inside brackets. Looking at the example above, a Service section has been set using **\[SERVICE\]** definition. Section rules:
23+
24+
* All section content must be indented \(4 spaces ideally\).
25+
* Multiple sections can exist on the same file.
26+
* A section is expected to have comments and entries, it cannot be empty.
27+
* Any commented line under a section, must be indented too.
28+
29+
## Entries: Key/Value <a id="entries_kv"></a>
30+
31+
A section may contain **Entries**, an entry is defined by a line of text that contains a **Key** and a **Value**, using the above example, the `[SERVICE]` section contains two entries, one is the key **Daemon** with value **off** and the other is the key **Log\_Level** with the value **debug**. Entries rules:
32+
33+
* An entry is defined by a key and a value.
34+
* A key must be indented.
35+
* A key must contain a value which ends in the breakline.
36+
* Multiple keys with the same name can exist.
37+
38+
Also commented lines are set prefixing the **\#** character, those lines are not processed but they must be indented too.
39+
40+
## Indented Configuration Mode <a id="indented_mode"></a>
41+
42+
Fluent Bit configuration files are based in a strict **Indented Mode**, that means that each configuration file must follow the same pattern of alignment from left to right when writing text. By default an indentation level of four spaces from left to right is suggested. Example:
43+
44+
```python
45+
[FIRST_SECTION]
46+
# This is a commented line
47+
Key1 some value
48+
Key2 another value
49+
# more comments
50+
51+
[SECOND_SECTION]
52+
KeyN 3.14
53+
```
54+
55+
As you can see there are two sections with multiple entries and comments, note also that empty lines are allowed and they do not need to be indented.
56+

‎administration/configuring-fluent-bit/variables.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,10 +41,13 @@ Run Fluent Bit with the recently created configuration file:
4141

4242
```text
4343
$ bin/fluent-bit -c fluent-bit.conf
44-
Fluent-Bit v0.11.0
45-
Copyright (C) Treasure Data
44+
Fluent Bit v1.4.0
45+
* Copyright (C) 2019-2020 The Fluent Bit Authors
46+
* Copyright (C) 2015-2018 Treasure Data
47+
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
48+
* https://fluentbit.io
4649
47-
[2017/04/03 12:25:25] [ info] [engine] started
50+
[2020/03/03 12:25:25] [ info] [engine] started
4851
[0] cpu.local: [1491243925, {"cpu_p"=>1.750000, "user_p"=>1.750000, "system_p"=>0.000000, "cpu0.p_cpu"=>3.000000, "cpu0.p_user"=>2.000000, "cpu0.p_system"=>1.000000, "cpu1.p_cpu"=>0.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>0.000000, "cpu2.p_cpu"=>4.000000, "cpu2.p_user"=>4.000000, "cpu2.p_system"=>0.000000, "cpu3.p_cpu"=>1.000000, "cpu3.p_user"=>1.000000, "cpu3.p_system"=>0.000000}]
4952
```
5053

‎administration/dump-internals-signal.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Dump Internals / Signal
2+
3+
When the service is running we can export [metrics](monitoring.md) to see the overall status of the data flow of the service. But there are other use cases where we would like to know the current status of the internals of the service, specifically to answer questions like _what's the current status of the internal buffers ?_ , the Dump Internals feature is the answer.
4+
5+
Fluent Bit v1.4 introduces the Dump Internals feature that can be triggered easily from the command line triggering the `CONT` Unix signal.
6+
7+
## Usage
8+
9+
Run the following `kill` command to signal Fluent Bit:
10+
11+
```text
12+
kill -CONT `pidof fluent-bit`
13+
```
14+
15+
> The command `pidof` aims to lookup the Process ID of Fluent Bit. You can replace the
16+
17+
Fluent But will dump the following information to the standard output interface \(stdout\):
18+
19+
```text
20+
[engine] caught signal (SIGCONT)
21+
[2020/03/23 17:39:02] Fluent Bit Dump
22+
23+
===== Input =====
24+
syslog_debug (syslog)
25+
│
26+
├─ status
27+
│ └─ overlimit : no
28+
│ ├─ mem size : 60.8M (63752145 bytes)
29+
│ └─ mem limit : 61.0M (64000000 bytes)
30+
│
31+
├─ tasks
32+
│ ├─ total tasks : 92
33+
│ ├─ new : 0
34+
│ ├─ running : 92
35+
│ └─ size : 171.1M (179391504 bytes)
36+
│
37+
└─ chunks
38+
└─ total chunks : 92
39+
├─ up chunks : 35
40+
├─ down chunks: 57
41+
└─ busy chunks: 92
42+
├─ size : 60.8M (63752145 bytes)
43+
└─ size err: 0
44+
45+
===== Storage Layer =====
46+
total chunks : 92
47+
├─ mem chunks : 0
48+
└─ fs chunks : 92
49+
├─ up : 35
50+
└─ down : 57
51+
```
52+
53+
## Input Plugins Dump
54+
55+
The dump provides insights for every input instance configured.
56+
57+
### Status
58+
59+
Overall ingestion status of the plugin.
60+
61+
| Entry | Sub-entry | Description |
62+
| :--- | :--- | :--- |
63+
| overlimit | | If the plugin has been configured with [Mem\_Buf\_Limit](backpressure.md), this entry will report if the plugin is over the limit or not at the moment of the dump. If it is overlimit, it will print `yes`, otherwise `no`. |
64+
| | mem\_size | Current memory size in use by the input plugin in-memory. |
65+
| | mem\_limit | Limit set by Mem\_Buf\_Limit. |
66+
67+
### Tasks
68+
69+
When an input plugin ingest data into the engine, a Chunk is created. A Chunk can contains multiple records. Upon flush time, the engine creates a Task that contains the routes for the Chunk associated in question.
70+
71+
The Task dump describes the tasks associated to the input plugin:
72+
73+
| Entry | Description |
74+
| :--- | :--- |
75+
| total\_tasks | Total number of active tasks associated to data generated by the input plugin. |
76+
| new | Number of tasks not assigned yet to an output plugin. Tasks are in `new` status for a very short period of time \(most of the time this value is very low or zero\). |
77+
| running | Number of active tasks being processed by output plugins. |
78+
| size | Amount of memory used by the Chunks being processed \(Total chunks size\). |
79+
80+
### Chunks
81+
82+
The Chunks dump tells more details about all the chunks that the input plugin has generated and are still being processed.
83+
84+
Depending of the buffering strategy and limits imposed by configuration, some Chunks might be `up` \(in memory\) or `down` \(filesystem\).
85+
86+
| Entry | Sub-entry | Description |
87+
| :--- | :--- | :--- |
88+
| total\_chunks | | Total number of Chunks generated by the input plugin that are still being processed by the engine. |
89+
| up\_chunks | | Total number of Chunks that are loaded in memory. |
90+
| down\_chunks | | Total number of Chunks that are stored in the filesystem but not loaded in memory yet. |
91+
| busy\_chunks | | Chunks marked as busy \(being flushed\) or locked. Busy Chunks are immutable and likely are ready to \(or being\) processed. |
92+
| | size | Amount of bytes used by the Chunk. |
93+
| | size err | Number of Chunks in an error state where it size could not be retrieved. |
94+
95+
## Storage Layer Dump
96+
97+
Fluent Bit relies on a custom storage layer interface designed for hybrid buffering. The `Storage Layer` entry contains a total summary of Chunks registered by Fluent Bit:
98+
99+
| Entry | Sub-Entry | Description |
100+
| :--- | :--- | :--- |
101+
| total chunks | | Total number of Chunks |
102+
| mem chunks | | Total number of Chunks memory-based |
103+
| fs chunks | | Total number of Chunks filesystem based |
104+
| | up | Total number of filesystem chunks up in memory |
105+
| | down | Total number of filesystem chunks down \(not loaded in memory\) |
106+

0 commit comments

Comments
 (0)