Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,17 @@
## Administration

* [Configuring Fluent Bit](administration/configuring-fluent-bit/README.md)
* [Format and Schema](administration/configuring-fluent-bit/format-schema.md)
* [Configuration File](administration/configuring-fluent-bit/configuration-file.md)
* [Variables](administration/configuring-fluent-bit/variables.md)
* [Commands](administration/configuring-fluent-bit/commands.md)
* [Upstream Servers](administration/configuring-fluent-bit/upstream-servers.md)
* [Classic mode](administration/configuring-fluent-bit/classic-mode/README.md)
* [Format and Schema](administration/configuring-fluent-bit/classic-mode/format-schema.md)
* [Configuration File](administration/configuring-fluent-bit/classic-mode/configuration-file.md)
* [Variables](administration/configuring-fluent-bit/classic-mode/variables.md)
* [Commands](administration/configuring-fluent-bit/classic-mode/commands.md)
* [Upstream Servers](administration/configuring-fluent-bit/classic-mode/upstream-servers.md)
* [Record Accessor](administration/configuring-fluent-bit/classic-mode/record-accessor.md)
* [YAML Configuration](administration/configuring-fluent-bit/yaml/README.md)
* [Configuration File](administration/configuring-fluent-bit/yaml/configuration-file.md)
* [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md)
* [Multiline Parsing](administration/configuring-fluent-bit/multiline-parsing.md)
* [Record Accessor](administration/configuring-fluent-bit/record-accessor.md)
* [Security](administration/security.md)
* [Buffering & Storage](administration/buffering-and-storage.md)
* [Backpressure](administration/backpressure.md)
Expand Down
2 changes: 1 addition & 1 deletion administration/buffering-and-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ The known Service section configure a global environment for the storage layer,

### Service Section Configuration

The Service section refers to the section defined in the main [configuration file](configuring-fluent-bit/configuration-file.md):
The Service section refers to the section defined in the main [configuration file](configuring-fluent-bit/classic-mode/configuration-file.md):

| Key | Description | Default |
| :--- | :--- | :--- |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Fluent Bit classic mode
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ The _Service_ section defines global properties of the service, the keys availab
| log_level | Set the logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. Values are accumulative, e.g: if 'debug' is set, it will include error, warning, info and debug. Note that _trace_ mode is only available if Fluent Bit was built with the _WITH\_TRACE_ option enabled. | info |
| parsers_file | Path for a `parsers` configuration file. Multiple Parsers_File entries can be defined within the section. | |
| plugins_file | Path for a `plugins` configuration file. A _plugins_ configuration file allows to define paths for external plugins, for an example [see here](https://github.com/fluent/fluent-bit/blob/master/conf/plugins.conf). | |
| streams_file | Path for the Stream Processor configuration file. To learn more about Stream Processing configuration go [here](../../stream-processing/introduction.md). | |
| streams_file | Path for the Stream Processor configuration file. To learn more about Stream Processing configuration go [here](../../../stream-processing/introduction.md). | |
| http_server | Enable built-in HTTP Server | Off |
| http_listen | Set listening interface for HTTP Server when it's enabled | 0.0.0.0 |
| http_port | Set TCP Port for the HTTP Server | 2020 |
Expand All @@ -41,24 +41,12 @@ The _Service_ section defines global properties of the service, the keys availab

The following is an example of a _SERVICE_ section:

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python
[SERVICE]
Flush 5
Daemon off
Log_Level debug
```
{% endtab %}
{% tab title="fluent-bit.yaml" %}
```yaml
service:
flush: 5
daemon: off
log_level: debug
```
{% endtab %}
{% endtabs %}

## Input <a href="config_input" id="config_input"></a>

Expand All @@ -75,23 +63,11 @@ The _Name_ is mandatory and it let Fluent Bit know which input plugin should be

The following is an example of an _INPUT_ section:

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python
[INPUT]
Name cpu
Tag my_cpu
```
{% endtab %}
{% tab title="fluent-bit.yaml" %}
```yaml
pipeline:
inputs:
cpu:
tag: my_cpu
```
{% endtab %}
{% endtabs %}

## Filter <a href="config_filter" id="config_filter"></a>

Expand All @@ -109,25 +85,12 @@ The _Name_ is mandatory and it let Fluent Bit know which filter plugin should be

The following is an example of an _FILTER_ section:

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python
[FILTER]
Name grep
Match *
Regex log aa
```
{% endtab %}
{% tab title="fluent-bit.yaml" %}
```yaml
pipeline:
filters:
regex:
match: *
regex: log aa
```
{% endtab %}
{% endtabs %}

## Output <a href="config_output" id="config_output"></a>

Expand All @@ -143,30 +106,16 @@ The _OUTPUT_ section specify a destination that certain records should follow af

The following is an example of an _OUTPUT_ section:

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python
[OUTPUT]
Name stdout
Match my*cpu
```
{% endtab %}
{% tab title="fluent-bit.yaml" %}
```yaml
pipeline:
outputs:
stdout:
match: my*cpu
```
{% endtab %}
{% endtabs %}

### Example: collecting CPU metrics

The following configuration file example demonstrates how to collect CPU metrics and flush the results every five seconds to the standard output:

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python
[SERVICE]
Flush 5
Expand All @@ -181,30 +130,12 @@ The following configuration file example demonstrates how to collect CPU metrics
Name stdout
Match my*cpu
```
{% endtab %}
{% tab title="fluent-bit.yaml" %}
```yaml
service:
flush: 5
daemon: off
log_level: debug

pipeline:
inputs:
cpu:
tag: my_cpu
outputs:
stdout:
match: my*cpu
```
{% endtab %}
{% endtabs %}

## Visualize <a href="config_include_file" id="config_include_file"></a>

You can also visualize Fluent Bit INPUT, FILTER, and OUTPUT configuration via [https://cloud.calyptia.com](https://cloud.calyptia.com/visualizer)

![](../../.gitbook/assets/image.png)
![](../../../.gitbook/assets/image.png)

## Include File <a href="config_include_file" id="config_include_file"></a>

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Formats and Schema

Fluent Bit might optionally use a configuration file to define how the service will behave. Starting in Fluent Bit 1.9, it supports yaml format plus the usual plain text.
Fluent Bit might optionally use a configuration file to define how the service will behave.

Before proceeding we need to understand how the configuration schema works.

Expand All @@ -12,26 +12,12 @@ The schema is defined by three concepts:

A simple example of a configuration file is as follows:

{% tabs %}
{% tab title="fluent-bit.conf" %}

```python
[SERVICE]
# This is a commented line
Daemon off
log_level debug
```
{% endtab %}

{% tab title="fluent-bit.yaml" %}
```yaml
service:
# This is a commented line
daemon: off
log_level: debug
```
{% endtab %}
{% endtabs %}

## Sections <a id="sections"></a>

Expand All @@ -57,8 +43,6 @@ Also commented lines are set prefixing the **\#** character, those lines are not

Fluent Bit configuration files are based in a strict **Indented Mode**, that means that each configuration file must follow the same pattern of alignment from left to right when writing text. By default an indentation level of four spaces from left to right is suggested. Example:

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python
[FIRST_SECTION]
# This is a commented line
Expand All @@ -69,20 +53,6 @@ Fluent Bit configuration files are based in a strict **Indented Mode**, that mea
[SECOND_SECTION]
KeyN 3.14
```
{% endtab %}
{% tab title="fluent-bit.yaml" %}
```yaml
first_section:
# This is a commented line
Key1: some value
Key2: another value
# more comments

second_section:
KeyN: 3.14
```
{% endtab %}
{% endtabs %}

As you can see there are two sections with multiple entries and comments, note also that empty lines are allowed and they do not need to be indented.

Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ If the accessor key does not exist in the record like the last example `$labels[

## Usage Example

The feature is enabled on a per plugin basis, not all plugins enable this feature. As an example consider a configuration that aims to filter records using [grep](../../pipeline/filters/grep.md) that only matches where labels have a color blue:
The feature is enabled on a per plugin basis, not all plugins enable this feature. As an example consider a configuration that aims to filter records using [grep](../../../pipeline/filters/grep.md) that only matches where labels have a color blue:

```text
[SERVICE]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Upstream Servers

It's common that Fluent Bit [output plugins](../../pipeline/outputs/) aims to connect to external services to deliver the logs over the network, this is the case of [HTTP](../../pipeline/outputs/http.md), [Elasticsearch](../../pipeline/outputs/elasticsearch.md) and [Forward](../../pipeline/outputs/forward.md) within others. Being able to connect to one node \(host\) is normal and enough for more of the use cases, but there are other scenarios where balancing across different nodes is required. The _Upstream_ feature provides such capability.
It's common that Fluent Bit [output plugins](../../pipeline/outputs/) aims to connect to external services to deliver the logs over the network, this is the case of [HTTP](../../../pipeline/outputs/http.md), [Elasticsearch](../../../pipeline/outputs/elasticsearch.md) and [Forward](../../../pipeline/outputs/forward.md) within others. Being able to connect to one node \(host\) is normal and enough for more of the use cases, but there are other scenarios where balancing across different nodes is required. The _Upstream_ feature provides such capability.

An _Upstream_ defines a set of nodes that will be targeted by an output plugin, by the nature of the implementation an output plugin **must** support the _Upstream_ feature. The following plugin\(s\) have _Upstream_ support:

* [Forward](../../pipeline/outputs/forward.md)
* [Forward](../../../pipeline/outputs/forward.md)

The current balancing mode implemented is _round-robin_.

Expand All @@ -27,7 +27,7 @@ A _Node_ might contain additional configuration keys required by the plugin, on

In addition to the properties defined in the table above, the network operations against a defined node can optionally be done through the use of TLS for further encryption and certificates use.

The TLS options available are described in the [TLS/SSL](../security.md) section and can be added to the any _Node_ section.
The TLS options available are described in the [TLS/SSL](../../security.md) section and can be added to the any _Node_ section.

### Configuration File Example

Expand Down
7 changes: 7 additions & 0 deletions administration/configuring-fluent-bit/yaml/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Fluent Bit YAML configuration

Since version 1.9, Fluent Bit supports the use of YAML for configuration.

{% hint style="info" %}
YAML configuration is tech preview so not recommended for production.
{% endhint %}
Loading