From 34cb44205baa4ce8690f2887909a6eec0d98f38d Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 10 Mar 2022 19:41:23 -0300 Subject: [PATCH 01/10] administration: configuration: separate classic mode and yaml mode. Signed-off-by: lecaros --- SUMMARY.md | 14 ++++++++------ administration/buffering-and-storage.md | 2 +- .../configuring-fluent-bit/classic-mode/README.md | 1 + .../{ => classic-mode}/commands.md | 0 .../{ => classic-mode}/configuration-file.md | 4 ++-- .../{ => classic-mode}/format-schema.md | 2 +- .../{ => classic-mode}/record-accessor.md | 2 +- .../{ => classic-mode}/upstream-servers.md | 6 +++--- .../{ => classic-mode}/variables.md | 0 .../configuring-fluent-bit/yaml/README.md | 3 +++ pipeline/filters/grep.md | 4 ++-- pipeline/filters/modify.md | 2 +- pipeline/outputs/forward.md | 2 +- pipeline/outputs/loki.md | 2 +- 14 files changed, 25 insertions(+), 19 deletions(-) create mode 100644 administration/configuring-fluent-bit/classic-mode/README.md rename administration/configuring-fluent-bit/{ => classic-mode}/commands.md (100%) rename administration/configuring-fluent-bit/{ => classic-mode}/configuration-file.md (98%) rename administration/configuring-fluent-bit/{ => classic-mode}/format-schema.md (96%) rename administration/configuring-fluent-bit/{ => classic-mode}/record-accessor.md (96%) rename administration/configuring-fluent-bit/{ => classic-mode}/upstream-servers.md (80%) rename administration/configuring-fluent-bit/{ => classic-mode}/variables.md (100%) create mode 100644 administration/configuring-fluent-bit/yaml/README.md diff --git a/SUMMARY.md b/SUMMARY.md index e223fa360..b24da02bb 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -48,14 +48,16 @@ ## Administration * [Configuring Fluent Bit](administration/configuring-fluent-bit/README.md) - * [Format and Schema](administration/configuring-fluent-bit/format-schema.md) - * [Configuration File](administration/configuring-fluent-bit/configuration-file.md) - * [Variables](administration/configuring-fluent-bit/variables.md) - * [Commands](administration/configuring-fluent-bit/commands.md) - * [Upstream Servers](administration/configuring-fluent-bit/upstream-servers.md) + * [Classic mode](administration/configuring-fluent-bit/classic-mode/README.md) + * [Format and Schema](administration/configuring-fluent-bit/classic-mode/format-schema.md) + * [Configuration File](administration/configuring-fluent-bit/classic-mode/configuration-file.md) + * [Variables](administration/configuring-fluent-bit/classic-mode/variables.md) + * [Commands](administration/configuring-fluent-bit/classic-mode/commands.md) + * [Upstream Servers](administration/configuring-fluent-bit/classic-mode/upstream-servers.md) + * [Record Accessor](administration/configuring-fluent-bit/classic-mode/record-accessor.md) + * [Config with YAML](administration/configuring-fluent-bit/yaml/README.md) * [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md) * [Multiline Parsing](administration/configuring-fluent-bit/multiline-parsing.md) - * [Record Accessor](administration/configuring-fluent-bit/record-accessor.md) * [Security](administration/security.md) * [Buffering & Storage](administration/buffering-and-storage.md) * [Backpressure](administration/backpressure.md) diff --git a/administration/buffering-and-storage.md b/administration/buffering-and-storage.md index a88518320..411b6d7e3 100644 --- a/administration/buffering-and-storage.md +++ b/administration/buffering-and-storage.md @@ -60,7 +60,7 @@ The known Service section configure a global environment for the storage layer, ### Service Section Configuration -The Service section refers to the section defined in the main [configuration file](configuring-fluent-bit/configuration-file.md): +The Service section refers to the section defined in the main [configuration file](configuring-fluent-bit/classic-mode/configuration-file.md): | Key | Description | Default | | :--- | :--- | :--- | diff --git a/administration/configuring-fluent-bit/classic-mode/README.md b/administration/configuring-fluent-bit/classic-mode/README.md new file mode 100644 index 000000000..4d30e8a3b --- /dev/null +++ b/administration/configuring-fluent-bit/classic-mode/README.md @@ -0,0 +1 @@ +# Fluent Bit classic mode \ No newline at end of file diff --git a/administration/configuring-fluent-bit/commands.md b/administration/configuring-fluent-bit/classic-mode/commands.md similarity index 100% rename from administration/configuring-fluent-bit/commands.md rename to administration/configuring-fluent-bit/classic-mode/commands.md diff --git a/administration/configuring-fluent-bit/configuration-file.md b/administration/configuring-fluent-bit/classic-mode/configuration-file.md similarity index 98% rename from administration/configuring-fluent-bit/configuration-file.md rename to administration/configuring-fluent-bit/classic-mode/configuration-file.md index c67bcdc7d..1993eecd3 100644 --- a/administration/configuring-fluent-bit/configuration-file.md +++ b/administration/configuring-fluent-bit/classic-mode/configuration-file.md @@ -31,7 +31,7 @@ The _Service_ section defines global properties of the service, the keys availab | log_level | Set the logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. Values are accumulative, e.g: if 'debug' is set, it will include error, warning, info and debug. Note that _trace_ mode is only available if Fluent Bit was built with the _WITH\_TRACE_ option enabled. | info | | parsers_file | Path for a `parsers` configuration file. Multiple Parsers_File entries can be defined within the section. | | | plugins_file | Path for a `plugins` configuration file. A _plugins_ configuration file allows to define paths for external plugins, for an example [see here](https://github.com/fluent/fluent-bit/blob/master/conf/plugins.conf). | | -| streams_file | Path for the Stream Processor configuration file. To learn more about Stream Processing configuration go [here](../../stream-processing/introduction.md). | | +| streams_file | Path for the Stream Processor configuration file. To learn more about Stream Processing configuration go [here](../../../stream-processing/introduction.md). | | | http_server | Enable built-in HTTP Server | Off | | http_listen | Set listening interface for HTTP Server when it's enabled | 0.0.0.0 | | http_port | Set TCP Port for the HTTP Server | 2020 | @@ -204,7 +204,7 @@ pipeline: You can also visualize Fluent Bit INPUT, FILTER, and OUTPUT configuration via [https://cloud.calyptia.com](https://cloud.calyptia.com/visualizer) -![](../../.gitbook/assets/image.png) +![](../../../.gitbook/assets/image.png) ## Include File diff --git a/administration/configuring-fluent-bit/format-schema.md b/administration/configuring-fluent-bit/classic-mode/format-schema.md similarity index 96% rename from administration/configuring-fluent-bit/format-schema.md rename to administration/configuring-fluent-bit/classic-mode/format-schema.md index 8afa749ce..3ba2ffc75 100644 --- a/administration/configuring-fluent-bit/format-schema.md +++ b/administration/configuring-fluent-bit/classic-mode/format-schema.md @@ -1,6 +1,6 @@ # Formats and Schema -Fluent Bit might optionally use a configuration file to define how the service will behave. Starting in Fluent Bit 1.9, it supports yaml format plus the usual plain text. +Fluent Bit might optionally use a configuration file to define how the service will behave. Before proceeding we need to understand how the configuration schema works. diff --git a/administration/configuring-fluent-bit/record-accessor.md b/administration/configuring-fluent-bit/classic-mode/record-accessor.md similarity index 96% rename from administration/configuring-fluent-bit/record-accessor.md rename to administration/configuring-fluent-bit/classic-mode/record-accessor.md index 87be41b5a..81fffd20c 100644 --- a/administration/configuring-fluent-bit/record-accessor.md +++ b/administration/configuring-fluent-bit/classic-mode/record-accessor.md @@ -42,7 +42,7 @@ If the accessor key does not exist in the record like the last example `$labels[ ## Usage Example -The feature is enabled on a per plugin basis, not all plugins enable this feature. As an example consider a configuration that aims to filter records using [grep](../../pipeline/filters/grep.md) that only matches where labels have a color blue: +The feature is enabled on a per plugin basis, not all plugins enable this feature. As an example consider a configuration that aims to filter records using [grep](../../../pipeline/filters/grep.md) that only matches where labels have a color blue: ```text [SERVICE] diff --git a/administration/configuring-fluent-bit/upstream-servers.md b/administration/configuring-fluent-bit/classic-mode/upstream-servers.md similarity index 80% rename from administration/configuring-fluent-bit/upstream-servers.md rename to administration/configuring-fluent-bit/classic-mode/upstream-servers.md index 951684ff2..1696bba8e 100644 --- a/administration/configuring-fluent-bit/upstream-servers.md +++ b/administration/configuring-fluent-bit/classic-mode/upstream-servers.md @@ -1,10 +1,10 @@ # Upstream Servers -It's common that Fluent Bit [output plugins](../../pipeline/outputs/) aims to connect to external services to deliver the logs over the network, this is the case of [HTTP](../../pipeline/outputs/http.md), [Elasticsearch](../../pipeline/outputs/elasticsearch.md) and [Forward](../../pipeline/outputs/forward.md) within others. Being able to connect to one node \(host\) is normal and enough for more of the use cases, but there are other scenarios where balancing across different nodes is required. The _Upstream_ feature provides such capability. +It's common that Fluent Bit [output plugins](../../pipeline/outputs/) aims to connect to external services to deliver the logs over the network, this is the case of [HTTP](../../../pipeline/outputs/http.md), [Elasticsearch](../../../pipeline/outputs/elasticsearch.md) and [Forward](../../../pipeline/outputs/forward.md) within others. Being able to connect to one node \(host\) is normal and enough for more of the use cases, but there are other scenarios where balancing across different nodes is required. The _Upstream_ feature provides such capability. An _Upstream_ defines a set of nodes that will be targeted by an output plugin, by the nature of the implementation an output plugin **must** support the _Upstream_ feature. The following plugin\(s\) have _Upstream_ support: -* [Forward](../../pipeline/outputs/forward.md) +* [Forward](../../../pipeline/outputs/forward.md) The current balancing mode implemented is _round-robin_. @@ -27,7 +27,7 @@ A _Node_ might contain additional configuration keys required by the plugin, on In addition to the properties defined in the table above, the network operations against a defined node can optionally be done through the use of TLS for further encryption and certificates use. -The TLS options available are described in the [TLS/SSL](../security.md) section and can be added to the any _Node_ section. +The TLS options available are described in the [TLS/SSL](../../security.md) section and can be added to the any _Node_ section. ### Configuration File Example diff --git a/administration/configuring-fluent-bit/variables.md b/administration/configuring-fluent-bit/classic-mode/variables.md similarity index 100% rename from administration/configuring-fluent-bit/variables.md rename to administration/configuring-fluent-bit/classic-mode/variables.md diff --git a/administration/configuring-fluent-bit/yaml/README.md b/administration/configuring-fluent-bit/yaml/README.md new file mode 100644 index 000000000..b5c43d4d5 --- /dev/null +++ b/administration/configuring-fluent-bit/yaml/README.md @@ -0,0 +1,3 @@ +# Fluent Bit YAML configuration + +Since version 1.9, Fluent Bit supports the use of YAML for configuration. \ No newline at end of file diff --git a/pipeline/filters/grep.md b/pipeline/filters/grep.md index 5cd37280b..0fd3c3993 100644 --- a/pipeline/filters/grep.md +++ b/pipeline/filters/grep.md @@ -17,7 +17,7 @@ The plugin supports the following configuration parameters: #### Record Accessor Enabled -This plugin enables the [Record Accessor](../../administration/configuring-fluent-bit/record-accessor.md) feature to specify the KEY. Using the _record accessor_ is suggested if you want to match values against nested values. +This plugin enables the [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) feature to specify the KEY. Using the _record accessor_ is suggested if you want to match values against nested values. ## Getting Started @@ -66,7 +66,7 @@ The filter allows to use multiple rules which are applied in order, you can have ### Nested fields example -If you want to match or exclude records based on nested values, you can use a [Record Accessor ](../../administration/configuring-fluent-bit/record-accessor.md)format as the KEY name. Consider the following record example: +If you want to match or exclude records based on nested values, you can use a [Record Accessor ](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md)format as the KEY name. Consider the following record example: ```javascript { diff --git a/pipeline/filters/modify.md b/pipeline/filters/modify.md index a90a09190..3d19a9436 100644 --- a/pipeline/filters/modify.md +++ b/pipeline/filters/modify.md @@ -71,7 +71,7 @@ The plugin supports the following conditions: * Any number of conditions can be set. * Conditions apply to the whole filter instance and all its rules. _Not_ to individual rules. * All conditions have to be `true` for the rules to be applied. -* You can set [Record Accessor](../../administration/configuring-fluent-bit/record-accessor.md) as `STRING:KEY` for nested key. +* You can set [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) as `STRING:KEY` for nested key. ## Example \#1 - Add and Rename diff --git a/pipeline/outputs/forward.md b/pipeline/outputs/forward.md index 68ff1f0fa..43923e5cb 100644 --- a/pipeline/outputs/forward.md +++ b/pipeline/outputs/forward.md @@ -17,7 +17,7 @@ The following parameters are mandatory for either Forward for Secure Forward mod | Host | Target host where Fluent-Bit or Fluentd are listening for Forward messages. | 127.0.0.1 | | Port | TCP Port of the target service. | 24224 | | Time_as_Integer | Set timestamps in integer format, it enable compatibility mode for Fluentd v0.12 series. | False | -| Upstream | If Forward will connect to an _Upstream_ instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the [Upstream Servers ](../../administration/configuring-fluent-bit/upstream-servers.md)documentation section. | | +| Upstream | If Forward will connect to an _Upstream_ instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the [Upstream Servers ](../../administration/configuring-fluent-bit/classic-mode/upstream-servers.md)documentation section. | | | Tag | Overwrite the tag as we transmit. This allows the receiving pipeline start fresh, or to attribute source. | | | Send_options | Always send options (with "size"=count of messages) | False | | Require_ack_response | Send "chunk"-option and wait for "ack" response from server. Enables at-least-once and receiving server can control rate of traffic. (Requires Fluentd v0.14.0+ server) | False | diff --git a/pipeline/outputs/loki.md b/pipeline/outputs/loki.md index 604f36ea0..300b795aa 100644 --- a/pipeline/outputs/loki.md +++ b/pipeline/outputs/loki.md @@ -57,7 +57,7 @@ If you decide that your Loki Stream will be composed by two labels called `job` ``` As you can see the label `job` has the value `fluentbit` and the second label is configured to access the nested map called `sub` targeting the value of the key `stream` . -Note that the second label name **must** starts with a `$`, that means that's a [Record Accessor](../../administration/configuring-fluent-bit/record-accessor.md) pattern so it provide you the ability to retrieve values from nested maps by using the key names. +Note that the second label name **must** starts with a `$`, that means that's a [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) pattern so it provide you the ability to retrieve values from nested maps by using the key names. When processing above's configuration, internally the ending labels for the stream in question becomes: From 24949049668effc3bc709da81fe272ea68e1ca31 Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 10 Mar 2022 20:21:50 -0300 Subject: [PATCH 02/10] administration: configuration: yaml: add configuration-file Signed-off-by: lecaros --- .../yaml/configuration-file.md | 147 ++++++++++++++++++ 1 file changed, 147 insertions(+) create mode 100644 administration/configuring-fluent-bit/yaml/configuration-file.md diff --git a/administration/configuring-fluent-bit/yaml/configuration-file.md b/administration/configuring-fluent-bit/yaml/configuration-file.md new file mode 100644 index 000000000..84955b241 --- /dev/null +++ b/administration/configuring-fluent-bit/yaml/configuration-file.md @@ -0,0 +1,147 @@ +--- +description: This page describes the yaml configuration file used by Fluent Bit +--- + +# YAML Configuration File + +One of the ways to configure Fluent Bit is using a YAML configuration file that works at a global scope. + +The yaml configuration file supports these sections: + +* Service +* Pipeline + * Inputs + * Filters + * Outputs +* Env + +## Service + +The _service_ section defines global properties of the service, the keys available as of this version are described in the following table: + +| Key | Description | Default Value | +| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | +| flush | Set the flush time in `seconds.nanoseconds`. The engine loop uses a Flush timeout to define when is required to flush the records ingested by input plugins through the defined output plugins. | 5 | +| grace | Set the grace time in `seconds` as Integer value. The engine loop uses a Grace timeout to define wait time on exit | 5 | +| daemon | Boolean value to set if Fluent Bit should run as a Daemon (background) or not. Allowed values are: yes, no, on and off. note: If you are using a Systemd based unit as the one we provide in our packages, do not turn on this option. | Off | +| dns.mode | Set the primary transport layer protocol used by the asynchronous DNS resolver which can be overriden on a per plugin basis | UDP | +| log_file | Absolute path for an optional log file. By default all logs are redirected to the standard error interface (stderr). | | +| log_level | Set the logging verbosity level. Allowed values are: off, error, warn, info, debug and trace. Values are accumulative, e.g: if 'debug' is set, it will include error, warning, info and debug. Note that _trace_ mode is only available if Fluent Bit was built with the _WITH\_TRACE_ option enabled. | info | +| parsers_file | Path for a `parsers` configuration file. Multiple Parsers_File entries can be defined within the section. | | +| plugins_file | Path for a `plugins` configuration file. A _plugins_ configuration file allows to define paths for external plugins, for an example [see here](https://github.com/fluent/fluent-bit/blob/master/conf/plugins.conf). | | +| streams_file | Path for the Stream Processor configuration file. To learn more about Stream Processing configuration go [here](../../../stream-processing/introduction.md). | | +| http_server | Enable built-in HTTP Server | Off | +| http_listen | Set listening interface for HTTP Server when it's enabled | 0.0.0.0 | +| http_port | Set TCP Port for the HTTP Server | 2020 | +| coro_stack_size | Set the coroutines stack size in bytes. The value must be greater than the page size of the running system. Don't set too small value (say 4096), or coroutine threads can overrun the stack buffer. Do not change the default value of this parameter unless you know what you are doing. | 24576 | +| scheduler.cap | Set a maximum retry time in second. The property is supported from v1.8.7. | 2000 | +| scheduler.base | Set a base of exponential backoff. The property is supported from v1.8.7. | 5 | + +The following is an example of a _service_ section: + +```yaml +service: + flush: 5 + daemon: off + log_level: debug +``` + +## Pipeline + +A _pipeline_ section will define a complete pipeline configuration, including _inputs_, _filters_ and _outputs_ subsections. + +```yaml +pipeline: + inputs: + ... + filters: + ... + outputs: + ... +``` + +### Input + +An _input_ section defines a source (related to an input plugin). Here we will describe the base configuration for each _input_ section. Note that each input plugin may add it own configuration keys: + +| Key | Description | +| ---- |--------------------------------------------------------------------------| +| Name | Name of the input plugin. Defined as subsection of the _inputs_ section. | +| Tag | Tag name associated to all records coming from this plugin. | + +The _Name_ is mandatory and it let Fluent Bit know which input plugin should be loaded. The _Tag_ is mandatory for all plugins except for the _input forward_ plugin (as it provides dynamic tags). + +#### Example + +The following is an example of an _input_ section for the _cpu_ plugin. + +```yaml +pipeline: + inputs: + cpu: + tag: my_cpu +``` + +### Filter + +A _filter_ section defines a filter (related to an filter plugin). Here we will describe the base configuration for each _filter_ section. Note that each filter plugin may add it own configuration keys: + +| Key | Description | +|--- |---------------------------------------------------------------------------------------------------------------------------------| +| Name | Name of the filter plugin. Defined as a subsection of the _filters_ section. | +| Match | A pattern to match against the tags of incoming records. It's case sensitive and support the star (\*) character as a wildcard. | +| Match_Regex | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regex syntax. | + +The _Name_ is mandatory and it let Fluent Bit know which filter plugin should be loaded. The _Match_ or _Match_Regex_ is mandatory for all plugins. If both are specified, _Match_Regex_ takes precedence. + +#### Example + +The following is an example of a _filter_ section for the grep plugin: + +```yaml +pipeline: + filters: + grep: + match: * + regex: log aa +``` + +### Output + +The _outputs_ section specify a destination that certain records should follow after a Tag match. The configuration support the following keys: + +| Key | Description | +| ----------- | ------------------------------------------------------------------------------------------------------------------------------- | +| Name | Name of the output plugin. Defined as a subsection of the _outputs_ section. | | +| Match | A pattern to match against the tags of incoming records. It's case sensitive and support the star (\*) character as a wildcard. | +| Match_Regex | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regex syntax. | + +#### Example + +The following is an example of an _output_ section: + +```yaml +pipeline: + outputs: + stdout: + match: my*cpu +``` + +#### Example: collecting CPU metrics + +The following configuration file example demonstrates how to collect CPU metrics and flush the results every five seconds to the standard output: + +```yaml +service: + flush: 5 + daemon: off + log_level: debug + +pipeline: + inputs: + cpu: + tag: my_cpu + outputs: + stdout: + match: my*cpu +``` From 1af7cb7a717df2fd42ca78e285e8776f66297174 Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 10 Mar 2022 20:26:09 -0300 Subject: [PATCH 03/10] administration: configuration: format-schema: remove yaml samples Signed-off-by: lecaros --- .../classic-mode/format-schema.md | 30 ------------------- 1 file changed, 30 deletions(-) diff --git a/administration/configuring-fluent-bit/classic-mode/format-schema.md b/administration/configuring-fluent-bit/classic-mode/format-schema.md index 3ba2ffc75..5e38ba962 100644 --- a/administration/configuring-fluent-bit/classic-mode/format-schema.md +++ b/administration/configuring-fluent-bit/classic-mode/format-schema.md @@ -12,26 +12,12 @@ The schema is defined by three concepts: A simple example of a configuration file is as follows: -{% tabs %} -{% tab title="fluent-bit.conf" %} - ```python [SERVICE] # This is a commented line Daemon off log_level debug ``` -{% endtab %} - -{% tab title="fluent-bit.yaml" %} -```yaml -service: - # This is a commented line - daemon: off - log_level: debug -``` -{% endtab %} -{% endtabs %} ## Sections @@ -57,8 +43,6 @@ Also commented lines are set prefixing the **\#** character, those lines are not Fluent Bit configuration files are based in a strict **Indented Mode**, that means that each configuration file must follow the same pattern of alignment from left to right when writing text. By default an indentation level of four spaces from left to right is suggested. Example: -{% tabs %} -{% tab title="fluent-bit.conf" %} ```python [FIRST_SECTION] # This is a commented line @@ -69,20 +53,6 @@ Fluent Bit configuration files are based in a strict **Indented Mode**, that mea [SECOND_SECTION] KeyN 3.14 ``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} -```yaml -first_section: - # This is a commented line - Key1: some value - Key2: another value - # more comments - -second_section: - KeyN: 3.14 -``` -{% endtab %} -{% endtabs %} As you can see there are two sections with multiple entries and comments, note also that empty lines are allowed and they do not need to be indented. From 779c18d317aa3400ab2b2a37c6980051f2b7899e Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 10 Mar 2022 20:27:20 -0300 Subject: [PATCH 04/10] administration: configuration: configuration-file: remove yaml samples Signed-off-by: lecaros --- .../classic-mode/configuration-file.md | 69 ------------------- 1 file changed, 69 deletions(-) diff --git a/administration/configuring-fluent-bit/classic-mode/configuration-file.md b/administration/configuring-fluent-bit/classic-mode/configuration-file.md index 1993eecd3..9799537bd 100644 --- a/administration/configuring-fluent-bit/classic-mode/configuration-file.md +++ b/administration/configuring-fluent-bit/classic-mode/configuration-file.md @@ -41,24 +41,12 @@ The _Service_ section defines global properties of the service, the keys availab The following is an example of a _SERVICE_ section: -{% tabs %} -{% tab title="fluent-bit.conf" %} ```python [SERVICE] Flush 5 Daemon off Log_Level debug ``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} -```yaml -service: - flush: 5 - daemon: off - log_level: debug -``` -{% endtab %} -{% endtabs %} ## Input @@ -75,23 +63,11 @@ The _Name_ is mandatory and it let Fluent Bit know which input plugin should be The following is an example of an _INPUT_ section: -{% tabs %} -{% tab title="fluent-bit.conf" %} ```python [INPUT] Name cpu Tag my_cpu ``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} -```yaml -pipeline: - inputs: - cpu: - tag: my_cpu -``` -{% endtab %} -{% endtabs %} ## Filter @@ -109,25 +85,12 @@ The _Name_ is mandatory and it let Fluent Bit know which filter plugin should be The following is an example of an _FILTER_ section: -{% tabs %} -{% tab title="fluent-bit.conf" %} ```python [FILTER] Name grep Match * Regex log aa ``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} -```yaml -pipeline: - filters: - regex: - match: * - regex: log aa -``` -{% endtab %} -{% endtabs %} ## Output @@ -143,30 +106,16 @@ The _OUTPUT_ section specify a destination that certain records should follow af The following is an example of an _OUTPUT_ section: -{% tabs %} -{% tab title="fluent-bit.conf" %} ```python [OUTPUT] Name stdout Match my*cpu ``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} -```yaml -pipeline: - outputs: - stdout: - match: my*cpu -``` -{% endtab %} -{% endtabs %} ### Example: collecting CPU metrics The following configuration file example demonstrates how to collect CPU metrics and flush the results every five seconds to the standard output: -{% tabs %} -{% tab title="fluent-bit.conf" %} ```python [SERVICE] Flush 5 @@ -181,24 +130,6 @@ The following configuration file example demonstrates how to collect CPU metrics Name stdout Match my*cpu ``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} -```yaml -service: - flush: 5 - daemon: off - log_level: debug - -pipeline: - inputs: - cpu: - tag: my_cpu - outputs: - stdout: - match: my*cpu -``` -{% endtab %} -{% endtabs %} ## Visualize From b69916fae9be73370f26898dad93ca7960f9130d Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 10 Mar 2022 20:28:46 -0300 Subject: [PATCH 05/10] administration: configuration: yaml: add configuration-file Signed-off-by: lecaros --- SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/SUMMARY.md b/SUMMARY.md index b24da02bb..df159b797 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -56,6 +56,7 @@ * [Upstream Servers](administration/configuring-fluent-bit/classic-mode/upstream-servers.md) * [Record Accessor](administration/configuring-fluent-bit/classic-mode/record-accessor.md) * [Config with YAML](administration/configuring-fluent-bit/yaml/README.md) + * [Configuration File](administration/configuring-fluent-bit/yaml/configuration-file.md) * [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md) * [Multiline Parsing](administration/configuring-fluent-bit/multiline-parsing.md) * [Security](administration/security.md) From 994c8aa86d54173393db72496850647beca08d88 Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 10 Mar 2022 20:36:13 -0300 Subject: [PATCH 06/10] administration: configuration: yaml: add env section to configuration-file Signed-off-by: lecaros --- .../yaml/configuration-file.md | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/administration/configuring-fluent-bit/yaml/configuration-file.md b/administration/configuring-fluent-bit/yaml/configuration-file.md index 84955b241..64b2d9cbf 100644 --- a/administration/configuring-fluent-bit/yaml/configuration-file.md +++ b/administration/configuring-fluent-bit/yaml/configuration-file.md @@ -8,12 +8,30 @@ One of the ways to configure Fluent Bit is using a YAML configuration file that The yaml configuration file supports these sections: +* Env * Service * Pipeline * Inputs * Filters * Outputs -* Env + +## Env + +The _env_ section allows to configure variables that will be used later on this configuration file. + +Example: + +```yaml +# setting up a local environment variable +env: + flush_interval: 1 + +# service configuration +service: + flush: ${flush_interval} + log_level: info + http_server: on +``` ## Service From 0bdcf126fd20757692724d025d0b48bae45a7ffb Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 10 Mar 2022 20:38:46 -0300 Subject: [PATCH 07/10] administration: configuration: yaml: configuration-file: fix a href Signed-off-by: lecaros --- .../configuring-fluent-bit/yaml/configuration-file.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/administration/configuring-fluent-bit/yaml/configuration-file.md b/administration/configuring-fluent-bit/yaml/configuration-file.md index 64b2d9cbf..a2ebba576 100644 --- a/administration/configuring-fluent-bit/yaml/configuration-file.md +++ b/administration/configuring-fluent-bit/yaml/configuration-file.md @@ -15,7 +15,7 @@ The yaml configuration file supports these sections: * Filters * Outputs -## Env +## Env The _env_ section allows to configure variables that will be used later on this configuration file. @@ -64,7 +64,7 @@ service: log_level: debug ``` -## Pipeline +## Pipeline A _pipeline_ section will define a complete pipeline configuration, including _inputs_, _filters_ and _outputs_ subsections. From 8a418ef6136c013ce1d6c60189146745c3fe5c23 Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 10 Mar 2022 20:41:36 -0300 Subject: [PATCH 08/10] administration: configuration: yaml: configuration-file: fix a href Signed-off-by: lecaros --- .../configuring-fluent-bit/yaml/configuration-file.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/administration/configuring-fluent-bit/yaml/configuration-file.md b/administration/configuring-fluent-bit/yaml/configuration-file.md index a2ebba576..66f7f20f3 100644 --- a/administration/configuring-fluent-bit/yaml/configuration-file.md +++ b/administration/configuring-fluent-bit/yaml/configuration-file.md @@ -15,7 +15,7 @@ The yaml configuration file supports these sections: * Filters * Outputs -## Env +## Env The _env_ section allows to configure variables that will be used later on this configuration file. @@ -64,7 +64,7 @@ service: log_level: debug ``` -## Pipeline +## Pipeline A _pipeline_ section will define a complete pipeline configuration, including _inputs_, _filters_ and _outputs_ subsections. From 50dc91210a998cae41bf9c019ea6951d0a6c85b7 Mon Sep 17 00:00:00 2001 From: lecaros Date: Fri, 11 Mar 2022 08:32:07 -0300 Subject: [PATCH 09/10] administration: configuration: yaml: add warning hint IRT tech preview. Signed-off-by: lecaros --- administration/configuring-fluent-bit/yaml/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/administration/configuring-fluent-bit/yaml/README.md b/administration/configuring-fluent-bit/yaml/README.md index b5c43d4d5..79894e320 100644 --- a/administration/configuring-fluent-bit/yaml/README.md +++ b/administration/configuring-fluent-bit/yaml/README.md @@ -1,3 +1,7 @@ # Fluent Bit YAML configuration -Since version 1.9, Fluent Bit supports the use of YAML for configuration. \ No newline at end of file +Since version 1.9, Fluent Bit supports the use of YAML for configuration. + +{% hint style="info" %} +YAML configuration is tech preview so not recommended for production. +{% endhint %} From 662476acb370ae5610afcc4b16b4b98640cbf9b2 Mon Sep 17 00:00:00 2001 From: lecaros Date: Fri, 11 Mar 2022 08:36:01 -0300 Subject: [PATCH 10/10] administration: configuration: yaml: fix typos Signed-off-by: lecaros --- SUMMARY.md | 2 +- .../configuring-fluent-bit/yaml/configuration-file.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index df159b797..33c978424 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -55,7 +55,7 @@ * [Commands](administration/configuring-fluent-bit/classic-mode/commands.md) * [Upstream Servers](administration/configuring-fluent-bit/classic-mode/upstream-servers.md) * [Record Accessor](administration/configuring-fluent-bit/classic-mode/record-accessor.md) - * [Config with YAML](administration/configuring-fluent-bit/yaml/README.md) + * [YAML Configuration](administration/configuring-fluent-bit/yaml/README.md) * [Configuration File](administration/configuring-fluent-bit/yaml/configuration-file.md) * [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md) * [Multiline Parsing](administration/configuring-fluent-bit/multiline-parsing.md) diff --git a/administration/configuring-fluent-bit/yaml/configuration-file.md b/administration/configuring-fluent-bit/yaml/configuration-file.md index 66f7f20f3..25d6a3596 100644 --- a/administration/configuring-fluent-bit/yaml/configuration-file.md +++ b/administration/configuring-fluent-bit/yaml/configuration-file.md @@ -6,7 +6,7 @@ description: This page describes the yaml configuration file used by Fluent Bit One of the ways to configure Fluent Bit is using a YAML configuration file that works at a global scope. -The yaml configuration file supports these sections: +The yaml configuration file supports the following sections: * Env * Service