consolidate multiline parsers info

.gitbook.yaml

@@ -102,3 +102,4 @@ redirects:
     installation/downloads/amazon-ec2: ./installation/downloads/linux/amazon-linux.md
     administration/configuring-fluent-bit/yaml/configuration-file: ./administration/configuring-fluent-bit/yaml.md
     administration/configuring-fluent-bit/unit-sizes: ./administration/configuring-fluent-bit.md#unit-sizes
+    administration/configuring-fluent-bit/multiline-parsing: ./pipeline/parsers/multiline-parsing.md

CONTRIBUTING.md

@@ -15,7 +15,6 @@ Before you contribute to the Fluent Bit docs, review these critical tips:
 
 - [Sign off](#sign-off-your-git-commits) your Git commits.
 - Use [soft line wraps](#line-wraps) in Markdown files.
-- To link between pages, use [absolute file paths](#links).
 - Review the results of [linters](#linters) for style and formatting guidance.
 
 ## Review process
@@ -69,14 +68,6 @@ The active [linters](#linters) in this repository flag certain style errors and,
 
 The Fluent Bit docs library is built and hosted through [GitBook](https://docs.gitbook.com/). Unfortunately, GitBook doesn't support local previews for contributors, but a Fluent Bit maintainer with a dedicated GitBook account can verify that things are formatted correctly after you open a new pull request.
 
-### Links
-
-When cross-linking between in this repository, use a full absolute path whenever possible. For example:
-
-```text
-[LTSV](../pipeline/parsers/ltsv.md) and [Logfmt](../pipeline/parsers/logfmt.md)
-```
-
 ### Line wraps
 
 When GitBook renders pages, it treats all newlines literally, which means hard line wraps in Markdown files create awkward line breaks in the Fluent Bit docs site. Due to this, docs contributions must use soft line wraps.

SUMMARY.md

@@ -54,7 +54,6 @@
     * [Record accessor syntax](administration/configuring-fluent-bit/classic-mode/record-accessor.md)
     * [Upstream servers](administration/configuring-fluent-bit/classic-mode/upstream-servers.md)
     * [Variables](administration/configuring-fluent-bit/classic-mode/variables.md)
-  * [Multiline parsing](administration/configuring-fluent-bit/multiline-parsing.md)
 * [AWS credentials](administration/aws-credentials.md)
 * [Backpressure](administration/backpressure.md)
 * [Buffering and storage](administration/buffering-and-storage.md)
@@ -126,6 +125,7 @@
   * [Windows System Statistics (winstat)](pipeline/inputs/windows-system-statistics.md)
 * [Parsers](pipeline/parsers.md)
   * [Configuring custom parsers](pipeline/parsers/configuring-parser.md)
+  * [Multiline parsing](pipeline/parsers/multiline-parsing.md)
   * [JSON format](pipeline/parsers/json.md)
   * [Logfmt format](pipeline/parsers/logfmt.md)
   * [LTSV format](pipeline/parsers/ltsv.md)

administration/configuring-fluent-bit/yaml.md

@@ -8,14 +8,14 @@ don't support, like processors.
 
 YAML configuration files support the following top-level sections:
 
-- `env`: Configures [environment variables](../administration/configuring-fluent-bit/yaml/environment-variables-section.md).
-- `includes`: Specifies additional YAML configuration files to [include as part of a parent file](../administration/configuring-fluent-bit/yaml/includes-section.md).
-- `service`: Configures global properties of the Fluent Bit [service](../administration/configuring-fluent-bit/yaml/service-section.md).
-- `pipeline`: Configures active [`inputs`, `filters`, and `outputs`](../administration/configuring-fluent-bit/yaml/pipeline-section.md).
-- `parsers`: Defines [custom parsers](../administration/configuring-fluent-bit/yaml/parsers-section.md).
-- `multiline_parsers`: Defines [custom multiline parsers](../administration/configuring-fluent-bit/yaml/multiline-parsers-section.md).
-- `plugins`: Defines paths for [custom plugins](../administration/configuring-fluent-bit/yaml/plugins-section.md).
-- `upstream_servers`: Defines [nodes](../administration/configuring-fluent-bit/yaml/upstream-servers-section.md) for output plugins.
+- `env`: Configures [environment variables](./yaml/environment-variables-section.md).
+- `includes`: Specifies additional YAML configuration files to [include as part of a parent file](./yaml/includes-section.md).
+- `service`: Configures global properties of the Fluent Bit [service](./yaml/service-section.md).
+- `pipeline`: Configures active [`inputs`, `filters`, and `outputs`](./yaml/pipeline-section.md).
+- `parsers`: Defines [custom parsers](./yaml/parsers-section.md).
+- `multiline_parsers`: Defines [custom multiline parsers](./yaml/multiline-parsers-section.md).
+- `plugins`: Defines paths for [custom plugins](./yaml/plugins-section.md).
+- `upstream_servers`: Defines [nodes](./yaml/upstream-servers-section.md) for output plugins.
 
 {% hint style="info" %}
 YAML configuration is used in the smoke tests for containers. An always-correct up-to-date example is here: <https://github.com/fluent/fluent-bit/blob/master/packaging/testing/smoke/container/fluent-bit.yaml>.

administration/configuring-fluent-bit/yaml/multiline-parsers-section.md

@@ -1,10 +1,19 @@
 # Multiline parsers
 
-Multiline parsers are used to combine logs that span multiple events into a single, cohesive message. Use this parser for handling stack traces, error logs, or any log entry that contains multiple lines of information.
+You can define custom [multiline parsers](../../pipeline/parsers/multiline-parsing.md) in the `multiline_parsers` section of YAML configuration files.
 
-In YAML configuration, the syntax for defining multiline parsers differs slightly from the classic configuration format introducing minor breaking changes, specifically on how the rules are defined.
+{% hint style="info" %}
 
-The following example demonstrates how to define a multiline parser directly in the main configuration file, and how to include additional definitions from external files:
+To define standard custom parsers, use [the `parsers` section](./parsers-section.md) of YAML configuration files.
+
+{% endhint %}
+
+## Syntax
+
+To define custom parsers in the `multiline_parsers` section of a YAML configuration file, use the following syntax:
+
+{% tabs %}
+{% tab title="fluent-bit.yaml" %}
 
 ```yaml
 multiline_parsers:
@@ -20,6 +29,9 @@ multiline_parsers:
         next_state: cont
 ```
 
+{% endtab %}
+{% endtabs %}
+
 This example defines a multiline parser named `multiline-regex-test` that uses regular expressions to handle multi-event logs. The parser contains two rules: the first rule transitions from `start_state` to cont when a matching log entry is detected, and the second rule continues to match subsequent lines.
 
-For more detailed information on configuring multiline parsers, including advanced options and use cases, refer to the Configuring Multiline Parsers documentation.
+For information about supported configuration options for custom multiline parsers, see [configuring multiline parsers](../../pipeline/parsers/multiline-parsing.md#configuring-multiline-parsers).

administration/configuring-fluent-bit/yaml/parsers-section.md

@@ -4,13 +4,13 @@ You can define customer [parsers](../pipeline/parsers.md) in the `parsers` secti
 
 {% hint style="info" %}
 
-To define custom multiline parsers, use [the `multiline_parsers` section](../administration/configuring-fluent-bit/yaml/multiline-parsers-section.md) of YAML configuration files.
+To define custom multiline parsers, use [the `multiline_parsers` section](./multiline-parsers-section.md) of YAML configuration files.
 
 {% endhint %}
 
 ## Syntax
 
-To define customers parsers in the `parsers` section of a YAML configuration file, use the following syntax.
+To define custom parsers in the `parsers` section of a YAML configuration file, use the following syntax.
 
 {% tabs %}
 {% tab title="fluent-bit.yaml" %}
@@ -35,7 +35,7 @@ parsers:
 {% endtab %}
 {% endtabs %}
 
-For information about supported configuration options for custom parsers, see [configuring parsers](../pipeline/parsers/configuring-parser.md).
+For information about supported configuration options for custom parsers, see [configuring parsers](../../pipeline/parsers/configuring-parser.md).
 
 ## Standalone parsers files
 

pipeline/filters/multiline-stacktrace.md

@@ -15,7 +15,7 @@ Along with multiline filters, you can enable one of the following built-in Fluen
 
 When using this filter:
 
-- The usage of this filter depends on a previous configuration of a [multiline parser](../../administration/configuring-fluent-bit/multiline-parsing.md) definition.
+- The usage of this filter depends on a previous configuration of a [multiline parser](../pipeline/parsers/multiline-parsing.md) definition.
 - To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages that were originally one line, but split by Docker or CRI container engines because of their size, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) in combination with the `docker` or `cri` parser. To concatenate application logs like stacktraces on top of that, you can use this multiline filter.
 
 {% hint style="warning" %}
@@ -40,7 +40,7 @@ The plugin supports the following configuration parameters:
 
 | Property | Description |
 | -------- | ----------- |
-| `multiline.parser` | Specify one or multiple [Multiline Parser definitions](../../administration/configuring-fluent-bit/multiline-parsing.md) to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma. |
+| `multiline.parser` | Specify one or multiple [Multiline Parser definitions](../pipeline/parsers/multiline-parsing.md) to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma. |
 | `multiline.key_content` | Key name that holds the content to process. A multiline parser definition can specify the `key_content` This option allows for overwriting that value for the purpose of the filter. |
 | `mode` | Mode can be `parser` for regular expression concatenation, or `partial_message` to concatenate split Docker logs. |
 | `buffer` | Enable buffered mode. In buffered mode, the filter can concatenate multiple lines from inputs that ingest records one by one (like Forward), rather than in chunks, re-emitting them into the beginning of the pipeline (with the same tag) using the `in_emitter` instance. With buffer off, this filter won't work with most inputs, except Tail. |
@@ -68,7 +68,7 @@ service:
   flush: 1
   log_level: info
   parsers_file: parsers_multiline.yaml
-    
+
 pipeline:
   inputs:
     - name: tail
@@ -157,10 +157,10 @@ This file defines a multiline parser for the example. A second multiline parser
   # Regex rules for multiline parsing
   # ---------------------------------
   #
-  # configuration hints:  
+  # configuration hints:
   #
   #  - first state always has the name: start_state
-  #  - every field in the rule must be inside double quotes 
+  #  - every field in the rule must be inside double quotes
   #
   # rules |   state name  | regex pattern                  | next state
   # ------|---------------|--------------------------------------------
@@ -314,7 +314,7 @@ Lines that don't match a pattern aren't considered as part of the multiline mess
 
 ## Docker partial message use case
 
-When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16&nbspKB. 
+When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16&nbspKB.
 If your application emits a 100K log line, it will be split into seven partial messages. The docker parser will merge these back to one line. If instead you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this:
 
 ```text

pipeline/inputs/tail.md

@@ -158,9 +158,9 @@ Multiline core is exposed by the following configuration:
 
 | Key                | Description    |
 |:-------------------|:---------------|
-| `multiline.parser` | Specify one or multiple [Multiline Parser definitions](../../administration/configuring-fluent-bit/multiline-parsing.md) to apply to the content. |
+| `multiline.parser` | Specify one or multiple [Multiline Parser definitions](../parsers/multiline-parsing.md) to apply to the content. |
 
-[Multiline Parser](../../administration/configuring-fluent-bit/multiline-parsing.md) provides built-in configuration modes. When using a new `multiline.parser` definition, you must disable the old configuration from your tail section like:
+[Multiline Parser](../parsers/multiline-parsing.md) provides built-in configuration modes. When using a new `multiline.parser` definition, you must disable the old configuration from your tail section like:
 
 - `parser`
 - `parser_firstline`

pipeline/parsers.md

@@ -39,19 +39,19 @@ For example, a parser can turn an unstructured log entry like this:
 
 ## How parsers work
 
-Parsers modify the data ingested by input plugins. This modification happens before Fluent Bit applies any [filters](../pipeline/filters.md) or [processors](..pipeline/processors.md) to that data.
+Parsers modify the data ingested by input plugins. This modification happens before Fluent Bit applies any [filters](../pipeline/filters.md) or [processors](../pipeline/processors.md) to that data.
 
 Each input plugin can have one active parser. Multiple plugins within the same Fluent Bit configuration file can use the same parser or use different parsers from each other.
 
 ### Default parsers and custom parsers
 
-Fluent Bit includes a variety of [default parsers](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) for parsing common data formats, like Apache and Docker logs. You can also [define custom parsers](../configuring-fluent-bit/yaml/parsers-section.md).
+Fluent Bit includes a variety of [default parsers](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) for parsing common data formats, like Apache and Docker logs. You can also [define custom parsers](../administration/configuring-fluent-bit/yaml/parsers-section.md).
 
 ## Add a parser to an input plugin
 
 To add a parser to an input plugin, follow these steps.
 
-1. Either identify the name of the [default parser](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) you want to use, or [define a custom parser](../configuring-fluent-bit/yaml/parsers-section.md) with your desired [configuration settings](../pipeline/parsers/configuring-parser.md).
+1. Either identify the name of the [default parser](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) you want to use, or [define a custom parser](../administration/configuring-fluent-bit/yaml/parsers-section.md) with your desired [configuration settings](../pipeline/parsers/configuring-parser.md).
 
 1. Add a `parsers` key to the plugin's settings in the [`inputs`](../administration/configuring-fluent-bit/yaml/pipeline-section.md#inputs) section of your YAML configuration file.
 

pipeline/parsers/configuring-parser.md

@@ -1,10 +1,10 @@
 # Configuring custom parsers
 
-Use the information on this page to configure custom [parsers](../pipeline/parsers).
+Use the information on this page to configure custom [parsers](../parsers.md).
 
 {% hint style="info" %}
 
-To define a custom parser, add an entry to the [`parsers` section](../configuring-fluent-bit/yaml/parsers-section.md) of your YAML configuration file, or create a [standalone parser file](../administration/configuring-fluent-bit/yaml/parsers-section.md#standalone-parsers-files).
+To define a custom parser, add an entry to the [`parsers` section](../administration/configuring-fluent-bit/yaml/parsers-section.md) of your YAML configuration file, or create a [standalone parser file](../administration/configuring-fluent-bit/yaml/parsers-section.md#standalone-parsers-files).
 
 {% endhint %}
 

administration/configuring-fluent-bit/multiline-parsing.md → pipeline/parsers/multiline-parsing.md

@@ -4,11 +4,11 @@
 
 In an ideal world, applications might log their messages within a single line, but in reality applications generate multiple log messages that sometimes belong to the same context. Processing this information can be complex, like in application stack traces, which always have multiple log lines.
 
-Fluent Bit v1.8 implemented a unified Multiline core capability to solve corner cases.
+Multiline parsers are used to combine logs that span multiple events into a single, cohesive message. Use this parser for handling stack traces, error logs, or any log entry that contains multiple lines of information.
 
 ## Concepts
 
-The Multiline parser engine exposes two ways to configure and use the feature:
+The multiline parser engine exposes two ways to configure and use the feature:
 
 - Built-in multiline parser
 - Configurable multiline parser
@@ -26,13 +26,15 @@ Fluent Bit exposes certain pre-configured parsers (built-in) to solve specific m
 | `python` | Process log entries generated by a Python-based language application and perform concatenation if multiline messages are detected.      |
 | `ruby`   | Process log entries generated by a Ruby-based language application and perform concatenation if multiline messages are detected.        |
 
-### Configurable multiline parsers
+### Configuring multiline parsers
 
-You can define your own Multiline parsers with their own rules, using a configuration file.
+Custom multiline parsers support the following configuration parameters.
 
-A multiline parser is defined in a `parsers configuration file` by using a `[MULTILINE_PARSER]` section definition. The multiline parser must have a unique name and a type, plus other configured properties associated with each type.
+{% hint style="info" %}
+
+To define a custom multiline parser, add an entry to the [`multiline_parsers` section](../configuring-fluent-bit/yaml/multiline-parsers-section.md) of your YAML configuration file.
 
-To understand which multiline parser type is required for your use case you have to know the conditions in the content that determine the beginning of a multiline message, and the continuation of subsequent lines. Fluent Bit provides a regular expression-based configuration that supports states to handle from the most cases.
+{% endhint %}
 
 | Property | Description | Default |
 | -------- | ----------- | ------- |
@@ -43,7 +45,7 @@ To understand which multiline parser type is required for your use case you have
 | `negate`        | Negate the pattern matching result. When set to `true`, a non-matching line is treated as matching. | `false` |
 | `parser`        | Name of a pre-defined parser that must be applied to the incoming content before applying the regular expression rule. If no parser is defined, it's assumed that's a raw text and not a structured message. When a parser is applied to a raw text, the regular expression is applied against a specific key of the structured message by using the `key_content` configuration property. | _none_ |
 | `rule`          | Configure a rule to match a multiline pattern. The rule has a [specific format](#rules-definition). Multiple rules can be defined. Only used with `regex` type. | _none_|
-| `type`          | Set the multiline mode. Fluent Bit supports `regex`, `endswith`, and `equal` (or `eq`). | _none_ |
+| `type`          | Set the multiline mode. Fluent Bit supports `regex`, `endswith`, and `equal` (or `eq`). To understand which multiline parser type is required for your use case you have to know the conditions in the content that determine the beginning of a multiline message, and the continuation of subsequent lines. Fluent Bit provides a regular expression-based configuration that supports states to handle from the most cases. | _none_ |
 
 #### Lines and states
 
@@ -836,4 +838,4 @@ app/models/user.rb:42:in `validate_email'
 ```
 
 {% endtab %}
-{% endtabs %}
+{% endtabs %}