From 41e754941dafbf0134b66c111c88c3455a8184cb Mon Sep 17 00:00:00 2001 From: Nico Monchy <4321410+n-my@users.noreply.github.com> Date: Wed, 23 Apr 2025 18:59:44 +0000 Subject: [PATCH 1/9] filters as processors Signed-off-by: Nico Monchy <4321410+n-my@users.noreply.github.com> --- pipeline/processors/README.md | 1 + pipeline/processors/filters.md | 28 ++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+) create mode 100644 pipeline/processors/filters.md diff --git a/pipeline/processors/README.md b/pipeline/processors/README.md index c1f055904..2dd91aa1e 100644 --- a/pipeline/processors/README.md +++ b/pipeline/processors/README.md @@ -19,6 +19,7 @@ Fluent Bit offers the following processors: - [OpenTelemetry Envelope](opentelemetry-envelope.md): Transform logs into an OpenTelemetry-compatible format. - [SQL](sql.md): Use SQL queries to extract log content. +- [Filters](filters.md): Any filter can be used as a processor. ## Features diff --git a/pipeline/processors/filters.md b/pipeline/processors/filters.md new file mode 100644 index 000000000..4ee2f6c16 --- /dev/null +++ b/pipeline/processors/filters.md @@ -0,0 +1,28 @@ +# Filters + +Any Fluent Bit [filters](../../filters/README.md) can be used as a processor. + +## grep example + +In this example the grep filter is used as an output processor to send Log records only if they match a regular expression. + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: tail + path: lines.txt + parser: json + outputs: + - name: stdout + match: '*' + processors: + logs: + - name: grep + regex: log aa +``` + +{% endtab %} +{% endtabs %} From 0e3c660acaf0e8b82b6b7ae90b6595ed947d82fc Mon Sep 17 00:00:00 2001 From: Nico Monchy <4321410+n-my@users.noreply.github.com> Date: Wed, 23 Apr 2025 19:03:30 +0000 Subject: [PATCH 2/9] filters as processors Signed-off-by: Nico Monchy <4321410+n-my@users.noreply.github.com> --- pipeline/processors/filters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/processors/filters.md b/pipeline/processors/filters.md index 4ee2f6c16..12844bce4 100644 --- a/pipeline/processors/filters.md +++ b/pipeline/processors/filters.md @@ -1,6 +1,6 @@ # Filters -Any Fluent Bit [filters](../../filters/README.md) can be used as a processor. +Any Fluent Bit [filter](../../filters/README.md) can be used as a processor. ## grep example From 769e0b6321937a56563ee03984eb634ef3387c3b Mon Sep 17 00:00:00 2001 From: Nico Monchy <4321410+n-my@users.noreply.github.com> Date: Wed, 23 Apr 2025 19:15:09 +0000 Subject: [PATCH 3/9] symmary: add processors filters page Signed-off-by: Nico Monchy <4321410+n-my@users.noreply.github.com> --- SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/SUMMARY.md b/SUMMARY.md index ea3ab6cea..c45bbcc55 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -147,6 +147,7 @@ * [Metrics Selector](pipeline/processors/metrics-selector.md) * [OpenTelemetry Envelope](pipeline/processors/opentelemetry-envelope.md) * [SQL](pipeline/processors/sql.md) + * [Filters](pipeline/processors/filters.md) * [Conditional processing](pipeline/processors/conditional-processing.md) * [Filters](pipeline/filters/README.md) * [AWS Metadata](pipeline/filters/aws-metadata.md) From dea157d1e61e55f627aab9792fc7798c064d28e2 Mon Sep 17 00:00:00 2001 From: Nico Monchy <4321410+n-my@users.noreply.github.com> Date: Wed, 23 Apr 2025 19:21:14 +0000 Subject: [PATCH 4/9] pipeline: processors: filters: fix link relative path Signed-off-by: Nico Monchy <4321410+n-my@users.noreply.github.com> --- pipeline/processors/filters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/processors/filters.md b/pipeline/processors/filters.md index 12844bce4..2358f6d47 100644 --- a/pipeline/processors/filters.md +++ b/pipeline/processors/filters.md @@ -1,6 +1,6 @@ # Filters -Any Fluent Bit [filter](../../filters/README.md) can be used as a processor. +Any Fluent Bit [filter](../filters/README.md) can be used as a processor. ## grep example From ea9058965be9a603b70e39edde0512b15978eeb5 Mon Sep 17 00:00:00 2001 From: Nico Monchy <4321410+n-my@users.noreply.github.com> Date: Wed, 23 Apr 2025 16:16:41 -0400 Subject: [PATCH 5/9] Update pipeline/processors/filters.md Co-authored-by: Alexa Kreizinger Signed-off-by: Nico Monchy <4321410+n-my@users.noreply.github.com> --- pipeline/processors/filters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/processors/filters.md b/pipeline/processors/filters.md index 2358f6d47..b25f0b749 100644 --- a/pipeline/processors/filters.md +++ b/pipeline/processors/filters.md @@ -1,4 +1,4 @@ -# Filters +# Filters as processors Any Fluent Bit [filter](../filters/README.md) can be used as a processor. From 9278e6517b9cb76c6f6d0ff8033948cf8a374c68 Mon Sep 17 00:00:00 2001 From: Nico Monchy <4321410+n-my@users.noreply.github.com> Date: Wed, 23 Apr 2025 16:17:01 -0400 Subject: [PATCH 6/9] Update pipeline/processors/filters.md Co-authored-by: Alexa Kreizinger Signed-off-by: Nico Monchy <4321410+n-my@users.noreply.github.com> --- pipeline/processors/filters.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pipeline/processors/filters.md b/pipeline/processors/filters.md index b25f0b749..d7f00fda4 100644 --- a/pipeline/processors/filters.md +++ b/pipeline/processors/filters.md @@ -2,9 +2,9 @@ Any Fluent Bit [filter](../filters/README.md) can be used as a processor. -## grep example +## Grep example -In this example the grep filter is used as an output processor to send Log records only if they match a regular expression. +In this example, the [Grep](../filters/grep) filter is used as an output processor that sends log records only if they match a specified regular expression. {% tabs %} {% tab title="fluent-bit.yaml" %} From 2ae90fed9e101af5dc6f0d1b4c04bbd6b4a10007 Mon Sep 17 00:00:00 2001 From: Nico Monchy <4321410+n-my@users.noreply.github.com> Date: Wed, 23 Apr 2025 16:18:00 -0400 Subject: [PATCH 7/9] Update pipeline/processors/filters.md Co-authored-by: Alexa Kreizinger Signed-off-by: Nico Monchy <4321410+n-my@users.noreply.github.com> --- pipeline/processors/filters.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/pipeline/processors/filters.md b/pipeline/processors/filters.md index d7f00fda4..bb3b98c39 100644 --- a/pipeline/processors/filters.md +++ b/pipeline/processors/filters.md @@ -6,9 +6,7 @@ Any Fluent Bit [filter](../filters/README.md) can be used as a processor. In this example, the [Grep](../filters/grep) filter is used as an output processor that sends log records only if they match a specified regular expression. -{% tabs %} -{% tab title="fluent-bit.yaml" %} - +{% code title="fluent-bit.yaml" %} ```yaml pipeline: inputs: @@ -23,6 +21,4 @@ pipeline: - name: grep regex: log aa ``` - -{% endtab %} -{% endtabs %} +{% endcode %} From 58b9866f29c543456c149116d31b8da86e887b75 Mon Sep 17 00:00:00 2001 From: Nico Monchy <4321410+n-my@users.noreply.github.com> Date: Wed, 23 Apr 2025 16:18:14 -0400 Subject: [PATCH 8/9] Update SUMMARY.md Co-authored-by: Alexa Kreizinger Signed-off-by: Nico Monchy <4321410+n-my@users.noreply.github.com> --- SUMMARY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SUMMARY.md b/SUMMARY.md index c45bbcc55..e7c42940e 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -147,7 +147,7 @@ * [Metrics Selector](pipeline/processors/metrics-selector.md) * [OpenTelemetry Envelope](pipeline/processors/opentelemetry-envelope.md) * [SQL](pipeline/processors/sql.md) - * [Filters](pipeline/processors/filters.md) + * [Filters as processors](pipeline/processors/filters.md) * [Conditional processing](pipeline/processors/conditional-processing.md) * [Filters](pipeline/filters/README.md) * [AWS Metadata](pipeline/filters/aws-metadata.md) From 01201e6b2533f89ddc7bc2e115e9ff00836dc0f7 Mon Sep 17 00:00:00 2001 From: Nico Monchy <4321410+n-my@users.noreply.github.com> Date: Wed, 23 Apr 2025 20:22:16 +0000 Subject: [PATCH 9/9] pipeline: processors: filters: fix link to grep page Signed-off-by: Nico Monchy <4321410+n-my@users.noreply.github.com> --- pipeline/processors/filters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/processors/filters.md b/pipeline/processors/filters.md index bb3b98c39..12a6003fd 100644 --- a/pipeline/processors/filters.md +++ b/pipeline/processors/filters.md @@ -4,7 +4,7 @@ Any Fluent Bit [filter](../filters/README.md) can be used as a processor. ## Grep example -In this example, the [Grep](../filters/grep) filter is used as an output processor that sends log records only if they match a specified regular expression. +In this example, the [Grep](../filters/grep.md) filter is used as an output processor that sends log records only if they match a specified regular expression. {% code title="fluent-bit.yaml" %} ```yaml