From 46bc3b91d1ae0badb1afa982e8b135e9f18d48f4 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 15 Mar 2024 18:11:26 +0900 Subject: [PATCH 1/5] processor_metrics_selector: Add documents for metrics_selector processor Signed-off-by: Hiroshi Hatake --- .gitbook.yaml | 3 ++ SUMMARY.md | 2 + installation/sources/build-and-install.md | 7 +++ pipeline/processors/README.md | 2 + pipeline/processors/metrics-selector.md | 59 +++++++++++++++++++++++ 5 files changed, 73 insertions(+) create mode 100644 pipeline/processors/README.md create mode 100644 pipeline/processors/metrics-selector.md diff --git a/.gitbook.yaml b/.gitbook.yaml index 703fd8f6e..e20fb6dd3 100644 --- a/.gitbook.yaml +++ b/.gitbook.yaml @@ -76,3 +76,6 @@ redirects: output/stdout: ./pipeline/outputs/standard-output.md output/tcp: ./pipeline/outputs/tcp-and-tls.md output/td: ./pipeline/outputs/treasure-data.md + + # Processors + processor/metrics_selector: ./pipeline/processors/metrics_selector.md diff --git a/SUMMARY.md b/SUMMARY.md index a1addc3a0..057fa2e64 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -194,6 +194,8 @@ * [Treasure Data](pipeline/outputs/treasure-data.md) * [Vivo Exporter](pipeline/outputs/vivo-exporter.md) * [WebSocket](pipeline/outputs/websocket.md) +* [Processors](pipeline/processors/README.md) + * [Metrics Selector](pipeline/processors/metrics-selector.md) ## Stream Processing diff --git a/installation/sources/build-and-install.md b/installation/sources/build-and-install.md index 4453ce96e..18b4e66e6 100644 --- a/installation/sources/build-and-install.md +++ b/installation/sources/build-and-install.md @@ -212,3 +212,10 @@ The _output plugins_ gives the capacity to flush the information to some externa | FLB\_OUT\_TCP | Enable TCP/TLS output plugin | On | | [FLB\_OUT\_TD](../../pipeline/outputs/treasure-data.md) | Enable [Treasure Data](http://www.treasuredata.com) output plugin | On | +### Processor Plugins + +The _processor plugins_ gives the capability to handle the three types of events within the processor pipelines to allow modifying, enrich or drop events. The following table describes the processors available on this version: + +| option | description | default | +| :--- | :--- | :--- | +| [FLB\_PROCESSOR\_METRICS\_SELECTOR](../../pipeline/processors/metrics-selector.md) | Enable metrics selector processor | On | diff --git a/pipeline/processors/README.md b/pipeline/processors/README.md new file mode 100644 index 000000000..d2a3057f7 --- /dev/null +++ b/pipeline/processors/README.md @@ -0,0 +1,2 @@ +# Processors + diff --git a/pipeline/processors/metrics-selector.md b/pipeline/processors/metrics-selector.md new file mode 100644 index 000000000..70bffd7b1 --- /dev/null +++ b/pipeline/processors/metrics-selector.md @@ -0,0 +1,59 @@ +# Metrics Selector + +The **metric_selector** processor allows you to select metrics from the ingested metrics type of events. + +## Configuration Parameters + +The native processor plugin supports the following configuration parameters: + +| Key | Description | Default | +| :---------- | :--- | :--- | +| Metric\_Name | Keep metrics in which the metric of name matches with the actual name or the regular expression. | | +| Context | Specify matching context. Currently, metric_name is only supported. | `Metrics_Name` | +| Action | Specify the action for specified metrics. INCLUDE and EXCLUDE are allowed. | | +| Operation\_Type | Specify the operation type of action for metrics payloads. PREFIX and SUBSTRING are allowed. | | + +## Configuration Examples + +Here is a basic configuration example. + +{% tabs %} +{% tab title="fluent-bit.yaml" %} +```yaml +service: + flush: 5 + daemon: off + log_level: info + +pipeline: + inputs: + - name: fluentbit_metrics + tag: fluentbit.metrics + scrape_interval: 10 + + processors: + metrics: + - name: metrics_selector + metric_name: /storage/ + action: include + - name: metrics_selector + metric_name: /fs/ + action: exclude + + - name: labels + delete: name + + + outputs: + - name: stdout + match: '*' +``` +{% endtab %} +{% endtabs %} + + +Note that processor is only effective on the YAML format. Also, processor elements should be located under input or output plguins' definitions. + +Metric\_Name parameter will translate the strings which is quoted with backslashes `/.../` as Regular expressions. +Without them, users need to specify Operation\_Type whether prefix matching or substring matching. +The default operation is prefix matching. From d7b2e707ccc5fb33f6d096fbc14a1545e148c60c Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 15 Mar 2024 20:40:49 +0900 Subject: [PATCH 2/5] Update installation/sources/build-and-install.md Co-authored-by: Pat Signed-off-by: Hiroshi Hatake --- installation/sources/build-and-install.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/installation/sources/build-and-install.md b/installation/sources/build-and-install.md index 18b4e66e6..568130cb4 100644 --- a/installation/sources/build-and-install.md +++ b/installation/sources/build-and-install.md @@ -214,7 +214,8 @@ The _output plugins_ gives the capacity to flush the information to some externa ### Processor Plugins -The _processor plugins_ gives the capability to handle the three types of events within the processor pipelines to allow modifying, enrich or drop events. The following table describes the processors available on this version: +The _processor plugins_ provide the capability to handle the events within the processor pipelines to allow modifying, enrich or drop events. +The following table describes the processors available on this version: | option | description | default | | :--- | :--- | :--- | From f5d7f47c1b2763b10fdbe288b650857fffafab06 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 15 Mar 2024 20:40:54 +0900 Subject: [PATCH 3/5] Update pipeline/processors/metrics-selector.md Co-authored-by: Pat Signed-off-by: Hiroshi Hatake --- pipeline/processors/metrics-selector.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/processors/metrics-selector.md b/pipeline/processors/metrics-selector.md index 70bffd7b1..e4f4019d7 100644 --- a/pipeline/processors/metrics-selector.md +++ b/pipeline/processors/metrics-selector.md @@ -1,6 +1,6 @@ # Metrics Selector -The **metric_selector** processor allows you to select metrics from the ingested metrics type of events. +The **metric_selector** processor allows you to select metrics to include or exclude (similar to the `grep` filter for logs). ## Configuration Parameters From 4269015703672f14fbdba7cb1bcfda31f548382f Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 15 Mar 2024 20:41:03 +0900 Subject: [PATCH 4/5] Update pipeline/processors/metrics-selector.md Co-authored-by: Pat Signed-off-by: Hiroshi Hatake --- pipeline/processors/metrics-selector.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pipeline/processors/metrics-selector.md b/pipeline/processors/metrics-selector.md index e4f4019d7..58dcde0ce 100644 --- a/pipeline/processors/metrics-selector.md +++ b/pipeline/processors/metrics-selector.md @@ -52,7 +52,8 @@ pipeline: {% endtabs %} -Note that processor is only effective on the YAML format. Also, processor elements should be located under input or output plguins' definitions. +All processors are only valid with the YAML configuration format. +Processor configuration should be located under the relevant input or output plugin configuration. Metric\_Name parameter will translate the strings which is quoted with backslashes `/.../` as Regular expressions. Without them, users need to specify Operation\_Type whether prefix matching or substring matching. From a03c5b7da4a6cd8e33be974ed70ef486a45c3682 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 15 Mar 2024 20:43:25 +0900 Subject: [PATCH 5/5] processor_metrics_selector: Add a example description for a making regular expression Signed-off-by: Hiroshi Hatake --- pipeline/processors/metrics-selector.md | 1 + 1 file changed, 1 insertion(+) diff --git a/pipeline/processors/metrics-selector.md b/pipeline/processors/metrics-selector.md index 58dcde0ce..962577ad3 100644 --- a/pipeline/processors/metrics-selector.md +++ b/pipeline/processors/metrics-selector.md @@ -58,3 +58,4 @@ Processor configuration should be located under the relevant input or output plu Metric\_Name parameter will translate the strings which is quoted with backslashes `/.../` as Regular expressions. Without them, users need to specify Operation\_Type whether prefix matching or substring matching. The default operation is prefix matching. +For example, `/chunks/` will be translated as a regular expression.