From 5f4a90c4424cba61abf21d497cd545ef8aafa4f2 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Wed, 22 Oct 2025 14:28:37 -0400 Subject: [PATCH 1/9] Removing a readme file this is not standardized in classic-mode directory. Signed-off-by: Eric D. Schabell --- administration/configuring-fluent-bit/classic-mode/README.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 administration/configuring-fluent-bit/classic-mode/README.md diff --git a/administration/configuring-fluent-bit/classic-mode/README.md b/administration/configuring-fluent-bit/classic-mode/README.md deleted file mode 100644 index e4db3b567..000000000 --- a/administration/configuring-fluent-bit/classic-mode/README.md +++ /dev/null @@ -1 +0,0 @@ -# Classic mode From ca437dc7286a2b0ce96a7f2fe362aca473716c05 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Wed, 22 Oct 2025 14:29:22 -0400 Subject: [PATCH 2/9] Updated to not use readme file in classic-config directory, but the correct description file. Signed-off-by: Eric D. Schabell --- administration/configuring-fluent-bit.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/administration/configuring-fluent-bit.md b/administration/configuring-fluent-bit.md index 8fa4042a1..7da8a1473 100644 --- a/administration/configuring-fluent-bit.md +++ b/administration/configuring-fluent-bit.md @@ -3,7 +3,7 @@ Fluent Bit supports two configuration formats: - [YAML](./configuring-fluent-bit/yaml.md): Standard configuration format as of v3.2. -- [Classic mode](./configuring-fluent-bit/classic-mode/README.md): To be deprecated at the end of 2026. +- [Classic mode](./configuring-fluent-bit/classic-mode.md): To be deprecated at the end of 2026. ## Command line interface From 5bbc9561019320ed081d1bad29c75bdcd6074307 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Wed, 22 Oct 2025 14:30:07 -0400 Subject: [PATCH 3/9] Updated to better sort some sections and update for missing entries. Fixes #2108. Signed-off-by: Eric D. Schabell --- SUMMARY.md | 68 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 35 insertions(+), 33 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index 0cc20d535..6f3eeaad9 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -25,16 +25,16 @@ * [Build with static configuration](installation/downloads/source/build-with-static-configuration.md) * [Linux packages](installation/downloads/linux.md) * [Amazon Linux](installation/downloads/linux/amazon-linux.md) - * [Rocky Linux and Alma Linux ](installation/downloads/linux/alma-rocky.md) - * [Red Hat and CentOS](installation/downloads/linux/redhat-centos.md) + * [Buildroot embedded Linux](installation/downloads/linux/buildroot-embedded-linux.md) * [Debian](installation/downloads/linux/debian.md) - * [Ubuntu](installation/downloads/linux/ubuntu.md) * [Raspbian and Raspberry Pi](installation/downloads/linux/raspbian-raspberry-pi.md) + * [Red Hat and CentOS](installation/downloads/linux/redhat-centos.md) + * [Rocky Linux and Alma Linux ](installation/downloads/linux/alma-rocky.md) + * [Ubuntu](installation/downloads/linux/ubuntu.md) * [Yocto embedded Linux](installation/downloads/linux/yocto-embedded-linux.md) - * [Buildroot embedded Linux](installation/downloads/linux/buildroot-embedded-linux.md) - * [Docker](installation/downloads/docker.md) - * [Containers on AWS](installation/downloads/aws-container.md) * [Amazon EC2](installation/downloads/amazon-ec2.md) + * [Containers on AWS](installation/downloads/aws-container.md) + * [Docker](installation/downloads/docker.md) * [Kubernetes](installation/downloads/kubernetes.md) * [macOS](installation/downloads/macos.md) * [Windows](installation/downloads/windows.md) @@ -44,36 +44,37 @@ * [Configure Fluent Bit](administration/configuring-fluent-bit.md) * [YAML configuration](administration/configuring-fluent-bit/yaml.md) + * [Configuration file](administration/configuring-fluent-bit/yaml/configuration-file.md) + * [Environment variables](administration/configuring-fluent-bit/yaml/environment-variables-section.md) + * [Includes](administration/configuring-fluent-bit/yaml/includes-section.md) * [Service](administration/configuring-fluent-bit/yaml/service-section.md) * [Parsers](administration/configuring-fluent-bit/yaml/parsers-section.md) * [Multiline parsers](administration/configuring-fluent-bit/yaml/multiline-parsers-section.md) * [Pipeline](administration/configuring-fluent-bit/yaml/pipeline-section.md) * [Plugins](administration/configuring-fluent-bit/yaml/plugins-section.md) * [Upstream servers](administration/configuring-fluent-bit/yaml/upstream-servers-section.md) - * [Environment variables](administration/configuring-fluent-bit/yaml/environment-variables-section.md) - * [Includes](administration/configuring-fluent-bit/yaml/includes-section.md) * [Classic mode](administration/configuring-fluent-bit/classic-mode.md) - * [Format and schema](administration/configuring-fluent-bit/classic-mode/format-schema.md) - * [Configuration file](administration/configuring-fluent-bit/classic-mode/configuration-file.md) - * [Variables](administration/configuring-fluent-bit/classic-mode/variables.md) + * [Configuration file](administration/configuring-fluent-bit/classic-mode/configuration-file.md) * [Commands](administration/configuring-fluent-bit/classic-mode/commands.md) + * [Format and schema](administration/configuring-fluent-bit/classic-mode/format-schema.md) + * [Record accessor syntax](administration/configuring-fluent-bit/classic-mode/record-accessor.md) * [Upstream servers](administration/configuring-fluent-bit/classic-mode/upstream-servers.md) - * [Record accessor syntax](administration/configuring-fluent-bit/classic-mode/record-accessor.md) - * [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md) + * [Variables](administration/configuring-fluent-bit/classic-mode/variables.md) * [Multiline parsing](administration/configuring-fluent-bit/multiline-parsing.md) -* [TLS](administration/transport-security.md) -* [Buffering and storage](administration/buffering-and-storage.md) + * [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md) +* [AWS credentials](administration/aws-credentials.md) * [Backpressure](administration/backpressure.md) -* [Scheduling and retries](administration/scheduling-and-retries.md) -* [Networking](administration/networking.md) +* [Buffering and storage](administration/buffering-and-storage.md) +* [Hot reload](administration/hot-reload.md) +* [HTTP proxy](administration/http-proxy.md) * [Memory management](administration/memory-management.md) * [Monitoring](administration/monitoring.md) * [Multithreading](administration/multithreading.md) -* [HTTP proxy](administration/http-proxy.md) -* [Hot reload](administration/hot-reload.md) -* [Troubleshooting](administration/troubleshooting.md) +* [Networking](administration/networking.md) * [Performance tips](administration/performance.md) -* [AWS credentials](administration/aws-credentials.md) +* [Scheduling and retries](administration/scheduling-and-retries.md) +* [TLS](administration/transport-security.md) +* [Troubleshooting](administration/troubleshooting.md) ## Local testing @@ -91,8 +92,8 @@ * [Dummy](pipeline/inputs/dummy.md) * [eBPF](pipeline/inputs/ebpf.md) * [Elasticsearch](pipeline/inputs/elasticsearch.md) - * [Exec WASI](pipeline/inputs/exec-wasi.md) * [Exec](pipeline/inputs/exec.md) + * [Exec WASI](pipeline/inputs/exec-wasi.md) * [Fluent Bit metrics](pipeline/inputs/fluentbit-metrics.md) * [Forward](pipeline/inputs/forward.md) * [Head](pipeline/inputs/head.md) @@ -124,25 +125,25 @@ * [TCP](pipeline/inputs/tcp.md) * [Thermal](pipeline/inputs/thermal.md) * [UDP](pipeline/inputs/udp.md) - * [Windows Event logs (winevtlog)](pipeline/inputs/windows-event-log-winevtlog.md) * [Windows Event logs (winlog)](pipeline/inputs/windows-event-log.md) + * [Windows Event logs (winevtlog)](pipeline/inputs/windows-event-log-winevtlog.md) * [Windows exporter metrics](pipeline/inputs/windows-exporter-metrics.md) * [Parsers](pipeline/parsers.md) * [Configuring parsers](pipeline/parsers/configuring-parser.md) + * [Decoders](pipeline/parsers/decoders.md) * [JSON](pipeline/parsers/json.md) - * [Regular expression](pipeline/parsers/regular-expression.md) - * [LTSV](pipeline/parsers/ltsv.md) * [Logfmt](pipeline/parsers/logfmt.md) - * [Decoders](pipeline/parsers/decoders.md) + * [LTSV](pipeline/parsers/ltsv.md) + * [Regular expression](pipeline/parsers/regular-expression.md) * [Processors](pipeline/processors.md) + * [Conditional processing](pipeline/processors/conditional-processing.md) * [Content modifier](pipeline/processors/content-modifier.md) + * [Filters as processors](pipeline/processors/filters.md) * [Labels](pipeline/processors/labels.md) * [Metrics selector](pipeline/processors/metrics-selector.md) * [OpenTelemetry envelope](pipeline/processors/opentelemetry-envelope.md) * [Sampling](pipeline/processors/sampling.md) * [SQL](pipeline/processors/sql.md) - * [Filters as processors](pipeline/processors/filters.md) - * [Conditional processing](pipeline/processors/conditional-processing.md) * [Filters](pipeline/filters.md) * [AWS metadata](pipeline/filters/aws-metadata.md) * [CheckList](pipeline/filters/checklist.md) @@ -169,15 +170,15 @@ * [Buffering](pipeline/buffering.md) * [Routing](pipeline/router.md) * [Outputs](pipeline/outputs.md) + * [Azure Log Analytics](pipeline/outputs/azure.md) + * [Azure Blob](pipeline/outputs/azure_blob.md) + * [Azure Data Explorer](pipeline/outputs/azure_kusto.md) + * [Azure Logs Ingestion API](pipeline/outputs/azure_logs_ingestion.md) * [Amazon CloudWatch](pipeline/outputs/cloudwatch.md) * [Amazon Kinesis Data Firehose](pipeline/outputs/firehose.md) * [Amazon Kinesis Data Streams](pipeline/outputs/kinesis.md) * [Amazon S3](pipeline/outputs/s3.md) * [Apache SkyWalking](pipeline/outputs/skywalking.md) - * [Azure Blob](pipeline/outputs/azure_blob.md) - * [Azure Data Explorer](pipeline/outputs/azure_kusto.md) - * [Azure Log Analytics](pipeline/outputs/azure.md) - * [Azure Logs Ingestion API](pipeline/outputs/azure_logs_ingestion.md) * [Counter](pipeline/outputs/counter.md) * [Dash0](pipeline/outputs/dash0.md) * [Datadog](pipeline/outputs/datadog.md) @@ -199,10 +200,10 @@ * [New Relic](pipeline/outputs/new-relic.md) * [Null](pipeline/outputs/null.md) * [Observe](pipeline/outputs/observe.md) + * [Oracle Cloud Infrastructure Logging Analytics](pipeline/outputs/oci-logging-analytics.md) * [OpenObserve](pipeline/outputs/openobserve.md) * [OpenSearch](pipeline/outputs/opensearch.md) * [OpenTelemetry](pipeline/outputs/opentelemetry.md) - * [Oracle Cloud Infrastructure Logging Analytics](pipeline/outputs/oci-logging-analytics.md) * [Output Formats](pipeline/outputs/output_formats.md) * [Parseable](pipeline/outputs/parseable.md) * [PostgreSQL](pipeline/outputs/postgresql.md) @@ -234,6 +235,7 @@ * [C library API](development/library_api.md) * [Ingest records manually](development/ingest-records-manually.md) * [Golang output plugins](development/golang-output-plugins.md) +* [MessagePack format](development/msgpack-format.md) * [Wasm filter plugins](development/wasm-filter-plugins.md) * [Wasm input plugins](development/wasm-input-plugins.md) * [Contribution guide for beginners](development/developer-guide.md) From e4ea1ccc78f3c23d2356ba8d1a34e1f2688ae995 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Thu, 23 Oct 2025 15:09:44 -0400 Subject: [PATCH 4/9] Updated configuration variables for opentelemetry output plugin, many new ones added. Signed-off-by: Eric D. Schabell --- pipeline/outputs/opentelemetry.md | 126 ++++++++++++++++++++---------- 1 file changed, 83 insertions(+), 43 deletions(-) diff --git a/pipeline/outputs/opentelemetry.md b/pipeline/outputs/opentelemetry.md index ecc09cc63..4cc103048 100644 --- a/pipeline/outputs/opentelemetry.md +++ b/pipeline/outputs/opentelemetry.md @@ -1,54 +1,93 @@ ---- -description: An output plugin to submit Logs, Metrics, or Traces to an OpenTelemetry endpoint ---- +— -# OpenTelemetry +== description: An output plugin to submit Logs, Metrics, or Traces to an OpenTelemetry endpoint + += OpenTelemetry The OpenTelemetry plugin lets you take logs, metrics, and traces from Fluent Bit and submit them to an OpenTelemetry HTTP endpoint. Only HTTP endpoints are supported. -| Key | Description | Default | -|-----|-------------|---------| -| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | -| `http_user` | Basic Auth username. | _none_ | -| `http_passwd` | Basic Auth password. Requires `HTTP_user` to be set. | _none_ | -| `port` | TCP port of the target HTTP server. | `80` | -| `proxy` | Specify an HTTP Proxy. The expected format value is `http://HOST:PORT`. HTTPS isn't supported. It's recommended to configure the [HTTP proxy environment variables](https://docs.fluentbit.io/manual/administration/http-proxy) instead as they support both HTTP and HTTPS. | _none_ | -| `http2` | Defines whether HTTP/2 protocol is enabled. This setting also supports the `force` option, which forces HTTP/2 over a plain text connection. | `On` | -| `grpc` | Enables gRPC over an HTTP/2 connection. This setting applies to HTTP/2 only. | `off` | -| `metrics_uri` | Specify an optional HTTP URI for the target web server listening for metrics. For example, `/v1/metrics`. | `/` | -| `logs_uri` | Specify an optional HTTP URI for the target web server listening for logs, For example, `/v1/logs`. | `/` | -| `traces_uri` | Specify an optional HTTP URI for the target web server listening for traces, For example, `/v1/traces`. | `/` | -| `header` | Add a HTTP header key/value pair. Multiple headers can be set. | _none_ | -| `log_response_payload` | Log the response payload within the Fluent Bit log. | `false` | -| `logs_body_key` | The log body key to look up in the log events body/message. Sets the `Body` field of the OpenTelemetry logs data model. | `message` | -| `logs_trace_id_message_key` | The trace id key to look up in the log events body/message. Sets the `TraceId` field of the OpenTelemetry logs data model. | `traceId` | -| `logs_span_id_message_key` | The span id key to look up in the log events body/message. Sets the `SpanId` field of the OpenTelemetry logs data model. | `spanId` | -| `logs_severity_text_message_key` | The severity text id key to look up in the log events body/message. Sets the `SeverityText` field of the OpenTelemetry logs data model. | `severityText` | -| `logs_severity_number_message_key` | The severity number id key to look up in the log events body/message. Sets the `SeverityNumber` field of the OpenTelemetry logs data model. | `severityNumber` | -| `add_label` | Lets you add custom labels to all metrics and logs (resource attributes) exposed through the OpenTelemetry exporter. You can have multiple `add_label` fields for each label you want to add. | _none_ | -| `compress` | Set payload compression mechanism. Allowed value: `gzip`. | _none_ | -| `logs_observed_timestamp_metadata_key` | Specify an `ObservedTimestamp` key to look up in the metadata. | `$ObservedKey` | -| `logs_timestamp_metadata_key` | Specify a `Timestamp` key to look up in the metadata. | `$Timestamp` | -| `logs_severity_key_metadata_key` | Specify a `SeverityText` key to look up in the metadata. | `$SeverityText` | -| `logs_severity_number_metadata_key` | Specify a `SeverityNumber` key to look up in the metadata. | `$SeverityNumber` | -| `logs_trace_flags_metadata_key` | Specify a `Flags` key to look up in the metadata. | `$Flags` | -| `logs_span_id_metadata_key` | Specify a `SpanId` key to look up in the metadata. | `$SpanId` | -| `logs_trace_id_metadata_key` | Specify a `TraceId` key to look up in the metadata. | `$TraceId` | -| `logs_attributes_metadata_key` | Specify an `Attributes` key to look up in the metadata. | `$Attributes` | -| `logs_body_key_attributes` | When enabled (`true`), Fluent Bit will place all parsed JSON fields (except the one mapped by `logs_body_key`) inside the `attributes` section of the OpenTelemetry `LogRecord`. This is useful if you want structured key/value pairs as attributes, while still mapping a specific field (defined by `logs_body_key`) to the `body`. | `false` | -| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | - -## Get started + +| Key | Description | Default | +|-------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------| +| `add_label` | Lets you add custom labels to all metrics and logs (resource attributes) exposed through the OpenTelemetry exporter. You can have multiple `add_label` fields for each label you want to add. | _none_ | +| `alias` | Sets an alias for the output instance. This is useful when using multiple instances of the same output plugin. If no alias is set, the instance will be named using the plugin name and a sequence number. | _none_ | +| `aws_auth` | Enable AWS SigV4 authentication. | `false` | +| `aws_external_id` | Specify an external ID for the STS API, can be used with the `aws_role_arn` parameter if your role requires an external ID. | _none_ | +| `aws_profile` | AWS Profile name. AWS Profiles can be configured with AWS CLI and are usually stored in $HOME/.aws/ directory. | _none_ | +| `aws_region` | AWS region of your service. | _none_ | +| `aws_role_arn` | ARN of an IAM role to assume (ex. for cross account access). | _none_ | +| `aws_service` | AWS destination service code, used by SigV4 authentication. | `logs` | +| `aws_sts_endpoint` | Custom endpoint for the AWS STS API, used with the `aws_role_arn` option. | _none_ | +| `batch_size` | Set the maximum number of log records to be flushed at a time. | `1000` | +| `compress` | Set payload compression mechanism. Options available are `gzip` and `zstd`. | _none_ | +| `grpc` | Enables gRPC over an HTTP/2 connection. Supports `auto` option. This setting applies to HTTP/2 only. | `off` | +| `grpc_metrics_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export` | +| `grpc_logs_uri` | Specify an optional gRPCß URI for the target OTel endpoint. | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | +| `grpc_traces_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` | +| `grpc_profiles_uri` | Specify an optional gRPC URI for the profiles OTel endpoint. | `/opentelemetry.proto.collector.profiles.v1experimental.ProfilesService/Export` | +| `header` | Add a HTTP header key/value pair. Multiple headers can be set. | _none_ | +| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | +| `http2` | Defines whether HTTP/2 protocol is enabled. This setting also supports the `force` option, which forces HTTP/2 over a plain text connection. | `off` | +| `http_passwd` | Basic Auth password. Requires `HTTP_user` to be set. | _none_ | +| `http_user` | Basic Auth username. | _none_ | +| `logs_attributes_metadata_key` | Specify an `Attributes` key to look up in the metadata. | `$Attributes` | +| `log_level` | Specifies the log level for this output plugin. If not set, the plugin will use the global log level defined in the `service` section. If the global log level is also not specified, it uses default. | `info` | +| `log_suppress_interval` | Allows suppression of repetitive log messages from the output plugin that appear similar within a specified time interval. Default is no suppression. | `0` | +| `logs_body_key` | The log body key to look up in the log events body/message. Sets the `Body` field of the OpenTelemetry logs data model. | `message` | +| `logs_body_key_attributes` | When enabled (`true`), Fluent Bit will place all parsed JSON fields (except the one mapped by `logs_body_key`) inside the `attributes` section of the OpenTelemetry `LogRecord`. This is useful if you want structured key/value pairs as attributes, while still mapping a specific field (defined by `logs_body_key`) to the `body`. | `false` | +| `logs_instrumentation_scope_metadata_key` | Specify an `InstrumentationScope` key. | `InstrumentationScope` | +| `logs_metadata_key` | Set the key to lookup in the metadata. | `otlp` | +| `logs_observed_timestamp_metadata_key` | Specify an `ObservedTimestamp` key to look up in the metadata. | `$ObservedKey` | +| `logs_resource_metadata_key` | Specify a `Resource` key. | `Resource` | +| `log_response_payload` | Log the response payload within the Fluent Bit log. | `false` | +| `log_severity_text_metadata_key` | Specify a SeverityText key. | `$SeverityText` | +| `logs_severity_number_metadata_key` | Specify a `SeverityNumber` key to look up in the metadata. | `$SeverityNumber` | +| `logs_severity_number_message_key` | The severity number id key to look up in the log events body/message. Sets the `SeverityNumber` field of the OpenTelemetry logs data model. | `$severityNumber` | +| `logs_severity_text_message_key` | The severity text id key to look up in the log events body/message. Sets the `SeverityText` field of the OpenTelemetry logs data model. | `$SeverityText` | +| `logs_span_id_message_key` | The span id key to look up in the log events body/message. Sets the `SpanId` field of the OpenTelemetry logs data model. | `$SpanId` | +| `logs_span_id_metadata_key` | Specify a `SpanId` key to look up in the metadata. | `$SpanId` | +| `logs_timestamp_metadata_key` | Specify a `Timestamp` key to look up in the metadata. | `$Timestamp` | +| `logs_trace_flags_metadata_key` | Specify a `TraceFlags` key to look up in the metadata. | `$TraceFlags` | +| `logs_trace_id_message_key` | The trace id key to look up in the log events body/message. Sets the `TraceId` field of the OpenTelemetry logs data model. | `$TraceId` | +| `logs_trace_id_metadata_key` | Specify a `TraceId` key to look up in the metadata. | `$TraceId` | +| `logs_uri` | Specify an optional HTTP URI for the target web server listening for logs, For example, `/v1/logs`. | _none_ | +| `match` | Set a tag pattern to match the records that this output should process. Supports exact matches or wildcards (for example '*'). | `default` | +| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to simple wildcards. | `default` | +| `metrics_uri` | Specify an optional HTTP URI for the target web server listening for metrics. For example, `/v1/metrics`. | `/` | +| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | +| `port` | TCP port of the target HTTP server. | `80` | +| `profiles_uri` | Specify an optional HTTP URI for the profiles OTel endpoint. | `/v1development/profiles` | +| `proxy` | Specify an HTTP Proxy. The expected format value is `http://HOST:PORT`. HTTPS isn't supported. It's recommended to configure the [HTTP proxy environment variables](https://docs.fluentbit.io/manual/administration/http-proxy) instead as they support both HTTP and HTTPS. | _none_ | +| `retry_limit` | Set the retry limit for the output plugin when delivery fails. Accepted values: a positive integer, `no_limits`, `false`, or `off` to disable retry limits, or `no_retries` to disable retries entirely. | `1` | +| `tls` | Enable or disable TLS/SSL support. | `off` | +| `tls.ca_file` | Absolute path to CA certificate file. | `default` | +| `tls.ca_path` | Absolute path to scan for certificate files. | `default` | +| `tls.cert_file` | Absolute path to Certificate file. | `default` | +| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | `default` | +| `tls.debug` | Set TLS debug verbosity level. It accept the following values: `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | +| `tls.key_file` | Absolute path to private Key file. | `default` | +| `tls.key_passwd` | Optional password for tls.key_file file. | `default` | +| `tls.max_version` | Specify the maximum version of TLS. | `default` | +| `tls.min_version` | Specify the minimum version of TLS. | `default` | +| `tls.verify` | Force certificate validation. | `on` | +| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | +| `tls.vhost` | Hostname to be used for TLS SNI extension. | `default` | +| `tls.windows.certstore_name` | Sets the certstore name on an output (Windows). | `default` | +| `tls.windows.use_enterprise_store` | Sets whether using enterprise certstore or not on an output (Windows). | `default` | +| `traces_uri` | Specify an optional HTTP URI for the target web server listening for traces, For example, `/v1/traces`. | `/` | + + +== Get started The OpenTelemetry plugin works with logs and only the metrics collected from one of the metric input plugins. In the following example, log records generated by the dummy plugin and the host metrics collected by the node exporter metrics plugin are exported by the OpenTelemetry output plugin. {% tabs %} {% tab title="fluent-bit.yaml" %} -```yaml - +[source,yaml] +---- # Dummy Logs and traces with Node Exporter Metrics export using OpenTelemetry output plugin # ------------------------------------------- # The following example collects host metrics on Linux and dummy logs and traces and delivers @@ -73,7 +112,7 @@ pipeline: outputs: - name: opentelemetry - match: '*' + match: "*" host: localhost port: 443 metrics_uri: /v1/metrics @@ -91,12 +130,13 @@ pipeline: add_label: - app fluent-bit - color blue -``` +---- {% endtab %} {% tab title="fluent-bit.conf" %} -```text +[source,text] +---- # Dummy Logs and traces with Node Exporter Metrics export using OpenTelemetry output plugin # ------------------------------------------- # The following example collects host metrics on Linux and dummy logs and traces and delivers @@ -139,7 +179,7 @@ pipeline: # add user-defined labels add_label app fluent-bit add_label color blue -``` +---- {% endtab %} {% endtabs %} From f8ff78d500939668019e0ff977aeefd02caf9590 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Thu, 23 Oct 2025 15:21:45 -0400 Subject: [PATCH 5/9] Updated configuration variables for opentelemetry output plugin, many new ones added. Fixes #2083 Signed-off-by: Eric D. Schabell --- pipeline/outputs/opentelemetry.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/pipeline/outputs/opentelemetry.md b/pipeline/outputs/opentelemetry.md index 4cc103048..f0dae7dfc 100644 --- a/pipeline/outputs/opentelemetry.md +++ b/pipeline/outputs/opentelemetry.md @@ -56,7 +56,20 @@ Only HTTP endpoints are supported. | `match` | Set a tag pattern to match the records that this output should process. Supports exact matches or wildcards (for example '*'). | `default` | | `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to simple wildcards. | `default` | | `metrics_uri` | Specify an optional HTTP URI for the target web server listening for metrics. For example, `/v1/metrics`. | `/` | +| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | +| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | +| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it is retried. | `2000` | | `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | +| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | +| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | _none_ | +| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | +| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | +| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | +| `net.source_address` | Specify network address to bind for data traffic. | _none_ | +| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | +| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | +| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a keepidle probe. | `-1` | +| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | | `port` | TCP port of the target HTTP server. | `80` | | `profiles_uri` | Specify an optional HTTP URI for the profiles OTel endpoint. | `/v1development/profiles` | | `proxy` | Specify an HTTP Proxy. The expected format value is `http://HOST:PORT`. HTTPS isn't supported. It's recommended to configure the [HTTP proxy environment variables](https://docs.fluentbit.io/manual/administration/http-proxy) instead as they support both HTTP and HTTPS. | _none_ | From 8b007c047fe9c68693474f57655fcc9703587274 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Fri, 24 Oct 2025 14:55:19 -0400 Subject: [PATCH 6/9] Fixes reqeusted by reviewers for options table for opentelemetry output plugin. Signed-off-by: Eric D. Schabell --- pipeline/outputs/opentelemetry.md | 165 +++++++++++++++--------------- 1 file changed, 83 insertions(+), 82 deletions(-) diff --git a/pipeline/outputs/opentelemetry.md b/pipeline/outputs/opentelemetry.md index f0dae7dfc..f45be2bb1 100644 --- a/pipeline/outputs/opentelemetry.md +++ b/pipeline/outputs/opentelemetry.md @@ -9,88 +9,89 @@ The OpenTelemetry plugin lets you take logs, metrics, and traces from Fluent Bit Only HTTP endpoints are supported. -| Key | Description | Default | -|-------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------| -| `add_label` | Lets you add custom labels to all metrics and logs (resource attributes) exposed through the OpenTelemetry exporter. You can have multiple `add_label` fields for each label you want to add. | _none_ | -| `alias` | Sets an alias for the output instance. This is useful when using multiple instances of the same output plugin. If no alias is set, the instance will be named using the plugin name and a sequence number. | _none_ | -| `aws_auth` | Enable AWS SigV4 authentication. | `false` | -| `aws_external_id` | Specify an external ID for the STS API, can be used with the `aws_role_arn` parameter if your role requires an external ID. | _none_ | -| `aws_profile` | AWS Profile name. AWS Profiles can be configured with AWS CLI and are usually stored in $HOME/.aws/ directory. | _none_ | -| `aws_region` | AWS region of your service. | _none_ | -| `aws_role_arn` | ARN of an IAM role to assume (ex. for cross account access). | _none_ | -| `aws_service` | AWS destination service code, used by SigV4 authentication. | `logs` | -| `aws_sts_endpoint` | Custom endpoint for the AWS STS API, used with the `aws_role_arn` option. | _none_ | -| `batch_size` | Set the maximum number of log records to be flushed at a time. | `1000` | -| `compress` | Set payload compression mechanism. Options available are `gzip` and `zstd`. | _none_ | -| `grpc` | Enables gRPC over an HTTP/2 connection. Supports `auto` option. This setting applies to HTTP/2 only. | `off` | -| `grpc_metrics_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export` | -| `grpc_logs_uri` | Specify an optional gRPCß URI for the target OTel endpoint. | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | -| `grpc_traces_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` | -| `grpc_profiles_uri` | Specify an optional gRPC URI for the profiles OTel endpoint. | `/opentelemetry.proto.collector.profiles.v1experimental.ProfilesService/Export` | -| `header` | Add a HTTP header key/value pair. Multiple headers can be set. | _none_ | -| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | -| `http2` | Defines whether HTTP/2 protocol is enabled. This setting also supports the `force` option, which forces HTTP/2 over a plain text connection. | `off` | -| `http_passwd` | Basic Auth password. Requires `HTTP_user` to be set. | _none_ | -| `http_user` | Basic Auth username. | _none_ | -| `logs_attributes_metadata_key` | Specify an `Attributes` key to look up in the metadata. | `$Attributes` | -| `log_level` | Specifies the log level for this output plugin. If not set, the plugin will use the global log level defined in the `service` section. If the global log level is also not specified, it uses default. | `info` | -| `log_suppress_interval` | Allows suppression of repetitive log messages from the output plugin that appear similar within a specified time interval. Default is no suppression. | `0` | -| `logs_body_key` | The log body key to look up in the log events body/message. Sets the `Body` field of the OpenTelemetry logs data model. | `message` | -| `logs_body_key_attributes` | When enabled (`true`), Fluent Bit will place all parsed JSON fields (except the one mapped by `logs_body_key`) inside the `attributes` section of the OpenTelemetry `LogRecord`. This is useful if you want structured key/value pairs as attributes, while still mapping a specific field (defined by `logs_body_key`) to the `body`. | `false` | -| `logs_instrumentation_scope_metadata_key` | Specify an `InstrumentationScope` key. | `InstrumentationScope` | -| `logs_metadata_key` | Set the key to lookup in the metadata. | `otlp` | -| `logs_observed_timestamp_metadata_key` | Specify an `ObservedTimestamp` key to look up in the metadata. | `$ObservedKey` | -| `logs_resource_metadata_key` | Specify a `Resource` key. | `Resource` | -| `log_response_payload` | Log the response payload within the Fluent Bit log. | `false` | -| `log_severity_text_metadata_key` | Specify a SeverityText key. | `$SeverityText` | -| `logs_severity_number_metadata_key` | Specify a `SeverityNumber` key to look up in the metadata. | `$SeverityNumber` | -| `logs_severity_number_message_key` | The severity number id key to look up in the log events body/message. Sets the `SeverityNumber` field of the OpenTelemetry logs data model. | `$severityNumber` | -| `logs_severity_text_message_key` | The severity text id key to look up in the log events body/message. Sets the `SeverityText` field of the OpenTelemetry logs data model. | `$SeverityText` | -| `logs_span_id_message_key` | The span id key to look up in the log events body/message. Sets the `SpanId` field of the OpenTelemetry logs data model. | `$SpanId` | -| `logs_span_id_metadata_key` | Specify a `SpanId` key to look up in the metadata. | `$SpanId` | -| `logs_timestamp_metadata_key` | Specify a `Timestamp` key to look up in the metadata. | `$Timestamp` | -| `logs_trace_flags_metadata_key` | Specify a `TraceFlags` key to look up in the metadata. | `$TraceFlags` | -| `logs_trace_id_message_key` | The trace id key to look up in the log events body/message. Sets the `TraceId` field of the OpenTelemetry logs data model. | `$TraceId` | -| `logs_trace_id_metadata_key` | Specify a `TraceId` key to look up in the metadata. | `$TraceId` | -| `logs_uri` | Specify an optional HTTP URI for the target web server listening for logs, For example, `/v1/logs`. | _none_ | -| `match` | Set a tag pattern to match the records that this output should process. Supports exact matches or wildcards (for example '*'). | `default` | -| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to simple wildcards. | `default` | -| `metrics_uri` | Specify an optional HTTP URI for the target web server listening for metrics. For example, `/v1/metrics`. | `/` | -| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | -| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | -| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it is retried. | `2000` | -| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | -| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | -| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | _none_ | -| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | -| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | -| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | -| `net.source_address` | Specify network address to bind for data traffic. | _none_ | -| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | -| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | -| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a keepidle probe. | `-1` | -| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | -| `port` | TCP port of the target HTTP server. | `80` | -| `profiles_uri` | Specify an optional HTTP URI for the profiles OTel endpoint. | `/v1development/profiles` | -| `proxy` | Specify an HTTP Proxy. The expected format value is `http://HOST:PORT`. HTTPS isn't supported. It's recommended to configure the [HTTP proxy environment variables](https://docs.fluentbit.io/manual/administration/http-proxy) instead as they support both HTTP and HTTPS. | _none_ | -| `retry_limit` | Set the retry limit for the output plugin when delivery fails. Accepted values: a positive integer, `no_limits`, `false`, or `off` to disable retry limits, or `no_retries` to disable retries entirely. | `1` | -| `tls` | Enable or disable TLS/SSL support. | `off` | -| `tls.ca_file` | Absolute path to CA certificate file. | `default` | -| `tls.ca_path` | Absolute path to scan for certificate files. | `default` | -| `tls.cert_file` | Absolute path to Certificate file. | `default` | -| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | `default` | -| `tls.debug` | Set TLS debug verbosity level. It accept the following values: `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | -| `tls.key_file` | Absolute path to private Key file. | `default` | -| `tls.key_passwd` | Optional password for tls.key_file file. | `default` | -| `tls.max_version` | Specify the maximum version of TLS. | `default` | -| `tls.min_version` | Specify the minimum version of TLS. | `default` | -| `tls.verify` | Force certificate validation. | `on` | -| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | -| `tls.vhost` | Hostname to be used for TLS SNI extension. | `default` | -| `tls.windows.certstore_name` | Sets the certstore name on an output (Windows). | `default` | -| `tls.windows.use_enterprise_store` | Sets whether using enterprise certstore or not on an output (Windows). | `default` | -| `traces_uri` | Specify an optional HTTP URI for the target web server listening for traces, For example, `/v1/traces`. | `/` | - +| Key | Description | Default | +|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------| +| `add_label` | Adds a custom label to the metrics use format: `add_label name value`. | _none_ | +| `alias` | Sets an alias, useful for multiple instances of the same output plugin. | _none_ | +| `aws_auth` | Enable AWS SigV4 authentication. | `false` | +| `aws_external_id` | Specify an external ID for the STS API, can be used with the `aws_role_arn` parameter. | _none_ | +| `aws_profile` | AWS Profile name. AWS Profiles can be configured with AWS CLI. | _none_ | +| `aws_region` | AWS region of your service. | _none_ | +| `aws_role_arn` | ARN of an IAM role to assume (ex. for cross account access). | _none_ | +| `aws_service` | AWS destination service code, used by SigV4 authentication. | `logs` | +| `aws_sts_endpoint` | Custom endpoint for the AWS STS API, used with the `aws_role_arn` option. | _none_ | +| `grpc` | Enable, disable or force gRPC usage. Accepted values : `on`, `off`, `auto`. | `off` | +| `batch_size` | Set the maximum number of log records to be flushed at a time. | `1000` | +| `compress` | Set payload compression mechanism. Options available are `gzip` and `zstd`. | _none_ | +| `grpc_logs_uri` | Specify an optional gRPCß URI for the target OTel endpoint. | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` | +| `grpc_metrics_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export` | +| `grpc_profiles_uri` | Specify an optional gRPC URI for profiles OTel endpoint. | `/opentelemetry.proto.collector.profiles.v1experimental.ProfilesService/Export` | +| `grpc_traces_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` | +| `header` | Add a HTTP header key/value pair. Multiple headers can be set. | _none_ | +| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | +| `http2` | Enable, disable or force HTTP/2 usage. Accepted values : `on`, `off`, or `force`. | `off` | +| `http_passwd` | Set HTTP auth password. | _none_ | +| `http_user` | Set HTTP auth user. | _none_ | +| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | +| `logs_attributes_metadata_key` | Specify an `Attributes` key. | `$Attributes` | +| `logs_body_key` | Specify an optional HTTP URI for the target OTel endpoint. | _none_ | +| `logs_body_key_attributes` | If set and it matched a pattern, it includes the remaining fields in the record as attributes. | `false` | +| `logs_instrumentation_scope_metadata_key` | Specify an `InstrumentationScope` key. | `InstrumentationScope` | +| `logs_observed_timestamp_metadata_key` | Specify an `ObservedTimestamp` key. | `$ObservedTimestamp` | +| `logs_max_resources` | Set the maximum number of OTLP log resources per export request (`0` disables the limit). | `0` | +| `logs_max_scopes` | Set the maximum number of OTLP log scopes per resource (`0` disables the limit). | `0` | +| `logs_metadata_key` | Set the key to lookup in the metadata. | `otlp` | +| `logs_resource_metadata_key` | Specify a `Resource` key. | `Resource` | +| `log_response_payload` | Specify if the response payload should be logged or not. | `true` | +| `logs_severity_number_message_key` | Specify a `SeverityNumber` key. | `$severityNumber` | +| `logs_severity_number_metadata_key` | Specify a `SeverityNumber` key. | `$SeverityNumber` | +| `logs_severity_text_message_key` | Specify a `SeverityText` key. | `$SeverityText` | +| `log_severity_text_metadata_key` | Specify a `SeverityText` key. | `$SeverityText` | +| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | +| `logs_span_id_message_key` | Specify a `SpanId` key. | `$SpanId` | +| `logs_span_id_metadata_key` | Specify a `SpanId` key. | `$SpanId` | +| `logs_timestamp_metadata_key` | Specify a `Timestamp` key. | `$Timestamp` | +| `logs_trace_flags_metadata_key` | Specify a `TraceFlags` key. | `$TraceFlags` | +| `logs_trace_id_message_key` | Specify a `TraceId` key. | `$TraceId` | +| `logs_trace_id_metadata_key` | Specify a `TraceId` key. | `$TraceId` | +| `logs_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/logs` | +| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | +| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to simple wildcards. | _none_ | +| `metrics_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/metrics` | +| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | +| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | +| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it is retried. | `2000` | +| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | +| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | +| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | _none_ | +| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | +| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | +| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | +| `net.source_address` | Specify network address to bind for data traffic. | _none_ | +| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | +| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | +| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a keepidle probe. | `-1` | +| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | +| `port` | TCP port of the target HTTP server. | `80` | +| `profiles_uri` | Specify an optional HTTP URI for the profiles OTel endpoint. | `/v1development/profiles` | +| `proxy` | Specify an HTTP Proxy. The expected format of this value is `http://host:port`. | _none_ | +| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | +| `tls` | Enable or disable TLS/SSL support. | `off` | +| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | +| `tls.ca_path` | Absolute path to scan for certificate files. | _none_ | +| `tls.cert_file` | Absolute path to Certificate file. | _none_ | +| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | +| `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | +| `tls.key_file` | Absolute path to private Key file. | _none_ | +| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | +| `tls.max_version` | Specify the maximum version of TLS. | _none_ | +| `tls.min_version` | Specify the minimum version of TLS. | _none_ | +| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | +| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | +| `tls.verify` | Force certificate validation. | `on` | +| `tls.windows.certstore_name` | Sets the certstore name on an output (Windows). | _none_ | +| `tls.windows.use_enterprise_store` | Sets whether using enterprise certstore or not on an output (Windows). | _none_ | +| `traces_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/traces` | == Get started From 12dc85fe6ec7c3f833a790b395450d79fdd12f4b Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 28 Oct 2025 12:57:12 -0400 Subject: [PATCH 7/9] Updated azure kusto output plugin docs with configuration options table available for v4.1. Fixes #2086. Signed-off-by: Eric D. Schabell --- pipeline/outputs/azure_kusto.md | 106 ++++++++++++++++++++++---------- 1 file changed, 74 insertions(+), 32 deletions(-) diff --git a/pipeline/outputs/azure_kusto.md b/pipeline/outputs/azure_kusto.md index fae3b7fd4..b9036b74e 100644 --- a/pipeline/outputs/azure_kusto.md +++ b/pipeline/outputs/azure_kusto.md @@ -38,7 +38,7 @@ Fluent Bit ingests the event data into Kusto in a JSON format. By default, the t A table with the expected schema must exist in order for data to be ingested properly. -```sql +```shell .create table FluentBit (log:dynamic, tag:string, timestamp:datetime) ``` @@ -48,37 +48,79 @@ By default, Kusto will insert incoming ingestion data into a table by inferring ## Configuration parameters -| Key | Description | Default | -| --- | ----------- | ----------- | -| `tenant_id` | Required if `managed_identity_client_id` isn't set. The tenant/domain ID of the AAD registered application. | _none_ | -| `client_id` | Required if `managed_identity_client_id` isn't set. The client ID of the AAD registered application. | _none_ | -| `client_secret` | Required if `managed_identity_client_id` isn't set. The client secret of the AAD registered application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). | _none_ | -| `managed_identity_client_id` | Required if `tenant_id`, `client_id`, and `client_secret` aren't set. The managed identity ID to authenticate with. Set to `SYSTEM` for system-assigned managed identity, or set to the MI client ID (`GUID`) for user-assigned managed identity. | _none_ | -| `ingestion_endpoint` | The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net`. | _none_ | -| `database_name` | The database name. | _none_ | -| `table_name` | The table name. | _none_ | -| `ingestion_mapping_reference` | Optional. The name of a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping) that will be used to map the ingested payload into the table columns. | _none_ | -| `log_key` | Key name of the log content. | `log` | -| `include_tag_key` | If enabled, a tag is appended to output. The key name is used `tag_key` property. | `On` | -| `tag_key` | The key name of tag. If `include_tag_key` is false, This property is ignored. | `tag` | -| `include_time_key` | If enabled, a timestamp is appended to output. The key name is used `time_key` property. | `On` | -| `time_key` | The key name of time. If `include_time_key` is false, This property is ignored. | `timestamp` | -| `ingestion_endpoint_connect_timeout` | The connection timeout of various Kusto endpoints in seconds. | `60` | -| `compression_enabled` | If enabled, sends compressed HTTP payload (gzip) to Kusto. | `true` | -| `ingestion_resources_refresh_interval` | The ingestion resources refresh interval of Kusto endpoint in seconds. | _none_ | -| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | -| `buffering_enabled` | Optional. Enable buffering into disk before ingesting into Azure Kusto. | `Off` | -| `buffer_dir` | Optional. When buffering is `On`, specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-kusto/` | -| `upload_timeout` | Optional. When buffering is `On`, specifies a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached `upload_file_size` limit. | `30m` | -| `upload_file_size`| Optional. When buffering is `On`, specifies the size of files to be uploaded in megabytes. | `200MB` | -| `azure_kusto_buffer_key` | Optional. When buffering is `On`, set the Azure Kusto buffer key which must be specified when using multiple instances of Azure Kusto output plugin and buffering is enabled. | `key` | -| `store_dir_limit_size` | Optional. When buffering is `On`, set the max size of the buffer directory. | `8GB` | -| `buffer_file_delete_early` | Optional. When buffering is `On`, whether to delete the buffered file early after successful blob creation. | `Off` | -| `unify_tag` | Optional. This creates a single buffer file when the buffering mode is `On`. | `On` | -| `blob_uri_length` | Optional. Set the length of generated blob URI before ingesting to Kusto. | `64` | -| `scheduler_max_retries` | Optional. When buffering is `On`, set the maximum number of retries for ingestion using the scheduler. | `3` | -| `delete_on_max_upload_error` | Optional. When buffering is `On`, whether to delete the buffer file on maximum upload errors. | `Off` | -| `io_timeout` | Optional. Configure the HTTP IO timeout for uploads. | `60s` | +| Key | Description | Default | +|----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------| +| `alias` | Sets an alias, useful for multiple instances of the same output plugin. | _none_ | +| `auth_type` | Set the authentication type: `service_principal`, `managed_identity`, or `workload_identity`. For `managed_identity`, use `system` as `client_id` for `system-assigned identity`, or specify the managed identity's client ID. | `service_principal` | +| `azure_kusto_buffer_key` | When buffering is `true`, set the Azure Kusto buffer key which must be specified when using multiple instances of Azure Kusto output plugin and buffering is enabled. | `key` | +| `blob_uri_length` | Set the length of generated blob URI before ingesting to Kusto. | `64` | +| `buffer_dir` | When buffering is `true`, specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-kusto/` | +| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Kusto. | `false` | +| `buffer_file_delete_early` | When buffering is `true`, whether to delete the buffered file early after successful blob creation. | `false` | +| `unify_tag` | Optional. This creates a single buffer file when the buffering mode is `On`. | `On` | +| `client_id` | Required if `managed_identity_client_id` isn't set. The client ID of the AAD registered application. | _none_ | +| `client_secret` | Set the client secret (Application Password) of the AAD application used for authentication. | _none_ | +| `compression_enabled` | If enabled, sends compressed HTTP payload (gzip) to Kusto. | `true` | +| `database_name` | The database name. | _none_ | +| `delete_on_max_upload_error` | When buffering is `true`, whether to delete the buffer file on maximum upload errors. | `false` | +| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | +| `io_timeout` | Configure the HTTP IO timeout for uploads. | `60s` | +| `include_tag_key` | If enabled, a tag is appended to output. The key name is used `tag_key` property. | `true` | +| `include_time_key` | If enabled, a timestamp is appended to output. The key name is used `time_key` property. | `true` | +| `ingestion_endpoint` | The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net`. | _none_ | +| `ingestion_endpoint_connect_timeout` | The connection timeout of various Kusto endpoints in seconds. | `60` | +| `ingestion_mapping_reference` | The name of a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping) that will be used to map the ingested payload into the table columns. | _none_ | +| `ingestion_resources_refresh_interval` | Set the azure kusto ingestion resources refresh interval. | `3600` | +| `log_key` | Key name of the log content. | `log` | +| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | +| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | +| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | +| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to simple wildcards. | _none_ | +| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | +| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | +| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | +| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | `false` | +| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | `false` | +| `net.dns.resolver` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | +| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | +| `net.keepalive` | Enable or disable `Keepalive` support. | `false` | +| `net.keepalive_idle_timeout` | Set maximum time allowed for an idle `Keepalive` connection.. | `false` | +| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | +| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | +| `net.source_address` | Specify network address to bind for data traffic. | _none_ | +| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | +| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a keepidle probe. | `-1` | +| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it is retried. | `2000` | +| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | +| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | +| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | +| `port` | TCP port of the target HTTP server. | `0` | +| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | +| `scheduler_max_retries` | Optional. When buffering is `On`, set the maximum number of retries for ingestion using the scheduler. | `3` | +| `store_dir_limit_size` | When buffering is `true`, set the max size of the buffer directory. | `8G` | +| `tag_key` | The key name of tag. If `include_tag_key` is false, This property is ignored. | `tag` | +| `table_name` | The table name. | _none_ | +| `tenant_id` | Required if `managed_identity_client_id` isn't set. The tenant/domain ID of the AAD registered application. | _none_ | +| `time_key` | The key name of time. If `include_time_key` is false, this property is ignored. | `timestamp` | +| `tls` | Enable or disable TLS/SSL support. | `off` | +| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | +| `tls.ca_path` | Absolute path to scan for certificate files. | _none_ | +| `tls.cert_file` | Absolute path to Certificate file. | _none_ | +| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | +| `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | +| `tls.key_file` | Absolute path to private Key file. | _none_ | +| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | +| `tls.max_version` | Specify the maximum version of TLS. | _none_ | +| `tls.min_version` | Specify the minimum version of TLS. | _none_ | +| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | +| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | +| `tls.verify` | Force certificate validation. | `on` | +| `tls.windows.certstore_name` | Sets the certstore name on an output (Windows). | _none_ | +| `tls.windows.use_enterprise_store` | Sets whether using enterprise certstore or not on an output (Windows). | _none_ | +| `unify_tag` | This creates a single buffer file when the buffering mode is `true`. | `true` | +| `upload_file_size` | Specifies the size of files to be uploaded in megabytes. | `200MB` | +| `upload_timeout` | Optionally specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached `upload_file_size` limit yet. | `30m` | +| `workload_identity_token_file` | Set the token file path for workload identity authentication. | _none_ | ### Configuration file From d31782e33e87f11b27071a73fc1d40719e331034 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 28 Oct 2025 13:00:16 -0400 Subject: [PATCH 8/9] Fixed linting failure issues in doc. Signed-off-by: Eric D. Schabell --- pipeline/outputs/azure_kusto.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pipeline/outputs/azure_kusto.md b/pipeline/outputs/azure_kusto.md index b9036b74e..3dab84f88 100644 --- a/pipeline/outputs/azure_kusto.md +++ b/pipeline/outputs/azure_kusto.md @@ -116,11 +116,11 @@ By default, Kusto will insert incoming ingestion data into a table by inferring | `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | | `tls.verify` | Force certificate validation. | `on` | | `tls.windows.certstore_name` | Sets the certstore name on an output (Windows). | _none_ | -| `tls.windows.use_enterprise_store` | Sets whether using enterprise certstore or not on an output (Windows). | _none_ | +| `tls.windows.use_enterprise_store` | Sets whether using enterprise certificate store or not on an output (Windows). | _none_ | | `unify_tag` | This creates a single buffer file when the buffering mode is `true`. | `true` | | `upload_file_size` | Specifies the size of files to be uploaded in megabytes. | `200MB` | | `upload_timeout` | Optionally specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached `upload_file_size` limit yet. | `30m` | -| `workload_identity_token_file` | Set the token file path for workload identity authentication. | _none_ | +| `workload_identity_token_file` | Set the token path for workload identity authentication. | _none_ | ### Configuration file From 7eec919887269b99d75b7e572a9a380bbc83874b Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Wed, 29 Oct 2025 14:35:43 +0100 Subject: [PATCH 9/9] Update pipeline/outputs/azure_kusto.md Fix caps on plugin name in options table description. Co-authored-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> Signed-off-by: Eric D. Schabell --- pipeline/outputs/azure_kusto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/outputs/azure_kusto.md b/pipeline/outputs/azure_kusto.md index 3dab84f88..2e078cbb6 100644 --- a/pipeline/outputs/azure_kusto.md +++ b/pipeline/outputs/azure_kusto.md @@ -70,7 +70,7 @@ By default, Kusto will insert incoming ingestion data into a table by inferring | `ingestion_endpoint` | The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net`. | _none_ | | `ingestion_endpoint_connect_timeout` | The connection timeout of various Kusto endpoints in seconds. | `60` | | `ingestion_mapping_reference` | The name of a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping) that will be used to map the ingested payload into the table columns. | _none_ | -| `ingestion_resources_refresh_interval` | Set the azure kusto ingestion resources refresh interval. | `3600` | +| `ingestion_resources_refresh_interval` | Set the Azure Kusto ingestion resources refresh interval. | `3600` | | `log_key` | Key name of the log content. | `log` | | `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | | `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` |