From 959df7d9281989d746af41768fe7e7ec08ced104 Mon Sep 17 00:00:00 2001 From: Visha Angelova Date: Fri, 19 Sep 2025 15:26:02 +0200 Subject: [PATCH 1/2] Extend the topic setting description for the Kafka output --- reference/fleet/kafka-output-settings.md | 18 ++++++++++++++-- reference/fleet/kafka-output.md | 26 +++++++++++++++++++++++- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/reference/fleet/kafka-output-settings.md b/reference/fleet/kafka-output-settings.md index 7558403b43..1ddf0e56f3 100644 --- a/reference/fleet/kafka-output-settings.md +++ b/reference/fleet/kafka-output-settings.md @@ -131,13 +131,27 @@ Use this option to set the Kafka topic for each {{agent}} event. You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include: - * `data_stream_type` + * `data_stream.type` * `data_stream.dataset` * `data_stream.namespace` * `@timestamp` * `event-dataset` - You can also set a custom field. This is useful if you’re using the [`add_fields` processor](/reference/fleet/add_fields-processor.md) as part of your {{agent}} input. Otherwise, setting a custom field is not recommended. + You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. + + To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to any integration policies on your Fleet-managed {{agents}}. + + For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md): + + ```yaml + - add_fields: + target: '' + fields: + kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1> + ``` + 1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field. + + For more information, refer to [](/reference/fleet/agent-processors.md). ### Header settings [_header_settings] diff --git a/reference/fleet/kafka-output.md b/reference/fleet/kafka-output.md index 49a7951e97..fac4c73f59 100644 --- a/reference/fleet/kafka-output.md +++ b/reference/fleet/kafka-output.md @@ -69,7 +69,7 @@ inputs { The `kafka` output supports the following settings, grouped by category. Many of these settings have sensible defaults that allow you to run {{agent}} with minimal configuration. -* [Kafka output configuration settings](#output-kafka-commonly-used-settings) +* [Commonly used settings](#output-kafka-commonly-used-settings) * [Authentication settings](#output-kafka-authentication-settings) * [Memory queue settings](#output-kafka-memory-queue-settings) * [Topics settings](#output-kafka-topics-settings) @@ -164,6 +164,30 @@ Use these options to set the Kafka topic for each {{agent}} event. `topic` $$$kafka-topic-setting$$$ : The default Kafka topic used for produced events. + You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include: + + * `data_stream.type` + * `data_stream.dataset` + * `data_stream.namespace` + * `@timestamp` + * `event-dataset` + + You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. + + To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to the input configuration settings of your standalone {{agent}}. + + For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md): + + ```yaml + - add_fields: + target: '' + fields: + kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1> + ``` + 1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field. + + For more information, refer to [](/reference/fleet/agent-processors.md). + ## Partition settings [output-kafka-partition-settings] From 1dc81ceb13a15fb725e0cdd81c6b1ae4b7370277 Mon Sep 17 00:00:00 2001 From: Visha Angelova <91186315+vishaangelova@users.noreply.github.com> Date: Mon, 22 Sep 2025 12:12:46 +0200 Subject: [PATCH 2/2] Update reference/fleet/kafka-output-settings.md --- reference/fleet/kafka-output-settings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/fleet/kafka-output-settings.md b/reference/fleet/kafka-output-settings.md index 1ddf0e56f3..9b5685b0c3 100644 --- a/reference/fleet/kafka-output-settings.md +++ b/reference/fleet/kafka-output-settings.md @@ -139,7 +139,7 @@ Use this option to set the Kafka topic for each {{agent}} event. You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. - To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to any integration policies on your Fleet-managed {{agents}}. + To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to any integration policies on your {{fleet}}-managed {{agents}}. For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):