diff --git a/reference/fleet/kafka-output-settings.md b/reference/fleet/kafka-output-settings.md index 7558403b43..9b5685b0c3 100644 --- a/reference/fleet/kafka-output-settings.md +++ b/reference/fleet/kafka-output-settings.md @@ -131,13 +131,27 @@ Use this option to set the Kafka topic for each {{agent}} event. You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include: - * `data_stream_type` + * `data_stream.type` * `data_stream.dataset` * `data_stream.namespace` * `@timestamp` * `event-dataset` - You can also set a custom field. This is useful if you’re using the [`add_fields` processor](/reference/fleet/add_fields-processor.md) as part of your {{agent}} input. Otherwise, setting a custom field is not recommended. + You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. + + To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to any integration policies on your {{fleet}}-managed {{agents}}. + + For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md): + + ```yaml + - add_fields: + target: '' + fields: + kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1> + ``` + 1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field. + + For more information, refer to [](/reference/fleet/agent-processors.md). ### Header settings [_header_settings] diff --git a/reference/fleet/kafka-output.md b/reference/fleet/kafka-output.md index 49a7951e97..fac4c73f59 100644 --- a/reference/fleet/kafka-output.md +++ b/reference/fleet/kafka-output.md @@ -69,7 +69,7 @@ inputs { The `kafka` output supports the following settings, grouped by category. Many of these settings have sensible defaults that allow you to run {{agent}} with minimal configuration. -* [Kafka output configuration settings](#output-kafka-commonly-used-settings) +* [Commonly used settings](#output-kafka-commonly-used-settings) * [Authentication settings](#output-kafka-authentication-settings) * [Memory queue settings](#output-kafka-memory-queue-settings) * [Topics settings](#output-kafka-topics-settings) @@ -164,6 +164,30 @@ Use these options to set the Kafka topic for each {{agent}} event. `topic` $$$kafka-topic-setting$$$ : The default Kafka topic used for produced events. + You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include: + + * `data_stream.type` + * `data_stream.dataset` + * `data_stream.namespace` + * `@timestamp` + * `event-dataset` + + You can also set a custom field. This is useful if you need to construct a more complex or structured topic name. + + To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to the input configuration settings of your standalone {{agent}}. + + For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md): + + ```yaml + - add_fields: + target: '' + fields: + kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1> + ``` + 1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field. + + For more information, refer to [](/reference/fleet/agent-processors.md). + ## Partition settings [output-kafka-partition-settings]