Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions reference/fleet/kafka-output-settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,13 +131,27 @@ Use this option to set the Kafka topic for each {{agent}} event.

You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:

* `data_stream_type`
* `data_stream.type`
* `data_stream.dataset`
* `data_stream.namespace`
* `@timestamp`
* `event-dataset`

You can also set a custom field. This is useful if you’re using the [`add_fields` processor](/reference/fleet/add_fields-processor.md) as part of your {{agent}} input. Otherwise, setting a custom field is not recommended.
You can also set a custom field. This is useful if you need to construct a more complex or structured topic name.

To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to any integration policies on your {{fleet}}-managed {{agents}}.

For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):

```yaml
- add_fields:
target: ''
fields:
kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1>
```
1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field.

For more information, refer to [](/reference/fleet/agent-processors.md).


### Header settings [_header_settings]
Expand Down
26 changes: 25 additions & 1 deletion reference/fleet/kafka-output.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ inputs {

The `kafka` output supports the following settings, grouped by category. Many of these settings have sensible defaults that allow you to run {{agent}} with minimal configuration.

* [Kafka output configuration settings](#output-kafka-commonly-used-settings)
* [Commonly used settings](#output-kafka-commonly-used-settings)
* [Authentication settings](#output-kafka-authentication-settings)
* [Memory queue settings](#output-kafka-memory-queue-settings)
* [Topics settings](#output-kafka-topics-settings)
Expand Down Expand Up @@ -164,6 +164,30 @@ Use these options to set the Kafka topic for each {{agent}} event.
`topic` $$$kafka-topic-setting$$$
: The default Kafka topic used for produced events.

You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an [Elastic Common Schema (ECS)](ecs://reference/index.md) field. Available fields include:

* `data_stream.type`
* `data_stream.dataset`
* `data_stream.namespace`
* `@timestamp`
* `event-dataset`

You can also set a custom field. This is useful if you need to construct a more complex or structured topic name.

To set a dynamic topic value for outputting {{agent}} data to Kafka, you can add the [`add_fields` processor](/reference/fleet/add_fields-processor.md) to the input configuration settings of your standalone {{agent}}.

For example, the following `add_fields` processor creates a dynamic topic value by interpolating multiple [data stream fields](ecs://reference/ecs-data_stream.md):

```yaml
- add_fields:
target: ''
fields:
kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}' <1>
```
1. Depending on the values of the data stream fields, this generates topic names such as `logs-nginx.access-production` or `metrics-system.cpu-staging` as the value of the custom `kafka_topic` field.

For more information, refer to [](/reference/fleet/agent-processors.md).


## Partition settings [output-kafka-partition-settings]

Expand Down