Skip to content

input: kafka: Document data_format option #1118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 4, 2024
Merged

input: kafka: Document data_format option #1118

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b3a2505
input: kafka: Document `data_format` option
tchrono May 29, 2023
cfa3006
Rename data_format to format
tchrono Jul 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions pipeline/inputs/kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill
| :--- | :--- | :--- |
| brokers | Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. | |
| topics | Single entry or list of topics separated by comma \(,\) that Fluent Bit will subscribe to. | |
| format | Serialization format of the messages. If set to "json", the payload will be parsed as json. | none |
| client\_id | Client id passed to librdkafka. | |
| group\_id | Group id passed to librdkafka. | fluent-bit |
| poll\_ms | Kafka brokers polling interval in milliseconds. | 500 |
Expand Down Expand Up @@ -51,6 +52,7 @@ The fluent-bit source repository contains a full example of using fluent-bit to
brokers kafka-broker:9092
topics fb-source
poll_ms 100
format json

[FILTER]
Name lua
Expand All @@ -66,6 +68,8 @@ The fluent-bit source repository contains a full example of using fluent-bit to

The above will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds.

Since the payload will be in json format, we ask the plugin to automatically parse the payload with `format json`.

Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.

The example can be executed locally with `make start` in the `examples/kafka_filter` directory (docker/compose is used).