diff --git a/crowdsec-docs/docs/log_processor/data_sources/kafka.md b/crowdsec-docs/docs/log_processor/data_sources/kafka.md index f0092154c..23d9f4859 100644 --- a/crowdsec-docs/docs/log_processor/data_sources/kafka.md +++ b/crowdsec-docs/docs/log_processor/data_sources/kafka.md @@ -35,6 +35,30 @@ labels: type: nginx ``` +Adding a batch configuration: + +```yaml +source: kafka +brokers: + - "localhost:9093" +topic: "my-topic" +timeout: 5 +tls: + insecure_skip_verify: true + client_cert: /path/kafkaClient.certificate.pem + client_key: /path/kafkaClient.key + ca_cert: /path/ca.crt +labels: + type: nginx +batch: + min_bytes: 1024 # 1KB + max_bytes: 1048576 # 1MB + max_wait: 5s + queue_size: 1000 + commit_interval: 1s +``` + + :::info The reader will always start from the latest offset. ::: @@ -60,7 +84,7 @@ Required. The consumer group id to use. -Cannot be used with `partition`. +Cannot be used with `partition`. :::warning It is highly recommended to set this value, or crowdsec will only read logs from the 1st partition of the topic. @@ -72,6 +96,12 @@ Read messages from the given partition. Mostly useful for debugging. Cannot be used with `group_id`. +### `timeout` + +Maximum time to wait for new messages before returning an empty read. + +Default: 5 + ### `tls.insecure_skip_verify` To disable security checks on the certificate. @@ -96,6 +126,35 @@ The CA certificate path. Optional, when you want to enable TLS with client certificate. +### `batch.min_bytes` + +Minimum number of bytes to accumulate in the fetch buffer before returning results. + +Default: 1 + +### `batch.max_bytes` + +Maximum number of bytes to fetch in one go. + +Default: 1048576 (1MB) + +### `batch.max_wait` + +Maximum time to wait before returning a fetch, even if `batch.min_bytes` isn’t reached. + +Default: 250ms + +### `batch.queue_size` + +Maximum number of messages to buffer internally before processing. + +Default: 100 + +### `batch.commit_interval` + +Time interval between automatic commits of consumer offsets. + +Default: 0 (commit after every fetch) ### `source` @@ -104,4 +163,3 @@ Must be `kafka` ## DSN and command-line This datasource does not support acquisition from the command line. -