Skip to content

in_kafka: add enable_auto_commit option for Kafka consumer #10412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

nareshku
Copy link
Contributor

@nareshku nareshku commented May 31, 2025

This change introduces a new configuration option enable_auto_commit to control Kafka message commit behavior. When disabled (default), messages are committed only after successful processing, providing better control over message processing guarantees.

Users can enable auto-commit by setting enable_auto_commit true in their configuration if they prefer automatic batch commits. When enabled the overall throughput will be higher.

Fixes #9813


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Tested locally with the config enable_auto_commit option.

[INPUT]
    Name        kafka
    Tag         kafka-test
    Brokers     <broker endpoint>
    Topics      <topic-name>
    poll_ms     100
    poll_timeout_ms 1000
    enable_auto_commit true
    threaded    true

Results: Below shows Fluent Bit input metrics captured at every 10s intervals.
(note: the throughput depends on the number of partition count, compression, etc.)

fluentbit_input_bytes_total{name="kafka.0"} 136051025 1748731345204
fluentbit_input_records_total{name="kafka.0"} 82713 1748731345204

fluentbit_input_bytes_total{name="kafka.0"} 536194834 1748731355209
fluentbit_input_records_total{name="kafka.0"} 322963 1748731355209

fluentbit_input_bytes_total{name="kafka.0"} 936335445 1748731365215
fluentbit_input_records_total{name="kafka.0"} 563360 1748731365215

fluentbit_input_bytes_total{name="kafka.0"} 1336472098 1748731375221
fluentbit_input_records_total{name="kafka.0"} 805113 1748731375221

fluentbit_input_bytes_total{name="kafka.0"} 1736605503 1748731385227
fluentbit_input_records_total{name="kafka.0"} 1044700 1748731385227

fluentbit_input_bytes_total{name="kafka.0"} 1975992700 1748731395232
fluentbit_input_records_total{name="kafka.0"} 1188824 1748731395232

Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

This change introduces a new configuration option 'enable_auto_commit' to
control Kafka message commit behavior. When disabled (default), messages
are committed only after successful processing, providing better control
over message processing guarantees.

The option defaults to false to:
- Prevent potential message loss in case of failures
- Give users explicit control over commit behavior
- Ensure messages are only committed after successful processing

Users can enable auto-commit by setting enable_auto_commit true in their
configuration if they prefer automatic batch commits.

Signed-off-by: nareshku <nareshkumar4ever@gmail.com>
@cosmo0920
Copy link
Contributor

cosmo0920 commented Jun 3, 2025

@nareshku Could you use in_kafka: prefix in your latest commit message? It would be nice to follow the contribution guideline.

…avior

Signed-off-by: nareshku <nareshkumar4ever@gmail.com>
@cosmo0920 cosmo0920 added this to the Fluent Bit v4.0.4 milestone Jun 4, 2025
Signed-off-by: nareshku <nareshkumar4ever@gmail.com>
@edsiper
Copy link
Member

edsiper commented Jul 8, 2025

Thanks for this contribution, I have rebased this in a separate PR #10558

@edsiper
Copy link
Member

edsiper commented Jul 8, 2025

merged in #10558, thank you

@edsiper edsiper closed this Jul 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Slow performance with Kafka input module
4 participants