Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kafka_franz consumer offset does not reset based on start_from_oldest #2366

Open
bhataprameya opened this issue Feb 6, 2024 · 0 comments
Open
Labels
inputs Any tasks or issues relating specifically to inputs needs investigation It looks as though have all the information needed but investigation is required

Comments

@bhataprameya
Copy link

bhataprameya commented Feb 6, 2024

  • Issue: In the kafka_franz input client, the consumer offset doesn't reset based on start_from_oldest after Kafka deletes data due to retention policy.

  • Current Solution: Restarting Benthos is required to reset the offset and resume consumption.

  • Expected Behavior: Consumer offset should automatically reset its offset based on start_from_oldest when the data it is trying to consume is no longer found in Kafka.

input config:

      kafka_franz:
        seed_brokers: ['${KAFKA_URL}']
        topics: ["${KAFKA_TOPIC}"]
        regexp_topics: false
        consumer_group: "${KAFKA_GROUP_ID}"
        client_id: benthos
        rack_id: --RACK_PLACE_HOLDER--
        checkpoint_limit: 50000
        commit_period: 5s
        start_from_oldest: false
        multi_header: false
@Jeffail Jeffail added inputs Any tasks or issues relating specifically to inputs needs investigation It looks as though have all the information needed but investigation is required labels Feb 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
inputs Any tasks or issues relating specifically to inputs needs investigation It looks as though have all the information needed but investigation is required
Projects
None yet
Development

No branches or pull requests

2 participants