Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resuming partitions not paused, manually or through incremental subscribe, can cause to restart consuming from a previous offset #4686

Closed
1 of 7 tasks
emasab opened this issue Apr 15, 2024 · 1 comment
Labels

Comments

@emasab
Copy link
Collaborator

emasab commented Apr 15, 2024

Description

In case of subscription change with a consumer using the cooperative assignor it can resume fetching from a previous position.
That can also happen if resuming a partition that wasn't paused.
The reason is that fetch version is immediately bumped when enqueuing the resume operation and even if the operation is then discarded because partition isn't paused, the is no next_fetch_start to start from when offset validation is completed in rd_kafka_toppar_fetch_decide_start_from_next_fetch_start, or it's the next_fetch_start of previous pause operation, so there can be a reset to an offset that was already consumed.

How to reproduce

Execute test 0050/test_no_duplicate_messages("cooperative-sticky") or 0145/test_no_duplicate_messages_unnecessary_resume(*) in #4636.

Checklist

Please provide the following information:

  • librdkafka version (1.x)
  • Apache Kafka version: <REPLACE with e.g., 0.10.2.3>
  • librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
  • Operating system: <REPLACE with e.g., Centos 5 (x64)>
  • Provide logs (with debug=.. as necessary) from librdkafka
  • Provide broker log excerpts
  • Critical issue
@emasab emasab added the bug label Apr 15, 2024
@emasab
Copy link
Collaborator Author

emasab commented Apr 25, 2024

Fixed in #4636

@emasab emasab closed this as completed Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant