Skip to content
This repository has been archived by the owner on Jan 24, 2024. It is now read-only.

[BUG]The commit offset and partition data expire at the same time, and the ListOffset interface returns an incorrect result #1737

Open
keyboardbobo opened this issue Feb 23, 2023 · 1 comment
Labels

Comments

@keyboardbobo
Copy link
Contributor

keyboardbobo commented Feb 23, 2023

Describe the bug
The client requests the commit offset of all partitions according to offsetFetch. If the commit offset of the partition expires, the broker returns the offset of the partition as null, and the client willauto.offset.resetaccording to the configured policy, and then set the offset of the missing partition to the latest topic Or the earliest offset, get this offset and then pull the data.

For example, the latest offset of a partition in topicA is 10, the oldest offset is 0, and the consumption offset of this partition is also 10. After a period of time, if the commit offset data of __consumer_offset expires and the client cannot get the consumption offset according to the OffsetFetch request, it will request the earliest offset of the partition according to auto.offset.reset=earliest. If the partition data is also It has expired, and the earliest offset returned becomes 9 (one smaller than the latest offset), which should actually be 10

To Reproduce
Steps to reproduce the behavior:

  1. Send data to topicA
  2. Consumed all the data of topicA
  3. Record the partition offset and consumption offset at this time
  4. After a period of time (make sure all data has expired)...
  5. Use kafka client to query partition offset and consumption offset

Expected behavior
In this scenario, the earliest and latest offsets should be the same

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

@BewareMyPower
Copy link
Collaborator

I've reproduced this issue when the offset topic is expired by the retention policy:

beginning offsets test-0 => 4
end offsets test-0 => 5

If the offset topic is deleted by force, the result would be different:

beginning offsets test-0 => 5
end offsets test-0 => 5

I will look deeper into this issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants