-
Notifications
You must be signed in to change notification settings - Fork 346
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Message count estimation on compacted topics #83
Comments
In the short-term, it would be a significant improvement if we could NOT provide count estimation for compacted tables. I just spent more than a day debugging my Kafka consumer trying to figure out why it was missing messages. The estimate in the topic was ~20 million but my consumer was only pulling ~2 million. This was a real head scratcher. |
@JakeSummers We have a relative prominent hint that this is just an estimate in the configurations tab for any topic with We had many users that wished to have this feature, even for compacted topics as it would help them to have a rough idea how many messages could be in that topic. It's always hard to balance requests that may help some users, but confuse or disturb other users. I hope adding the hint in the statistics bar right next to the message count estimate makes it less worse. There's no way to figure out the exact count of messages without actually consuming all messages. And even while consuming all messages the count may already change (new compactions, deletions, etc). |
@weeco - In my case, the estimate of total messages was off by a factor of 10. I would argue that an estimate that is so wrong, does more harm than good.
With this implementation, you could provide a timestamp of when the message count was from. Aka:
This makes it clear that the data was an estimate.
I don't think a tooltip is a good UI feature to use here. I think a better strategy would be to rename |
We estimate the number of messages by calculating
highWatermark - lowWatermark
. This returns the correct number of messages for topics whose cleanup.policy is set todelete
but it's not reliable and possibly very wrong forcompacted
topics.The only way to get the exact number of messages per topic and per partition is to actually consume all these messages. Since this may take too long we elaborate other options:
Idea 1: Estimating the number of messages by using the partition size
We know each partitions log dir size as this can be queried via the Kafka API. The idea is to consume some messages in order to calculate a representative average message size and use that along with the partition size to calculate the number of messages in the compacted topic.
The text was updated successfully, but these errors were encountered: