Message count estimation on compacted topics

[weeco]

We estimate the number of messages by calculating highWatermark - lowWatermark. This returns the correct number of messages for topics whose cleanup.policy is set to delete but it's not reliable and possibly very wrong for compacted topics.

The only way to get the exact number of messages per topic and per partition is to actually consume all these messages. Since this may take too long we elaborate other options:

Idea 1: Estimating the number of messages by using the partition size

We know each partitions log dir size as this can be queried via the Kafka API. The idea is to consume some messages in order to calculate a representative average message size and use that along with the partition size to calculate the number of messages in the compacted topic.

[JakeSummers]

In the short-term, it would be a significant improvement if we could NOT provide count estimation for compacted tables. I just spent more than a day debugging my Kafka consumer trying to figure out why it was missing messages.

The estimate in the topic was ~20 million but my consumer was only pulling ~2 million. This was a real head scratcher.

[weeco]

@JakeSummers We have a relative prominent hint that this is just an estimate in the configurations tab for any topic with compact or compact,delete cleanup policy. We'll additionally add a tooltip in the statistics bar to make this clearer (see: #1064).

We had many users that wished to have this feature, even for compacted topics as it would help them to have a rough idea how many messages could be in that topic. It's always hard to balance requests that may help some users, but confuse or disturb other users. I hope adding the hint in the statistics bar right next to the message count estimate makes it less worse. There's no way to figure out the exact count of messages without actually consuming all messages. And even while consuming all messages the count may already change (new compactions, deletions, etc).

[JakeSummers]

@weeco - In my case, the estimate of total messages was off by a factor of 10. I would argue that an estimate that is so wrong, does more harm than good.

And even while consuming all messages the count may already change (new compactions, deletions, etc).

With this implementation, you could provide a timestamp of when the message count was from.

Aka:

Messages
2,103,103
<i>at Feb 6, 2023, 12:01 UTC</i>

This makes it clear that the data was an estimate.

We'll additionally add a tooltip in the statistics bar to make this clearer

I don't think a tooltip is a good UI feature to use here. I think a better strategy would be to rename Messages to Estimated Messages. Using Messages conveys that this is a factual piece of information.