Adding auto-refresh, statistic about messages, charts, and additional information about consumers

[antedesk]

I'm using your project and I found it very user-friendly and simple to configure and run.
Unfortunately, it does fit my monitoring requirements.

I believe that with few improvements this tool can be awesome.

Some enhancements are:

• auto-refresh at a given interval. This will help the user to monitor the progress;
• statistic about messages such as the rate for each consumer;
• some charts to display the trends;
• additional information about consumers groups.

[ekoutanov]

Thanks for your feedback; much valued. I think the auto-refresh is sensible and can be done with the current API.

Things like stats, rates and other metrics are probably something that should be taken out of JMX, and would need another integration into Kafka from Kafdrop. (Do-able but considerable amount of work.) It would also encroach into the sort of functionality that Prometheus/Grafana tend to offer. So I'm not sure if it's worth doing in Kafdrop, given that tools like that already do it and we'll only be playing catch up.

For your last point, would you be able to clarify what additional information could be presented?

[antedesk]

Sorry for the late response.

About stats etc, you are right you must get this information from JMX (other product like Kafka Manager by Yahoo requires that to extract stats) and it requires time.

About the additional information about consumers, I meant something like:

• the list of available consumer groups;
• topics they consumes from;
• which consumer is connected to one (or more) partition/s;
• the log size, the consumer offset, lag for each consumer;
• % of Partitions assigned to a consumer instance.

Those are the stats I used to check manually to assess the Kafka cluster/server is/are ok :)

[ekoutanov]

In terms of consumer groups, I believe most of the points you raised are already available in Kafdrop. The consumer groups show up in the topic overview, and you can click through the consumer to see more information, which includes things like lag. See screenshot below.

The only thing it's not showing at the moment is which consumer is currently assigned to which partition, which is not something you can easily get out of the Kafka Admin API (there's probably a hacky way to get that out of ZK).

[antedesk]

Cool! I missed that. My bad. Thank you for pointing me to it :)

which consumer is currently assigned to which partition

I've never had the opportunity to use the Kafka Admin API so I'm not able to help you right now. Currently to get this info from my cluster I'm using the following command

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group

obtaining the following output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
topic3          0          241019          395308          154289          consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1      consumer2
topic2          1          520678          803288          282610          consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1      consumer2
topic3          1          241018          398817          157799          consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1      consumer2
topic1          0          854144          855809          1665            consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1      consumer1
topic2          0          460537          803290          342753          consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1      consumer1
topic3          2          243655          398812          155157          consumer4-117fe4d3-c6c1-4178-8ee9-eb4

It's strange that there isn't a counterpart in the Kafka Admin API to extract this information.

[ekoutanov]

Then it likely does :) I must've missed it.