How to get a list of consumer groups?

[marghidanu]

Is there a way to obtain a list of consumer groups using the current Admin API?

[tulios]

We currently don't have a way to list of consumer groups, but this is a feature we want to add at some point.

[marghidanu]

Thanks! That would be great, I'm currently writing a lag monitoring application for our project and this would be a nice feature to have. Currently I have to manually provide the consumer group id, but it would be nice to be able to retrieve it via the Admin API.

[JaapRood]

For anyone who's keen, this would actually be a pretty good start to contributing (beyond something like documentation, etc). It's pretty much a one off request with the Kafka protocol, all the utilities of which are already in place!

[HunderlineK]

I had a very noob question, how does kafkajs invoke the kafka APIs? Is it through RPC or REST? I was trying to find the starting point of the connection to the brokers in the source could, but was unsuccessful >~< Thanks!

[Delapouite]

@HunderlineK kafkajs use a socket. See the network code in this dir: https://github.com/tulios/kafkajs/tree/master/src/network

[tulios]

Hi @HunderlineK, Kafka has its protocol on top of TCP. In KafkaJS, you have the network layer which is responsible for the socket connection, and the protocol layer, which works encoding and decoding the binary payloads returned by Kafka. The closest thing to a rest client in the library is the broker, which uses the connection and the protocol to interact with Kafka.

You can take a look at the broker code here:
https://github.com/tulios/kafkajs/blob/master/src/broker/index.js

You can find the protocol requests here:
https://github.com/tulios/kafkajs/tree/master/src/protocol/requests

And the network code here:
https://github.com/tulios/kafkajs/tree/master/src/network

[HunderlineK]

Oh, I think now I understand why developing a modern JavaScript Kafka client has been such an undertaking!

If I have understood the implementation correctly, all the helper functions are already provided, so this will only require, concretely speaking, creating a new folder under protocol/requests called ListGroups and implementing request and response handlers?

For each API, you need to first encode the message into bits based on its signature and then decode the response from bits. For each version of the API, we will also need a corresponding encoder/decoder? Though it seems sometimes one of them e.g. the request signature, remains the same across different versions.

Maybe I haven't noticed this, but all of the parameters are always required, correct? As in, we don't have situations where first we need to check the initial bits of the response in order to find out exactly what parameters we should be reading? (Or, set some initial bits in the request to specify what parameters we will be providing?)

Thanks.

[tulios]

@HunderlineK everything you will need is already in KafkaJS, the encoder and decoder can generate all types the Kafka protocol needs. The workflow is exactly what you have mentioned before, first you create a folder with the protocol method you want to implement, in this case, request/ListGroups. In there, you add all versions up to Kafka v1. Then you add your new protocol method in the list of available requests (https://github.com/tulios/kafkajs/blob/master/src/protocol/requests/index.js#L4). Finally, you add a method to the broker class.

Even if the parameters are optional they get encoded in the protocol. The list offsets is a good example for you (https://github.com/tulios/kafkajs/blob/master/src/protocol/requests/listOffsets/v2/request.js)

I can help you land a PR, you can create a proposal PR in GitHub, and we can bounce ideas there.

[HunderlineK]

@tulios So I've started implementing this; one thing I've noticed is that this particular ListGroups API does not accept any input, so after a bit of digging I found this:

The purpose of the ListGroups API is to get a simple list of the groups that are managed by each broker. To get a list of all groups in the cluster, tools will have to query each broker separately and combine the results.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-40%3A+ListGroups+and+DescribeGroup#KIP-40:ListGroupsandDescribeGroup-Motivation

I'm somewhat lost, as I'm not sure exactly how we can specify the broker?

Looking at a couple of other implementations, they approach it differently:

• kafka-node iterates over all the brokers and returns the aggregate result:
  https://github.com/SOHU-Co/kafka-node/blob/26fac1c16353b906e610def5fba4380778574339/lib/kafkaClient.js#L523

• sarama accepts the broker as a parameter:
  https://github.com/Shopify/sarama/blob/07c15a999b96f1a792f9ebd106a0a33924fa5042/broker.go#L442

(I prefer Sarama approach; the user of the library can easily iterate over the brokers if they need all of the consumer groups)

[Nevon]

I think kafka-node's approach makes more sense. We shouldn't offload that responsibility to the user, unless there's a compelling reason to.

I think the cluster has all the information you need - the current metadata with the broker's node id, and methods for retrieving those brokers. cluster.metadata() and cluster.findBroker(). That gives you a Broker instance per broker, where you'd make the ListGroups call.

[JaapRood]

I think kafka-node's approach makes more sense. We shouldn't offload that responsibility to the user, unless there's a compelling reason to.

Couldn't agree more. As far as the user should be concerned, you're talking to the Kafka Cluster. Brokers should be an implementation detail of that as much as possible (not possible everywhere, like the amount of acks in producing message, where you'd have to think about the fact that there are individual Brokers). Querying all brokers and aggregating seems like a straightforward way to achieve that in this case.

[lafeuil]

I think this issue can be closed. This is fixed in release 1.13 and PR #645.

[Nevon]

Yes indeed. Closing.