Support kafka 0.10.2.1 or newer versions

[panda87]

Hi

Im using kafka 0.10.2.1, and when I tried to run the binary with this cluster I got kafka_broker_consumer_group_describe_errors, and when I used with 0.10.0.1 as mentioned in the README file, I got the some good metrics.

I still don't know if there is any difference, but this will be nice to get support also for newer versions of kafka

Thanks
D.

[JensRantil]

@panda87 Sure! We should absolutely support that. Would you mind posting the exact CLI command that you'd execute to describe a Kafka topic (and it's lag etc.) on version 0.10.2.1? In older versions it's something like:

./kafka-consumer-groups.sh --new-consumer --bootstrap-server YOURSERVER --describe --group YOURTOPIC

Also, the specific code that queries Kafka is https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter/blob/master/kafka/collector.go#L147.

[panda87]

Thanks @JensRantil

This is the command:
./kafka-consumer-groups.sh --bootstrap-server YOURSERVER --describe --group GROUP
The --new-consumer is the default, so don't need to add it manually.

This is the result:

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
xxx           4          21555284        22970821        1415537    -                                                 -                              -
yyy           1          496377          525680          29303      -                                                 -                              -
yyy           7          478173          507408          29235      -                                                 -                              -
yyy           4          478949          508402          29453      -                                                 -                              -
xxx           8          21549325        22965791        1416466    -                                                 -                              -
yyy           0          497273          526634          29361      -                                                 -                              -
xxx           5          21555282        22973162        1417880    -                                                 -                              -
xxx           9          21547334        22965198        1417864    -                                                 -                              -
xxx           3          21552045        22968084        1416039    -                                                 -                              -
yyy           6          477752          507310          29558      -                                                 -                              -
xxx           0          21553771        22971842        1418071    -                                                 -                              -
yyy           3          477191          506351          29160      -                                                 -                              -
xxx           6          21550029        22966263        1416234    -                                                 -                              -
xxx           2          21561555        22978761        1417206    -                                                 -                              -
yyy           2          496662          526152          29490      -                                                 -                              -
xxx           1          21552286        22968941        1416655    -                                                 -                              -
xxx           7          21555863        22972704        1416841    -                                                 -                              -
yyy           5          478942          508314          29372      -                                                 -                              -

[JensRantil]

@panda87 Thanks! Do you get an error if you'd specify --new-consumer to the script?

[panda87]

No, it's just already the default, and it may be deprecated in the next versions.
(Since the new consumer API is the current standard)

[JensRantil]

@panda87 Hm, what's interesting here is that that command is the exact same one that the exporter is supposed to extract. Are you sure that you've properly set https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter/blob/master/cmd/kafka_consumer_group_exporter.go#L28 when starting the exporter?

[panda87]

Yes, I dockerized it, so I just changed in FROM to another version, and on 0.10.2.1 I got only errors while in 0.10.0.1 I got errors but also good results.

[panda87]

Im wondering maybe the results template is different?

[JensRantil]

@panda87 That's a very good theory. Could you verify that the output from the command separates its columns by spaces and not tabs (which sometimes gets converted to spaces when copy/pasting)?

[JensRantil]

@panda87 Thanks for all the help so far! I started on a pull request in #29. I'm on a rather tight schedule, but it looks like they've modified the order of the fields just like you said. I've been hoping we wouldn't have to require the user to give flags on which version of Kafka which is being running. Do you know if there's any way to get a Kafka version from the ./kafka-consumer-groups.sh script?

[panda87]

No, this script doesn't have version flag
I've tried few other scripts but none of them support this flag