kt group doesn't show consumer groups for Kafka v0.10.0.1

[nmaquet]

Hey Felix :)

We'd like to use kt group to view and manipulate offsets, however the command doesn't show any of our groups on Kafka v0.10.0.1

I've tested the exact same consumer code on Kafka 0.10.1.0 and in that case kt group shows the consumer groups as expected.

Note that we use sarama to create the consumer and mark offsets directly using sarama.PartitionOffsetManager.MarkOffset. We're not using sarama-cluster.

Do you know any reason why kt group wouldn't show all consumer groups for Kafka versions older than 0.10.1.0?

[fgeller]

hey Nic!

the only reason (that i remember) why kt would hide an "offset" is that it comes back as the value that indicates that it wasn't marked yet (sarama.OffsetNewest, i think?). but i'm guessing your test rules that out?

did you test with different server versions vs sarama versions?
maybe the sarama consumer needs to be configured to that particular kafka server version?

should find some time in the next days to look a bit closer, but time and internet is still a bit rare at the moment.

[nmaquet]

but i'm guessing your test rules that out?

Yup, that's not the case here.

did you test with different server versions vs sarama versions?

Tested with 0.10.0.1 and 0.10.1.0, latest Sarama

maybe the sarama consumer needs to be configured to that particular kafka server version?

I've tried that and it doesn't make a difference unfortunately.

Interestingly I just found out today that resetting the offset works for my group, even though it doesn't appear when listing groups!

[fgeller]

hey @nmaquet !

could you post (or email me privately) the result of

$ kt group -verbose -brokers some-broker  -topic x -group y -version v0.10.0.1
$ kt group -verbose -brokers some-broker  -topic x -group y -version v0.10.0.1 -reset 23
$ kt group -verbose -brokers some-broker  -topic x -group y -version v0.10.0.1

? the commands failed for me when i didn't specify v0.10.0.1, but with a specified version i got the expected results. the -verbose flag should help by printing out the sarama logs.

[fgeller]

hey @nmaquet again :)

i think you ran into (at least?) one of these issues:

• the version has a v prefix and quietly defaulted to the latest version when it didn't recognize the supplied version.
• the offset for the group/topic combination that you shared wasn't set to an explicit value from what i could see, so sarama would identify it as OffsetNewest which kt in turn decided to hide from you. the reason for hiding was that you will get the -1 offset for pretty much any group/topic combination that has no explicit value.

i addressed both in #58: kt now provides feedback on supported kafka versions and doesn't hide the -1 one offsets.

could you run go get -u github.com/fgeller/kt and test again? nb: current master also has same changes to the group commands ui, -partition is -partitions now.

closing for now - please re-open if the issue isn't resolved after an update :)