GH-1107: Kafka metrics lag check #1123
Conversation
When KafkaBinderMetrics checks lags for offsets in partitions, if the partitions have nothing committed yet, it unnecessarily tries to find the last committed offset. This is causing some unncessary logging. Avoid this by only checking the committed offsets in a partition, if the current end offset is greater than zero. Resolves spring-cloud#1107
There was a problem hiding this comment.
I am not sure what's driving this change; when I tested this, I simply got offset=0 for partitions that had no committed offset.
@KafkaListener(id = "kbgh1123", topics = "kbgh1123")
public void listen(String in, Consumer<?, ?> consumer) {
Map<TopicPartition, OffsetAndMetadata> committed =
consumer.committed(Set.of(new TopicPartition("kbgh1123", 0), new TopicPartition("kbgh1123", 1)));
System.out.println(in + "\n" + committed);
}foo
{kbgh1123-1=OffsetAndMetadata{offset=0, leaderEpoch=null, metadata=''}, kbgh1123-0=OffsetAndMetadata{offset=0, leaderEpoch=null, metadata=''}}
| @@ -210,11 +212,16 @@ private long findTotalTopicGroupLag(String topic, String group, Map<String, Cons | |||
|
|
|||
| for (Map.Entry<TopicPartition, Long> endOffset : endOffsets | |||
| .entrySet()) { | |||
There was a problem hiding this comment.
Why not get all the committed offsets in a single call here .committed(topicPartitions) rather that one-at-a-time in the loop below?
|
Oh; sorry - I see it's these logs he's complaining about I still think it would be better to fetch them in one call; why not advise the user the set the log level for |
|
@garyrussell Ya, we can certainly advise the user of that. I saw your comment on the issue. I was thinking that if we already know that the end offset for the partition is |
|
Agreed. But I would still prefer to see a single call to |
|
Closing this PR; The above concern is addressed through #1130. |
When KafkaBinderMetrics checks lags for offsets in partitions, if the partitions
have nothing committed yet, it unnecessarily tries to find the last committed offset.
This is causing some unncessary logging. Avoid this by only checking the committed
offsets in a partition, if the current end offset is greater than zero.
Resolves #1107