You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am running a node server that acts as a Kafka Consumer. I had seen instances of few offsetOutOfRange errors online, whereby the consumer tries to read messages that are no longer available. I tried to overcome that by having the following handler
consumer.on('offsetOutOfRange', function (topic) {
console.log("------------- offsetOutOfRange ------------");
topic.maxNum = 2;
offset.fetch([topic], function (err, offsets) {
var min = Math.min.apply(null, offsets[topic.topic][topic.partition]);
consumer.setOffset(topic.topic, topic.partition, min);
});
});
I am still not able to absolve this issue. I get the offsetOutOfRange issue in a continuous loop. But amazingly I am also getting all the messages generated out of kafka. Does anyone have a clue as to what is happening and how to solve it so that I dont get into this code block. I thought the consumer.setOffset(topic.topic, topic.partition, min); should be able to set the correct offset but that is not the case.
The text was updated successfully, but these errors were encountered:
We were experiencing the same loop and diagnosed the issue to be a type mismatch between the offset numbers returned from zookeeper and those in the payloads list (number in payloads, string from zookeeper). We were preparing a pull request and then noticed it had been fixed in 2747d7c
I am running a node server that acts as a Kafka Consumer. I had seen instances of few offsetOutOfRange errors online, whereby the consumer tries to read messages that are no longer available. I tried to overcome that by having the following handler
I am still not able to absolve this issue. I get the offsetOutOfRange issue in a continuous loop. But amazingly I am also getting all the messages generated out of kafka. Does anyone have a clue as to what is happening and how to solve it so that I dont get into this code block. I thought the
consumer.setOffset(topic.topic, topic.partition, min);
should be able to set the correct offset but that is not the case.The text was updated successfully, but these errors were encountered: