New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
a given consumer group is not receiving messages from different topics and KafkaJS didn't throw any error #699
Comments
Just to make sure, you don't have any ACLs applied on the topic that would prevent you from reading from it? Also, are you continuously producing messages from your producer? Otherwise with You can run your consumer with |
@Nevon Thanks for helping out, I finally found the issue. I was using the same consumer group (using the same groupId) to consume two different topics, After using a unique groupId my code is working. (I wonder if the kafkajs can be noisy about this issue) |
@Nevon I wonder why we cannot fetch messages from different topics in the same consumer group? if it is intentional can we mention this in the documentation or be noisy about it (throw an error) |
Well, the concept of a consumer group in Kafka is that you have multiple identical nodes that together form a group. It doesn't make sense from a Kafka conceptual point of view not to. We do emit a warn log about this if we receive assignments for topics that we aren't subscribed to, because that's the only way we can find out that you are doing this. The warn message is here: https://github.com/tulios/kafkajs/blob/master/src/consumer/consumerGroup.js#L181-L192 and it links to an FAQ explaining the issue: https://kafka.js.org/docs/faq#why-am-i-receiving-messages-for-topics-im-not-subscribed-to We cannot throw an error because if you are deploying a new version of your application that subscribes to new topics, your old consumers would receive assignments for topics they aren't assigned to until they are fully replaced by the new instances. |
Hello everyone, I'm using kafkajs
v1.12.0
with an AWS MSK clustermy producer code looks like this:
After publishing messages, my
RecordMetadata
looks like:in a different application my consumer code looks like this:
My consumer logs are:
For further inspection I tried getting the topics meta data with the admin and this was the result
I could see Network RX/TX packets traffic in the AWS brokers
I'm not sure why the
eachMessage
function was never executed so I wasn't able to receive the messages in the consumerThe text was updated successfully, but these errors were encountered: