No broker partitions consumed by consumer thread

[gideononline]

My environment:
-- logstash 1.4.0 (installed kafka support using your build instruction)
-- elasticsearch 1.1.0 (16 node cluster)
-- kafka_0.8.0-2 (3 node cluster)
-- zookeeper 3.3.5--1 (3 node quorum running on the kafka cluster)

I have 6 logstash indexers set up to connect to the kafka cluster of 3 nodes as input to logstash.

Three of the logstash indexers are able to connect and retrieve logs.
However, other three nodes are getting the following error:

---

log4j, [2014-04-15T06:04:17.763] WARN: kafka.consumer.ZookeeperConsumerConnector: [logstash_logstash-indexer-i-db49728b-1397541855799-bd574597], No broker partitions consumed by consumer thread logstash_logstash-indexer-i-db49728b-1397541855799-bd574597-0 for topic convertro_logstash

Any help?

regards,
Sunny

[joekiller]

How many partitions do you have per Kafka topic? You cannot have more
consumer threads than partitions per consumer work group. I'm guessing you
have more consumers than partitions on your logging target.
On Apr 15, 2014 2:08 AM, "gideononline" notifications@github.com wrote:

My environment:
-- logstash 1.4.0 (installed kafka support using your build instruction)
-- elasticsearch 1.1.0 (16 node cluster)
-- kafka_0.8.0-2 (3 node cluster)
-- zookeeper 3.3.5--1 (3 node quorum running on the kafka cluster)

I have 6 logstash indexers set up to connect to the kafka cluster of 3
nodes as input to logstash.

Three of the logstash indexers are able to connect and retrieve logs.

However, other three nodes are getting the following error:

log4j, [2014-04-15T06:04:17.763] WARN:
kafka.consumer.ZookeeperConsumerConnector:
[logstash_logstash-indexer-i-db49728b-1397541855799-bd574597], No broker
partitions consumed by consumer thread
logstash_logstash-indexer-i-db49728b-1397541855799-bd574597-0 for topic
convertro_logstash

Any help?

regards,
Sunny

Reply to this email directly or view it on GitHubhttps://github.com//issues/16
.

[gideononline]

That was it. I should've looked at the doc more carefully. Thanks again!

[bianchaorun]

I have the same number of partitions and consumers,but only one consumer can consumer the partitions ,others don't work.Why?

[joekiller]

This really should be on the logstash-users email list. You need to
include your configurations and any error messages. Earlier in this issue
the problem was determined to be a configuration problem.
On Sep 16, 2014 11:27 PM, "bianchaorun" notifications@github.com wrote:

I have the same number of partitions and consumers,but only one consumer
can consumer the partitions ,others don't work.Why?

—
Reply to this email directly or view it on GitHub
#16 (comment)
.

[bianchaorun]

Thanks for your answer and I'm sorry that it's so long to see and reply for your messages
My environment:
--kafka_2.10-0.8.1.1(2 nodes cluster)
--zookeeper-3.3.4(2 node quorum running on the kafka
I have one topic.And it has three partitions.So I use three consumer to consume the partitions.
But only one consumer works,others do not anything.When I kill the consumer that works,it will one of them begin to work.
the following is the consumer configuration file contents:

zookeeper.connect=51.96.2.85:2181
##,127.0.0.1:2182,127.0.0.1:2183
zookeeper.connectiontimeout.ms=1000000
#consumer group id
group.id=4
consumer.id=2
client.id=2
#consumer timeout
#consumer.timeout.ms=5000
#auto.commit.enable=true
auto.commit.interval.ms=60000
#queued.max.message.chunks=10000

the following is the log for the consumer:

[QC] INFO [main] kafka.utils.Logging$class.info(68) | [ConsumerFetcherManager-1411004480151] Stopping leader finder thread
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [ConsumerFetcherManager-1411004480151] Stopping all fetchers
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [ConsumerFetcherManager-1411004480151] All connections stopped
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Cleared all relevant queues for this fetcher
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Cleared the data chunks in all the consumer message iterators
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Committing all offsets after clearing the fetcher queues
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Releasing partition ownership
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Consumer 4_3 rebalancing the following partitions: ArrayBuffer(0, 1, 2) for topic test with consumers: List(4_2-0, 4_2-1, 4_2-2, 4_3-0, 4_3-1, 4_3-2)
[QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker partitions consumed by consumer thread 4_3-2 for topic test
[QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker partitions consumed by consumer thread 4_3-0 for topic test
[QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker partitions consumed by consumer thread 4_3-1 for topic test
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Updating the cache
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Consumer 4_3 selected partitions :
[QC] INFO [4_3-leader-finder-thread] kafka.utils.Logging$class.info(68) | [4_3-leader-finder-thread], Starting
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], end rebalancing consumer 4_3 try #0

And thanks again!

[joekiller]

Try not setting the consumer id or client id. I think those setting will
interfere with the consumer group.
On Sep 17, 2014 10:11 PM, "bianchaorun" notifications@github.com wrote:

Thanks for your answer and I'm sorry that it's so long to see and reply
for your messages
My environment:
--kafka_2.10-0.8.1.1(2 nodes cluster)
--zookeeper-3.3.4(2 node quorum running on the kafka
I have one topic.And it has three partitions.So I use three consumer to
consume the partitions.
But only one consumer works,others do not anything.When I kill the
consumer that works,it will one of them begin to work.
the following is the consumer configuration file contents:

zookeeper.connect=51.96.2.85:2181
##,127.0.0.1:2182,127.0.0.1:2183
timeout in ms for connecting to zookeeper

zookeeper.connectiontimeout.ms=1000000
#consumer group id
group.id=4
consumer.id=2
client.id=2
#consumer timeout
#consumer.timeout.ms=5000
#auto.commit.enable=true
auto.commit.interval.ms=60000
#queued.max.message.chunks=10000

the following is the log for the consumer:

[QC] INFO [main] kafka.utils.Logging$class.info(68) |
[ConsumerFetcherManager-1411004480151] Stopping leader finder thread
[QC] INFO [main] kafka.utils.Logging$class.info(68) |
[ConsumerFetcherManager-1411004480151] Stopping all fetchers
[QC] INFO [main] kafka.utils.Logging$class.info(68) |
[ConsumerFetcherManager-1411004480151] All connections stopped
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Cleared all
relevant queues for this fetcher
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Cleared the
data chunks in all the consumer message iterators
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Committing
all offsets after clearing the fetcher queues
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Releasing
partition ownership
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Consumer 4_3
rebalancing the following partitions: ArrayBuffer(0, 1, 2) for topic test
with consumers: List(4_2-0, 4_2-1, 4_2-2, 4_3-0, 4_3-1, 4_3-2)
[QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker
partitions consumed by consumer thread 4_3-2 for topic test
[QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker
partitions consumed by consumer thread 4_3-0 for topic test
[QC] WARN [main] kafka.utils.Logging$class.warn(83) | [4_3], No broker
partitions consumed by consumer thread 4_3-1 for topic test
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Updating the
cache
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], Consumer 4_3
selected partitions :
[QC] INFO [4_3-leader-finder-thread] kafka.utils.Logging$class.info(68) |
[4_3-leader-finder-thread], Starting
[QC] INFO [main] kafka.utils.Logging$class.info(68) | [4_3], end
rebalancing consumer 4_3 try #0

And thanks again!

—
Reply to this email directly or view it on GitHub
#16 (comment)
.