Marking the coordinator dead

[dilipsundarraj1]

We are using the below spring client version in our consumer.

gradle dependency:
compile('org.springframework.kafka:spring-kafka':1.1.7.RELEASE)

We often get the error in our consumer logs.

Marking the coordinator <broker>:9093 (id: 12345 rack: null) dead for group group1
Discovered coordinator <broker>:9093 (id: 12345 rack: null) for group group1

Sometimes this leads to losing some messages in the topic.

Our consumer config have the below values :

heartbeat.interval.ms : 3000
session.timeout.ms: 15000

Questions:

1. My understanding is that Spring kafka version that we are using sends the heartbeats in the background. Hearbeats are sent only when the poll method is invoked. Does this mean the consumer polls the topic even though no new messages in the topic ? Is that correct ?
2. We receive the messages only every 5 mins to the topic(4 partitions). We have two instances of the consumer running and each consumer listens to two partitions in that topic.
   Would really like to understand , under what scenarios this can happen ?

Any inputs on this issue is much appreciated. Thanks in advance!

[garyrussell]

1.1.7 is very old; I would recommend you move up the 1.3.2 or higher - the current version is 2.1.2.

1.3.x introduced a much simpler threading model, since the 0.10.1.0 client takes care of the heartbeats by itself (KIP-62).

[dilipsundarraj1]

@garyrussell - Sure, I would definitely do that.

When you say client takes care of the heartbeats, are u referring to the kafka client from the consumer app ?

Under what scenarios the below can happen ?

Marking the coordinator <broker>:9093 (id: 12345 rack: null) dead for group group1
Discovered coordinator <broker>:9093 (id: 12345 rack: null) for group group1

Does the consumer poll only a new message arrives in to a topic or all the time ?

[garyrussell]

See KIP-62 for more information.

The consumer polls all the time; with those old versions if the listener didn't respond in time, the consumer was paused and resumed when the listener caught up. That is no longer necessary since the kafka-clients Consumer takes care of heartbeats in the background.

I can't answer your question; sorry; you would have to ask the Kafka folks.

[dilipsundarraj1]

Thanks @garyrussell for your response.