Container exception java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null

[sweat123]

2018-08-08 11:19:50,482 ERROR [org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer] - Container exception
java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:217)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:367)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:295)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1138)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1103)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
2018-08-08 11:19:50,482 ERROR [org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer] - Container exception
java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:217)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:367)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:295)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1138)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1103)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
2018-08-08 11:19:50,482 ERROR [org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer] - Container exception
java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:217)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:367)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:295)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1138)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1103)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)

We are using spring-kafka which the version is 1.1.7.RELEASE; The application work correctly and there is no errors in log; But some hours later, we find there are so many error logs which I have list above;
When the error appeared, the only way what I can do is restarting application;

kafka cluster version is 1.0.0

So is any guidance to solve this problem. Thanks a lot!

[sweat123]

We have gotten a new exception now;

2018-08-08 17:54:04,442 ERROR [org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer] - Container exception
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'version': java.nio.BufferUnderflowException
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:76)
at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeAssignment(ConsumerProtocol.java:105)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:219)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:367)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:295)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1138)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1103)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)
2018-08-08 17:54:04,442 ERROR [org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer] - Container exception
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'version': java.nio.BufferUnderflowException
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:76)
at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeAssignment(ConsumerProtocol.java:105)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:219)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:367)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:295)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1138)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1103)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:556)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)

[garyrussell]

1.1.x is very old; as mentioned on the project page, all 1.x users are recommended to upgrade to 1.3.x (currently 1.3.6) because it has a simpler threading model thanks to KIP-62. You should use the 0.11.0.2 kafka-clients with 1.3.6.

[sweat123]

@garyrussell Thank for your feedback；

In fact I have upgrade the kafka client to 1.0.0, but the error is appeared again; According to the code of AbstractCoordinator.java and ConsumerCoordinator, the possible reason is a bad network between kafka server and client; I think is not a problem of spring-kafka;

[garyrussell]

Thanks. Closing for now; please re-open if you consider this a spring problem.

[sairamlrg]

@garyrussell we are using spring-kafka 1.3.9 version and 0.11.0.2 kafka client.
we have observed the above mentioned error and got flooded with logs

One more this is this is occurring on only one consumer thread (1-c-1)

[org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer] - Container exception
java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null

But we are not able to reproduce the issue again ?
what may be the possibilities for this error ?
How can we handle this error , Do we have any exception handler for this ?

Please help us on this. Thank you.

[garyrussell]

Don"t comment on old issues. Also, GitHub issues are not for asking questions. Use stack overflow for that to reach the wider Kafka community since this has nothing to do with Spring.