Filebeat-5.0.0-beta1 is not able to publish message to Kafka Cluster

[tarinidash]

Filebeat version: Filebeat-5.0.0-beta1
Kafka cluster version : 0.10.0.0

ran with command :

./filebeat -c -e filebeat.yml -d "*"

Output logs

2016/09/28 18:03:11.714193 output.go:109: DBG output worker: publish 588 events
2016/09/28 18:03:11.714230 lb.go:72: DBG configure maxattempts: 0
2016/09/28 18:03:11.714292 async_worker.go:66: DBG load balancer: start client loop
2016/09/28 18:03:11.714322 client.go:61: DBG connect: [x.x.x.x:9092]
2016/09/28 18:03:11.714340 log.go:16: WARN kafka message: Initializing new client
2016/09/28 18:03:11.714402 log.go:12: WARN client/metadata fetching metadata for all topics from broker 10.96.72.26:9092
2016/09/28 18:03:11.714455 context.go:93: DBG forwards msg with attempts=-1
2016/09/28 18:03:11.714924 log.go:12: WARN Connected to broker at x.x.x.x:9092 (unregistered)
2016/09/28 18:03:11.716125 log.go:12: WARN client/brokers registered new broker #2 at hostname-01:9092
2016/09/28 18:03:11.716138 log.go:12: WARN client/brokers registered new broker #3 at hostname-01:9092
2016/09/28 18:03:11.716147 log.go:12: WARN client/brokers registered new broker #0 at hostname-01:9092
2016/09/28 18:03:11.716233 log.go:16: WARN kafka message: Successfully initialized new client
2016/09/28 18:03:11.716265 context.go:138: DBG events from worker worker queue
2016/09/28 18:03:11.716281 client.go:104: DBG publish events
2016/09/28 18:03:11.716359 context.go:98: DBG message forwarded
2016/09/28 18:03:11.716587 log.go:12: WARN producer/broker/3 starting up
2016/09/28 18:03:11.716602 log.go:12: WARN producer/broker/3 state change to [open] on beats/0
2016/09/28 18:03:11.718304 log.go:12: WARN Connected to broker at hostname-01:9092 (registered as #3)
2016/09/28 18:03:11.736660 client.go:210: DBG finished kafka batch
2016/09/28 18:03:11.736686 client.go:215: DBG Kafka publish failed with: kafka server: Unexpected (unknown?) server error.

Kafka logs

Below is the output from KAFKA

[2016-09-28 14:56:10,719] ERROR [Replica Manager on Broker 3]: Error processing append operation on partition beats-0 (kafka.server.ReplicaManager)
java.lang.IllegalStateException: Compressed message has magic value 0 but inner message has magic value 1

Workaround:

comment the version in filebeat.yml under kafka output section.

exp:
#version: 0.10.0.0

[ruflin]

@tarinidash Thanks for reporting. Could you share your full config file? What is the exp: meaning above? I tried to apply some formatting to your issue to make it better readable.

[tarinidash]

@ruflin Thank you for looking into it.

exp => for example

filebeat.yml.zip

[tarinidash]

sorry, accidentally closed it.

[urso]

Checking the logs it's due to compressed and embedded messages using a different protocol version identifer. The problem should only occur if version 0.10.... is configured.

Related kafka client issues: IBM/sarama#757 and IBM/sarama#758

The fix is already available in IBM/sarama#759 , but no release version available yet. I managed to reproduce the issue and successfully tested the fix to work.

Also waiting for fix for #2717 becoming available before updating.

[johnny1952]

I got the same problem. After commented the "version: 0.10.0.0",the filebeat then produced msgs to kafka cluster. thanks.

[monicasarbu]

Closing the issue as the fix is already merged to master.