Logstash-Kafka Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member.

[Cyb3rWard0g]

Describe the problem
I started to get the following error message:

[2018-06-11T03:40:54,698][WARN ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-0, groupId=logstash] Synchronous auto-commit of offsets {winlogbeat-0=OffsetAndMetadata{offset=178360, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

This is the Kafka input defined in Logstash:

input {
  kafka
  {
    bootstrap_servers => "helk-kafka-broker:9092,helk-kafka-broker2:9093"
    topics => ["winlogbeat"]
    decorate_events => true
    codec => "json"
    auto_offset_reset => "earliest"
    ############################# HELK Optimizing Latency #############################
    fetch_min_bytes => "1"
    ############################# HELK Optimizing Availability #############################
    session_timeout_ms => "6000"
  }
}

What steps did you take trying to fix the issue?
I google some potential new configs. I looked into the default configs for a Kafka Consumer and I updated the Kafka Input Logstash file to:

input {
  kafka
  {
    bootstrap_servers => "helk-kafka-broker:9092,helk-kafka-broker2:9093"
    topics => ["winlogbeat"]
    decorate_events => true
    codec => "json"
    auto_offset_reset => "earliest"
    ############################# HELK Optimizing Latency #############################
    fetch_min_bytes => "1"
    request_timeout_ms => "305000"
    ############################# HELK Optimizing Availability #############################
    session_timeout_ms => "10000"
    max_poll_records => "550"
    max_poll_interval_ms => "300000"
  }
}

I also increased the pipeline.batch.size property in Logstash from 300 to 550 events.

How could we replicate the issue?
Just install HELK and monitor for helk-logstash logs:

sudo docker logs --follow helk-logstash

If you are having issue during the installation stage, please provide the HELK installation logs located at /var/log/helk-install.log

What version of HELK are you using?
6.2.4

What OS are you using to host the HELK?
Linux Ubuntu

Any additional context?
References: https://kafka.apache.org/documentation/#newconsumerconfigs

[Cyb3rWard0g]

Updated pipeline.batch.size to 325 and the max_poll_records to 300. better performance in the lab environment in a few computers. testing it more. so far I have not had any error messages.

[Cyb3rWard0g]

Updated max_poll_records to 325 and performance improved. Setting the kafka input property max_poll_records to the same value as the pipeline.batch.size value improves performance.

[Cyb3rWard0g]

Updated max_poll_records and pipeline.batch.size to 500. Performance has been steady with HELK running on a local VM. (2 virtual cores). 150 event/s 👍 Not tested at scale yet. Error is not showing anymore. Updates to request_timeout_ms and session_timeout_ms helped to fix the error message. Closing this issue with commit.

[Cyb3rWard0g]

More references:
https://discuss.elastic.co/t/kafka-input-performance-config-tuning/70923
https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-max_poll_interval_ms
logstash-plugins/logstash-input-kafka#114

[Saurabh995]

THis is a compatibility issue , try updating the kafka version