Metricbeat Kafka module does not publish consumergroup metrics (Cluster mode)

[rahuljoshitaulia]

Test setup:

1. Setup a kafka and zookeeper cluster for HA using 3 kafka nodes and 3 zookeeper nodes. Apache open source documentation is available on their website on how to set this up. Use latest kafka version. Make sure kafka is configured to store offsets on kafka broker and not on zookeeper.
2. Install metricbeat 5.3.2 on each kafka node
3. Configure each metricbeat installation to collectpartition and consumergroup metricsets. Send metrics to elastic search. turn on debug logging in the metricbeats configuration for *
4. Setup Logstash server to produce and consume from the kafka server. Produce to 1 one topic with 10 partitions and and startup 4 consumer groups consuming from that topic. You will have to change the logstash configuration provided below to use the parameter bootstrap_servers => "server1,server2,server3" so that logstash connects to the cluster.

Sample logstash configuration for producer:

input {
  generator {}
}

filter {
  sleep {
    time => "1"
    every => 1000
  }
}

output {
  kafka {
    client_id => "lsp"
    topic_id => "test1"
  }
}

Sample logstash configuration for consumer:

input {
  kafka {
    group_id => "lsc1"
    topics => "test1"
  }
  kafka {
    group_id => "lsc2"
    topics => "test1"
  }
  kafka {
    group_id => "lsc3"
    topics => "test1"
  }
  kafka {
    group_id => "lsc4"
    topics => "test1"
  }
  kafka {
    group_id => "lsc5"
    topics => "test1"
  }
}

output {
  null {}
}

Test Results:
You will notice in the metricbeats logs that you are receiving events for the partition metricset but not for the consumergroup metricset.

For a more detailed discussion of this issue you can check the following thread on the forum:
https://discuss.elastic.co/t/metricbeat-kafka-module-consumergroup-metric-set-does-not-report-metrics/83623

[ruflin]

Thanks a lot for the detailed steps on how to reproduce it.

[luckb008]

@ruflin Hi Nicolas. Do you have any ideas when this bug will be fixed?

[huangchaosuper]

we get the same problem

[ghost]

Got the same problem when updated Kafka cluster from 0.10.0.1 to 0.10.2.1 and reporting of consumer metrics stopped

[sergiolr100]

Same problem, something new?

[iyabchen]

It doesn't work for me even for single node of kafka. Have to send partitions with offsetFetchRequest to get something back.

[BDeus]

I confirm that consumergroup metricset is broken with new kafka version.
I think this issue (#3608) has plan to replace this metricset ?

[nerophon]

@ruflin I believe the above merged PR 5880 may fix this bug. Can we confirm this, and on what versions?
UPDATE: actually this is not fully tested yet according to @urso, and should come as part of a larger Kafka effort.

[npredey]

Running Metricbeats version 5.3.0 using Elasticsearch 5.3 and Logstash 5.3, I get the following JSON output:

"kafka": {
    "consumergroup": {
      "broker": {
        "address": "some-address",
        "id": 2
      },
      "client": {
        "host": "10.70.244.78",
        "id": "some-client-id",
        "member_id": "8c006f3-b42f-4f7d-b19e-630050ce00b4"
      },
      "error": {
        "code": 0
      },
      "id": "id-1",
      "meta": "",
      "offset": 71796868,
      "partition": 9,
      "topic": "sample-topic"
    }
  },
  "metricset": {
    "host": "hostname",
    "module": "kafka",
    "name": "consumergroup",
    "rtt": 213410
  }

It seems that most of those fields are represented in the Metricbeats consumergroup reference. Am I missing something? The bug doesn't seem to be affecting me.

[tsg]

I'm closing this one as I believe it was solved in the meantime.