Bad error message regarding missing key

[abhishenator]

beats/metricbeat/module/jolokia/jmx/data.go

Line 84 in d8282c5

return errors.Errorf("metric key '%v' not found in response", key)

If the metricname is missing from map, key would always be empty. I m getting following error with jolokia module. Not sure what is the problem here:

2017-11-25T09:16:10Z DBG  [jolokia.jmx] The response body from jolokia host localhost:8778 is: [{"request":{"mbean":"kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics","attribute":"Count","type":"read"},"value":{"Count":2862},"timestamp":1511601370,"status":200}]
2017-11-25T09:16:10Z DBG  Publish event: {
  "@timestamp": "2017-11-25T09:16:10.564Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "doc",
    "version": "6.0.0"
  },
  "jolokia": {
    "jmx": {}
  },
  "metricset": {
    "rtt": 990,
    "module": "jolokia",
    "name": "jmx",
    "host": "localhost:8778"
  },
  "error": {
    "message": "1 error: metric key '' not found in response"
  },
  "beat": {
    "name": "<hidden>",
    "hostname": "<hidden>",
    "version": "6.0.0"
  }
}

Similar issue was discussed here: https://discuss.elastic.co/t/metric-beat-jolokia-module-cannot-read-attribute-value/92190. As claimed in the post, it seemed to have been fixed in 6.0.0-alpha2 release. I m using latest metricbeat docker image which is 6.0.0. Jolokia version used is 1.3.7.

[ruflin]

Could you share your full metricbeat config?

[abhishenator]

Hi Nicolas,

Here is the config:

$ cat metricbeat.yml  |grep -v '\#' | tr -s '\n'

metricbeat.config.modules:
  path: ${path.config}/conf.d/*.yml
  reload.period: 10s
  reload.enabled: false
metricbeat.max_start_delay: 10s
metricbeat.modules:
- module: system
  metricsets:
  enabled: true
  period: 30s
  processes: ['.*']
- module: docker
  metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true
- module: jolokia
  metricsets: ["jmx"]
  period: 10s
  hosts: ["localhost:8778"]
  namespace: "metrics"
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  jmx.application:
  jmx.instance:
  enabled: true
  jmx.mappings:
    - mbean: 'kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec'
      attributes:
        - attr: Count
          field: count <--- I have tried removing this line also.
- module: kafka
  metricsets: ["partition", "consumergroup"]
  period: 10s
  hosts: ["localhost:9446"]
  enabled: true
output.elasticsearch:
  enabled: true
  hosts: ["10.4.244.155:9200"]
output.file:
  enabled: false
  path: "/tmp/metricbeat"
setup.dashboards.enabled: true
setup.kibana:
  host: "10.4.244.155:5601"

logging.level: debug
logging.to_files: true
logging.files:

Please let me know if you need anything else. I m currently blocked on this. Thanks

[ruflin]

The output of Jolokia reported in the logs above looks as following:

[
    {
        "request": {
            "mbean": "kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics",
            "attribute": "Count",
            "type": "read"
        },
        "value": {
            "Count": 2862
        },
        "timestamp": 1511601370,
        "status": 200
    }
]

A few ideas. Could you try the following config where the mbean is the same order as in the output?

  jmx.mappings:
    - mbean: "kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics"
      attributes:
        - attr: Count
          field: count

It should not make a difference. If it does not work, can you try to make field: Count upper case?

The last difference I see to all the objects we tests, is that we request multiple values and not only one. In these case of one value the data is not structured as an array. I wonder if this breaks our logic. Could you try to add a second value?

[abhishenator]

Thanks so much for help. Changing the mbean name to as reported in jolokia response worked: "kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics". You can close the issue.

[ruflin]

Great to hear it worked. Closing this. Probably we should add something about the order of the params to the docs.