Icingabeat v.6.3.3 does not send the hostname for the check, instead send own hostname

[5nafu]

After installing icingabeat (6.3.3) on a complete new machine, with minimal changes to the icingabeat.yml and outputting to a completely new elasticsearch index (5.5), accessing an icinga2 (2.9.1), icingabeat will log checkresults with the name of the host running icingabeat as host.name:

My icingabeat.yml (with comments, hostnames and credentials redacted):

icingabeat:
  host: "XXXX"
  port: 5665
  user: "XXXX"
  password: "****"
  ssl.verify: false
  eventstream.types:
    - CheckResult
    - StateChange
    - Notification
    - AcknowledgementSet
    - AcknowledgementCleared
    - CommentAdded
    - CommentRemoved
    - DowntimeAdded
    - DowntimeRemoved
    - DowntimeStarted
    - DowntimeTriggered

  eventstream.filter: ""
  eventstream.retry_interval: 10s
  statuspoller.interval: 30s
setup.dashboards.enabled: false
setup.kibana:
output.elasticsearch:
  hosts: ["XXXX:9200"]
  protocol: "https"
  username: "XXXX"
  password: "*****"

Representative checkresult (in json):

{
  "_index": "icingabeat-6.3.3-2018.09.12",
  "_type": "doc",
  "_id": "AWXO07xNOFh-eQsJ_2M4",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-09-12T17:27:45.615Z",
    "check_result": {
      "active": true,
      "type": "CheckResult",
      "execution_end": "2018-09-12T17:25:24.348Z",
      "check_source": "icinga2-sat...",
      "state": 0,
      "vars_before": {
        "reachable": true,
        "state": 0,
        "state_type": 1,
        "attempt": 1
      },
      "exit_status": 0,
      "output": "OK",
      "schedule_end": "2018-09-12T17:25:24.348Z",
      "ttl": 0,
      "schedule_start": "2018-09-12T17:25:23.970Z",
      "execution_start": "2018-09-12T17:25:23.970Z",
      "vars_after": {
        "reachable": true,
        "state": 0,
        "state_type": 1,
        "attempt": 1
      },
      "command": [
        "/usr/lib/nagios/custom_plugins/check_multi",
       "..."
      ]
    },
    "beat": {
      "version": "6.3.3",
      "name": "icingabeat-stg-01",
      "hostname": "icingabeat-stg-01"
    },
    "host": {
      "name": "icingabeat-stg-01"
    },
    "service": "$SERVICENAME",
    "timestamp": "2018-09-12T17:25:24.352Z",
    "type": "icingabeat.event.checkresult"
  },
  "fields": {
    "check_result.execution_end": [
      1536773124348
    ],
    "check_result.schedule_end": [
      1536773124348
    ],
    "@timestamp": [
      1536773265615
    ],
    "check_result.execution_start": [
      1536773123970
    ],
    "check_result.schedule_start": [
      1536773123970
    ],
    "timestamp": [
      1536773124352
    ]
  },
  "sort": [
    1536773265615
  ]
}

Expected Behavior

Somewhere in the document should the hostname for the checkresult (notification, ...) be analog to the service.

Current Behavior

All hostnames will be the icingabeat hostname

Steps to Reproduce (for bugs)

1. install icingabeat as documented
2. configure icingabeat with access to icinga & elasticsearch
3. let icingabeat create the elasticsearch template icingabeat setup
4. restart icingabeat
5. create index patterns in kibana (and ignore error while importing kibana dashboards)
6. check resulting documents in Kibana

Context

We can not use icingabeat 6.3.3 as there is no way to know to which of our hosts the messages are related to.

Your Environment

• Beat version (icingabeat -version): icingabeat version 6.3.3 (amd64), libbeat 6.3.3
• Icinga 2 version (icinga2 --version): r2.9.1-1
• Elasticsearch version (curl -XGET 'localhost:9200'): 5.5.0
• Logstash version, if used (bin/logstash -V): 5.6.7 (not configured to be used)
• Kibana version, if used (curl -XGET http://localhost:5601/status -I): 5.5.0
• Operating System and version: Debian GNU/Linux 9.5 (stretch)

[ekeih]

I can confirm this issue. We had to downgrade to 6.1.1 as a workaround.

[yoshi314]

on my end downgrade makes elastic reject the fields from 6.1.1

[5nafu]

Aparently this seems to be connected to the upgrade to libbeat 6.3:

As a solution for elastic/beats#7050, we're adding a host.name field to
all events. This is duplicate information from beat.name,
but is used to avoid the mapping conflict and to slowly
introduce the "host as an object" approach.

elastic/beats#7051

[jcarterch]

Since host.name is now populated by libbeat, it seems like we'd need to rename the host field, or move it out of the document root.

Moving it out could be as simple as

--- a/beater/eventstream.go
+++ b/beater/eventstream.go
@@ -51,7 +51,11 @@ func BuildEventstreamEvent(e []byte) beat.Event {
        event.Fields = common.MapStr{}

        for key, value := range icingaEvent {
-               event.Fields.Put(key, value)
+               if key == "host" {
+                       event.Fields.Put("event.host", value)
+               } else {
+                       event.Fields.Put(key, value)
+               }
        }

which will populate event.host in your beat output with the host value from the original event.

Since host shows up in a number of different event stream types, I think it makes sense to use a generic event key (and not nest it in check_result) so you can search by host across all event types.

Any thoughts? Renaming to avoid collision might make more sense, with the rest of the fields (e.g., service) living in the document root.

[bobapple]

Same issue as in #26