Elasticsearch module for stack monitoring sends incorrectly formatted documents

[Evesy]

Using the Elasticsearch module in Metricbeat to collect data for Kibana's stack monitoring as mentioned here does not produce documents that Kibana expects.

I have the below configuration on 8.8.1:

    - module: kubernetes
      metricsets:
        - event
      period: 10s
    - module: beat
      period: 10s
      hosts: ["http://localhost:5066"]
      xpack.enabled: true
    - hosts:
      - http://elasticsearch-client.default:9200
      metricsets:
      module: elasticsearch
      period: 10s
      scope: cluster
      xpack.enabled: true
    - enabled: true
      hosts:
      - http://kibana-ui.default:5601
      module: kibana
      period: 10s
      xpack.enabled: true

(I have also tried explicitly setting all the metricsets)

This correctly creates and emits data to a datastream in Elastic, .monitoring-beats-8-mb, however all of Kibana's query's for monitoring data expect the terms in this filter to exist, however the documents being emitted by metricbeat have none of these terms

Sample document from the ES monitoring index:

{
        "_index": ".ds-.monitoring-es-8-mb-2023.06.15-000001",
        "_id": "L4V1w4gBRWMS8-YHUAmY",
        "_score": null,
        "_source": {
          "@timestamp": "2023-06-16T09:07:04.182Z",
          "cloud": {
            "project": {
              "id": "at-signals-platform-nonprod"
            },
            "account": {},
            "service": {},
            "instance": {},
            "machine": {}
          },
          "service": {
            "address": "http://elasticsearch-client.default:9200",
            "name": "elasticsearch"
          },
          "event": {
            "duration": 311450477
          },
          "metricset": {
            "period": 10000
          },
          "elasticsearch": {
            "node": {
              "stats": {
                "os": {
                  "cpu": {
                    "load_avg": {
                      "1m": 0.08
                    }
                  },
                  "cgroup": {
                    "cpuacct": {
                      "usage": {
                        "ns": 4264477127521
                      }
                    },
                    "cpu": {
                      "stat": {
                        "elapsed_periods": {
                          "count": 0
                        },
                        "times_throttled": {
                          "count": 0
                        }
                      },
                      "cfs": {
                        "quota": {
                          "us": -1
                        }
                      }
                    },
                    "memory": {
                      "usage": {
                        "bytes": "18805882880"
                      },
                      "control_group": "/",
                      "limit": {
                        "bytes": "26843545600"
                      }
                    }
                  }
                },
                "process": {
                  "cpu": {
                    "pct": 0
                  }
                },
                "thread_pool": {
                  "write": {
                    "queue": {
                      "count": 0
                    },
                    "rejected": {
                      "count": 0
                    }
                  },
                  "get": {
                    "queue": {
                      "count": 0
                    },
                    "rejected": {
                      "count": 0
                    }
                  },
                  "search": {
                    "rejected": {
                      "count": 0
                    },
                    "queue": {
                      "count": 0
                    }
                  }
                },
                "jvm": {
                  "mem": {
                    "heap": {
                      "used": {
                        "bytes": 5040035280,
                        "pct": 37
                      },
                      "max": {
                        "bytes": 13421772800
                      }
                    }
                  },
                  "gc": {
                    "collectors": {
                      "old": {
                        "collection": {
                          "ms": 0,
                          "count": 0
                        }
                      },
                      "young": {
                        "collection": {
                          "count": 133,
                          "ms": 1402
                        }
                      }
                    }
                  }
                },
                "indices": {
                  "bulk": {
                    "avg_time": {
                      "ms": 0
                    },
                    "total_size": {
                      "bytes": 0
                    },
                    "total_time": {
                      "ms": 0
                    },
                    "operations": {
                      "total": {
                        "count": 0
                      }
                    },
                    "avg_size": {
                      "bytes": 0
                    }
                  },
                  "fielddata": {
                    "memory": {
                      "bytes": 31368
                    }
                  },
                  "query_cache": {
                    "memory": {
                      "bytes": 1088
                    }
                  },
                  "store": {
                    "size": {
                      "bytes": 4337689508
                    }
                  },
                  "search": {
                    "query_total": {
                      "count": 142569
                    },
                    "query_time": {
                      "ms": 9950
                    }
                  },
                  "indexing": {
                    "index_total": {
                      "count": 0
                    },
                    "throttle_time": {
                      "ms": 0
                    },
                    "index_time": {
                      "ms": 0
                    }
                  },
                  "segments": {
                    "doc_values": {
                      "memory": {
                        "bytes": 0
                      }
                    },
                    "term_vectors": {
                      "memory": {
                        "bytes": 0
                      }
                    },
                    "terms": {
                      "memory": {
                        "bytes": 0
                      }
                    },
                    "count": 308,
                    "memory": {
                      "bytes": 0
                    },
                    "version_map": {
                      "memory": {
                        "bytes": 0
                      }
                    },
                    "points": {
                      "memory": {
                        "bytes": 0
                      }
                    },
                    "fixed_bit_set": {
                      "memory": {
                        "bytes": 12384
                      }
                    },
                    "norms": {
                      "memory": {
                        "bytes": 0
                      }
                    },
                    "stored_fields": {
                      "memory": {
                        "bytes": 0
                      }
                    },
                    "index_writer": {
                      "memory": {
                        "bytes": 0
                      }
                    }
                  },
                  "docs": {
                    "count": 22983396,
                    "deleted": 0
                  },
                  "request_cache": {
                    "memory": {
                      "bytes": 71520
                    }
                  }
                },
                "fs": {
                  "summary": {
                    "total": {
                      "bytes": 2112619954176
                    },
                    "free": {
                      "bytes": 2108274810880
                    },
                    "available": {
                      "bytes": 2108258033664
                    }
                  },
                  "total": {
                    "available_in_bytes": 2108258033664,
                    "total_in_bytes": 2112619954176
                  },
                  "io_stats": {
                    "total": {
                      "read": {
                        "kb": 573980,
                        "operations": {
                          "count": 4946
                        }
                      },
                      "write": {
                        "operations": {
                          "count": 499271
                        },
                        "kb": 28540748
                      },
                      "operations": {
                        "count": 504217
                      }
                    }
                  }
                }
              },
              "id": "_CwUZaQmTiS6awmZRRsYpQ",
              "mlockall": true,
              "master": false,
              "name": "elasticsearch-data-warm-0"
            },
            "cluster": {
              "name": "docker-cluster",
              "id": "4JH7KXiPTd-yk6aXxJ5sxA"
            }
          },
          "ecs": {},
          "host": {},
          "agent": {}
        },
        "sort": [
          1686906424182
        ]
      },

Kibana will throw errors in the logs, Unable to find the cluster in the selected time range. UUID, because it cannot find any documents matching the cluster UUID as none of them have the correct term it is filtering for.

For confirmed bugs, please report:

• Version: 8.8.1
• Operating System: CentOS 8
• Discuss Forum URL: https://discuss.elastic.co/t/stack-monitoring-with-metricbeat-8-no-cluster-found/336104 / https://discuss.elastic.co/t/elasticsearch-stack-monitoring-feature-does-not-work-when-using-metricbeat-to-monitor-elasticsearch-cluster/333451/11
• Steps to Reproduce: Deploy metricbeat configuration as above against an 8.8 Elastic cluster and attempt to access the Kibana stack monitoring page

[botelastic]

This issue doesn't have a Team:<team> label.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!