Kibana Stack monitoring page not showing data from metricbeats

[souravsekhar]

Kibana version: 7.7.0

Elasticsearch version: 7.7.0

Metricbeat version: 7.7.1

License: Basic

Server OS version: Centos 7

Browser version: Chrome 83.0.4

Original install method (e.g. download page, yum, from source, etc.):

wget -O kibana-7.7.0-linux-x86_64.tar.gz https://artifacts.elastic.co/downloads/kibana/kibana-7.7.0-linux-x86_64.tar.gz
wget -O kibana-7.7.0-linux-x86_64.tar.gz.sha512 https://artifacts.elastic.co/downloads/kibana/kibana-7.7.0-linux-x86_64.tar.gz.sha512

Describe the bug:

I have setup a separate monitoring cluster for collecting monitoring data from the production cluster. I have installed metricbeat on each of the production cluster's nodes and metricbeats is successfully sending the metric related data to monitoring cluster. I have a separate kibana instance which is part of the monitoring cluster. On the kibana instance, there is a coordinating node configuration too and the source for kibana monitoring data is http://0.0.0.0:9200

However, I am not seeing any data on the kibana's monitoring page. I see the default page where it asks for options to enable the monitoring.

NOTE: the production cluster is security enabled but the monitoring cluster is not.

Steps to reproduce:

1. Production cluster with 3 master and multiple data nodes, security enabled. Metricbeat running on each node
   2.Separate monitoring cluster with 3 master, 2 data, 1 coordinating nodes with Kibana on same server. Security is not enabled.
2. Below are the configs:-

Production ES settings (Datanode & Master node)

xpack.monitoring.elasticsearch.collection.enabled: false
xpack.monitoring.collection.enabled: true

The above settings are applied on each of the master and data nodes.

NOTE: x-pack security is enabled in production cluster

The separate monitoring cluster has 3 master nodes with 2 data nodes and 1 coordinating node. Coordinating node and Kibana runs on the same machine.

There are no monitoring settings applied on any of the nodes in monitoring cluster, except on Kibana, below are the settings:-

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://0.0.0.0:9200"]
monitoring.ui.enabled: true
xpack.monitoring.kibana.collection.enabled: false

NOTE: x-pack security is not enabled on the monitoring cluster

Kibana instance is a part of the monitoring cluster

Below is the metricbeat.yml (metricbeat is running on every node of the production ES cluster)

metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false

setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
#_source.enabled: false

setup.kibana:
host: "x.x.x.x:5601"

output.elasticsearch:
# Array of hosts to connect to.
hosts: ["x.x.x.x:9200"]

processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~

elasticsearch-xpack.yml

metricsets:
ccr
cluster_stats
enrich
index
index_recovery
index_summary
ml_job
node_stats
shard
period: 10s
hosts: ["https://elasticsearch-host-production-cluster:9200"]
username: "XXXXX"
password: "XXXXX"
xpack.enabled: true

Expected behavior: Stack monitoring page in Kibana (which is part of monitoring cluster) should display the dashboards

Screenshots (if relevant):

Stack monitoring page

Indices in monitoring cluster

[souravsekhar]

@elastic/kibana-monitoring

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[souravsekhar]

The issue was with enabling the elasticsearch module, I had to explicitly provide the config path while enabling the module.