[Beats] Create integration dashboards similar to Stack Monitoring UI

[jlind23]

To accomplish this we will build new Kibana dashboards and ship them in integration packages so that users have an option to visualize monitoring data out of the box.

Later, these dashboards will be linked to or embedded into a new Ingest UI to give users a global overview of data ingestion metrics and management.

These new dashboards should strive to follow the best practices for integration dashboards. In the first iteration, we're aiming to simply replace the clone Stack Monitoring UI as much as we can 1-1.

Testing instructions

First deploy a local stack on your machine using elastic-package:
elastic-package stack up -v -d --version 8.9.0

As soon as the stack is up you should see the following console output:

/Users/julienlind/Downloads/elastic-package_0.84.0_darwin_amd64/elastic-package stack status
Status of Elastic stack services:
╭──────────────────┬────────────────┬───────────────────╮
│ SERVICE          │ VERSION        │ STATUS            │
├──────────────────┼────────────────┼───────────────────┤
│ elastic-agent    │ 8.9.0-SNAPSHOT │ running (healthy) │
│ elasticsearch    │ 8.9.0-SNAPSHOT │ running (healthy) │
│ fleet-server     │ 8.9.0-SNAPSHOT │ running (healthy) │
│ kibana           │ 8.9.0-SNAPSHOT │ running (healthy) │
│ package-registry │ latest         │ running (healthy) │
╰──────────────────┴────────────────┴───────────────────╯

Then you need to generate some monitoring metrics, so you can start by installing metricbeat and enabling the system metric:

> curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.8.2-darwin-x86_64.tar.gz
> tar xzvf metricbeat-8.8.2-darwin-x86_64.tar.gz
> cd metricbeat-8.8.2-darwin-x86_64
> sudo chown root metricbeat.yml
> sudo chown root modules.d/system.yml
> sudo ./metricbeat -e

In the metricbeat.yml config file you should set up your ES output and turn on monitoring.

You should now see data flowing in your Stack monitoring menu under the Beats section:

Here is a screenshot that contains both new dashboard (on the left) and Stack Monitoring UI (on the right).

The main differences here is on the CPU Utilization viz as the Stack Monitoring UI has a data point every 10 seconds whereas in dashboard there is one every 30 seconds.

The screenshot below represents the overview dashboard.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[jlind23]

After discussing this with @klacabane there are a couple of aggregations done today that can not easily be done in a simple dashboard, so for now the path forward would be:

• A first part of the dashboard that will used to list all the Beats currently running to monitor.
• A dashboard to dive into each beats

cc @joshdover @nimarezainia

[jlind23]

@jsoriano @mrodm I found something weird while trying to export a dashboard.
I added a control that is a filter based on one of the field. The index I use is an hidden one .monitoring-beats-* after saving the dashboard I tried to export it and I faced this error:
Error: dashboards export failed: exporting dashboards using Kibana client failed: at least Kibana object returned an error: [0] Saved object [index-pattern/02ff4995-b664-4c37-b5ad-2795b7eef2d0] not found

Removing the filter solved this. Have you ever experienced this?

[mrodm]

That error looks like is raised when getting the objects from kibana, even before elastic-package applies any transformations.

@jlind23 Could you check if that error also is raised when exporting the dashboard from the UI ?
https://localhost:5601/app/management/kibana/objects

Select the dashboard and export it including references.

[jlind23]

Same issue while exporting from Kibana.

[mrodm]

Not sure then what could be the cause of this.

Could it be that the index-pattern (or data view) has been used in that dashboard without being saved ? Maybe it's not even possible to do that...

[elasticmachine]

Package beat - 1.0.0-beta1 containing this change is available at https://epr.elastic.co/search?package=beat