[Monitoring] [Packaging] Include Metricbeat/Filebeat with Kibana package

[cachedout]

With the 8.0 release of the stack, internal collection will be removed from Kibana.

As such, each stack team may consider bundling Metricbeat with their software package so that it's available for users to configure and quickly set up monitoring.

The Elasticsearch team is planning to go ahead with this for Elasticsearch and, according to comment from @jasontedor in a meeting yesterday, they'll likely take the route of bundling Metricbeat and then having it be started automatically by init scripts alongside Elasticsearch. Metricbeat configuration, however, will remain separate and distinct from Elasticsearch.

This issue is being filed in the Kibana repo to discuss with the team whether Kibana wishes to adopt a similar pattern and, if so, any implementation details that may need to be discussed.

Below are complementary issues in the Elasticsearch and Logstash repos, just for reference:

Elasticsearch: elastic/elasticsearch#49399
Logstash: elastic/logstash#11328

cc: @elastic/stack-monitoring

@epixa and @tylersmalley I assigned this to the both of you since I think you're both familiar with the context here but of course please feel free to re-assign as necessary. Thanks!

[epixa]

I think this is an appropriate route to go down for our monitoring agent, and it's inline with discussions we've had around handling logging and such as well (through filebeat).

My biggest concern is in increased artifact size, which is something we'll need to evaluate.

We'll need to do some basic benchmarking to compare resource usage to make sure we're not complicating the story on Cloud.

Child process management is easy enough with node, so I don't necessarily see a problem there. Some day we need to get the child process spawning handled through the platform itself, but that shouldn't block this.

We already do architecture-specific builds, so no issue there.

@elastic/kibana-operations am I missing anything?

[tylersmalley]

@epixa, that should cover it. I am tagging this operations as this seems like it would be a joint effort with monitoring. We can work on embedding the beats, and monitoring can work to remove the hooks in the codebase.

[jbudz]

@pmuellr @tylersmalley had a mini discussion on this. Are we okay with UI dependencies on this workflow or should these scenarios be managed internally?

Context: we introduce a log of historical alerts, and an alert viewer. Is a dependency on filebeat being enabled acceptable?

+@kobelb

[tylersmalley]

To add another example; we're planning to add Elasticsearch query logging and it would be great to be able to list the average query time for a particular visualization in the UI. Or, curate a list of the slowest queries for visualizations or searches.

Utilizing these log events and Filebeat makes sense to me. What I am not sure of if they should be separate log files on the filesystem but still ingested into the same index. The user should be able to include/exclude log types, and the UI can provide information when these logs are missing on how to enable.

[tylersmalley]

One possible reason to not piggyback on the Kibana system logs for use in the UI would be how Cloud intends on using them. If Cloud intends on utilizing the integrated Filebeat, but wanted to continue to direct these at the centralized logging cluster, then the end user wouldn't have access to them. We would either need the ability to have multiple log configurations/outputs (Cloud only needs information helpful for debugging an issue) or we need to simply keep these paradigms separate.

[tylersmalley]

Closing for the same reason mentioned here