Monitor the Elastic Stack with Metricbeat

[ruflin]

Make it possible to fetch logs from the Elastic Stack with Metricbeat.

This is a meta issue to track the ongoing work.

• Kibana
  • Kibana stats metricset: [Kibana Module] Add additional metric fields  #6746
  • Add xpack config to all metricsets
    • Add xpack structure to stats metricset for kibana_stats docs: Create (X-Pack Monitoring) stats metricset for Kibana module #7525
    • Add xpack structure to stats metricset for kibana_settings docs: Report kibana_settings to X-Pack Monitoring #7664
  • Verify parity in documents indexed internally by Kibana (current mechanism) with documents indexed by Metricbeat (new mechanism): https://github.com/elastic/elastic-stack-testing/issues/25
  • Document how users can switch over from Kibana internal collection to Metricbeat for collecting+shipping metrics for X-Pack Monitoring: Document using Metricbeat for collecting and shipping monitoring metrics stack-docs#86
• Elasticsearch
  • Ads index_summary metricset: Elasticsearch index summary metricset #6918
  • Add shard metricset: Add basic shard metricset to Elasticsearch #7006
  • Add ml job metricset: Add ml_job metricset to Elasticsearch module #7196
  • Add index_recovery metricset: Add basic index recovery metricset #7225
  • Add cluster_stats metricset: Adding cluster_stats metricset for elasticsearch module #7638
  • Add ccr_stats metricset: Adding elasticsearch/ccr metricset #8335
  • Add xpack config to all metricsets
    • Add xpack structure to index metricset: Adding x-pack code for elasticsearch/index metricset #8260
    • Add xpack structure to index_summary metricset: Add xpack data structure for Elasticsearch index_summary metricset #7102
    • Add xpack structure to shard metricset: Add x-pack data for Elasticsearch shard metricset #7097
    • Add xpack structure to ml_job metricset: Adding xpack code for ES ML job metricset #8129
    • Add xpack structure to index_recovery metricset: Adding xpack code for ES index recovery metricset #8106
    • Add xpack structure to cluster_stats metricset: Adding xpack code for ES cluster stats metricset #7810
    • Add xpack structure to ccr_stats metricset: Adding x-pack monitoring code for elasticsearch/ccr metricset #8336
  • Create new xpack.monitoring.elasticsearch.collection.enabled setting: Implement xpack.monitoring.elasticsearch.collection.enabled setting elasticsearch#33474
  • Verify parity in documents indexed internally by Elasticsearch (current mechanism) with documents indexed by Metricbeat (new mechanism)
  • Document how users can switch over from Elasticsearch internal collection to Metricbeat for collecting+shipping metrics for X-Pack Monitoring
• Logstash
  • Enhance node metricset for logstash_state documents: Extend logstash.node metricset for logstash_state stack monitoring data #11506
  • Enhance node_stats metricset for logstash_stats documents: Extend logstash.node_stats metricset for logstash_stats stack monitoring data #11511
  • Verify parity in documents indexed internally by Logstash (current mechanism) with documents indexed by Metricbeat (new mechanism): https://github.com/elastic/elastic-stack-testing/pull/250
  • Document how users can switch over from Logstash internal collection to Metricbeat for collecting+shipping metrics for X-Pack Monitoring: Docs for monitoring Logstash with Metricbeat logstash#10920
• Beats including APM-Server
  • Make GET / Beats API return instance UUID: Set beat ID in registries after loading meta file #12180
  • Add stats metricset with xpack structure: Adding beat module with stats metricset #12181
  • Add state metricset with xpack structure
  • Verify parity in documents indexed internally by Beats (current mechanism) with documents indexed by Metricbeat (new mechanism): https://github.com/elastic/elastic-stack-testing/pull/249
  • Document how users can switch over from Beats internal collection to Metricbeat for collecting+shipping metrics for X-Pack Monitoring: Docs for monitoring Beats with Metricbeat #12647
• Common
  • Implement date base index naming for index: XPack helper for naming monitoring indices #7586
  • Assure that all metricsets (non-xpack path) report service.name

[tsullivan]

Does this meta issue also need to talk about removing the internal collectors from the products, and removing the _bulk endpoint from the ES monitoring plugin? If not, where is that being tracked?

[ycombinator]

Does this meta issue also need to talk about removing the internal collectors from the products, and removing the _bulk endpoint from the ES monitoring plugin? If not, where is that being tracked?

Good question! Since @ruflin created the issue originally I'll defer to him on his intent, but my 2 cents would be to keep this issue for just making sure that Metricbeat can monitor the entire stack. Separately, in a (probably much later) release, we'll want to remove internal collectors and the _bulk endpoint from the ES monitoring plugin.

If @ruflin agrees with this scoping, I can create a separate issue for the removal work.

[ruflin]

This issue is only to track the creation of the modules in Metricbeat. There is no issue to track the removal of the endpoint and I don't think we should have one yet as I'm not even sure anymore if we should do this short term. Let's take this discussion offline.

[robcowart]

Is there any targeted date for collecting all of the metric data available via the REST APIs with Metricbeat?

[ruflin]

There will be already a few modules and metricsets in the upcoming 6.4 release as experimental. More to come in the future.

[robcowart]

Good to hear. I am actually close to finishing some cron scheduled scripts to monitor a customer's cluster located close to you @ruflin. They are a platinum user, but had issues with the blocking behavior of X-Pack Monitoring resulting in outages. So I am moving them to a polling-based model.

Something more formal than my scripts, would definitely be the right long-term strategy. So I would prefer to move them towards a metricbeat solution.

Perhaps I will leverage the current metricbeat, and strip out of my scripts the values that it already provides. That way we can transition them over time as new functionality becomes available.

What are the thoughts here around adding dashboards, alerting and ML jobs for these metricsets? I am happy to contribute what I create for this customer.

[ruflin]

Glad to hear it becomes of use for you and the user. Will get back to you about the dashboards, ml jobs contribution in the next days. Thanks for considering it.

[pickypg]

Hi @robcowart

They are a platinum user, but had issues with the blocking behavior of X-Pack Monitoring resulting in outages.

Could you elaborate on what you mean here? X-Pack monitoring is internally a polling based solution where, every interval (10s by default), we collect the relevant stats by polling the various APIs, so I am perhaps misinterpreting what you mean here, but I want to be sure that we do not have a problem.

What are the thoughts here around adding dashboards, alerting and ML jobs for these metricsets?

In the long run, we may go in this direction, but early on our efforts are focused on enabling the curated UI experience. Unfortunately, that means that attempting to simultaneously support separate dashboards would slow down our shift toward Metricbeat / metricsets as we would be forced to validate every change against them. I would be particularly happy to revisit this discussion point once we get the workflow into Metricbeat.

[ycombinator]

All items in the checklist are now completed. Closing this issue.