Fix the telemetry collection of Logstash with metricbeat monitoring.

[mashhurs]

Summary

Telemetry data collection is broken for Logstash, monitoring with metricbeat. This PR change covers following issues faced:

1) Resolve cluster UUID

• With self monitoring, KB creates .monitoring-es* index with mapping type field and defaults to type:cluster_state key-value. It uses type:cluster_state condition when fetching cluster UUID.
• However, with metricbeat, this type field doesn't exist under mapping which metricbeat creates, so cluster UUID will not be resolved as query is wrong (results empty output).

2) type field mismatch in (especially in state) queries, also collapse field

• metricbeat sends the state event with metricset.name:node and state fetch query doesn't care about this condition, instead uses legacy type:logstash_state condition which is incorrect.
• in the queries, collapse field is not correct: it is due to data shape change from legacy to metricbeat monitoring and queries are tightly coupled with legacy one (1, 2, 3)
• in the queries, filter_path is also not correct: in both state query and stats query

3) Logstash state data frequency

• metricbeat sends state event (_node/stats/pipeline?graph=true) data once but legacy frequently sends. KB telemetry fetcher queries against ES with latest update period where state data will not be available. It might be a reasonable design which results network efficiency. If that's the case, we should decide the expectation
  • to still collect: we have to tune the query to collect the state data.
  • leave it as an empty assuming state didn't change (personally, I would not align with this option since collecting once it a data loss risky)

Checklist

Delete any items that are not applicable to this PR.

• [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• [ ] Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• [ ] Flaky Test Runner was used on any tests changed
• [ ] Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• [ ] This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• [ ] This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[afharo]

Thank you for taking a look at this! We needed someone with knowledge about the new shape of the data to fix this. 🧡

It Looks Great To Me! My only comment is: can we normalize the response of get_es_stats so that it doesn't leak out to other plugins and functions?

[mashhurs]

FYI: I have updated the unit test cases which align with current changes, wil try to add for metricbeat.

[afharo]

Thank you for bearing with me and explaining the reasoning behind the changes.

FWIW, my main concern is that we're leaking code outside the monitoring plugin. Other than that, all looking good aside from the global logstash request now being moved inside the loop.

[afharo]

Hmm, the failing tests indicate that we're somehow not returning the monitoring data... Maybe the new query is failing?

UPDATE: Found it in the logs:

[00:00:06]           │ proc [kibana] [2024-05-08T00:55:24.693+00:00][WARN ][plugins.usageCollection.usage-collection.collector-set] ResponseError: search_phase_execution_exception
[00:00:06]           │ proc [kibana] 	Caused by:
[00:00:06]           │ proc [kibana] 		illegal_argument_exception: no mapping found for `logstash.node.stats.logstash.uuid` in order to collapse on
[00:00:06]           │ proc [kibana] 	Root causes:
[00:00:06]           │ proc [kibana] 		illegal_argument_exception: no mapping found for `logstash.node.stats.logstash.uuid` in order to collapse on
[00:00:06]           │ proc [kibana]     at KibanaTransport.request (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@elastic/transport/lib/Transport.js:492:27)
[00:00:06]           │ proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:95:5)
[00:00:06]           │ proc [kibana]     at KibanaTransport.request (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
[00:00:06]           │ proc [kibana]     at ClientTraced.SearchApi [as search] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@elastic/elasticsearch/lib/api/api/search.js:66:12)
[00:00:06]           │ proc [kibana]     at fetchLogstashStats (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/monitoring-plugin/server/telemetry_collection/get_logstash_stats.js:225:19)
[00:00:06]           │ proc [kibana]     at getLogstashStats (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/monitoring-plugin/server/telemetry_collection/get_logstash_stats.js:312:5)
[00:00:06]           │ proc [kibana]     at async Promise.all (index 2)
[00:00:06]           │ proc [kibana]     at getAllStats (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/monitoring-plugin/server/telemetry_collection/get_all_stats.js:34:49)
[00:00:06]           │ proc [kibana]     at async Promise.all (index 1)
[00:00:06]           │ proc [kibana]     at Collector.fetch (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/monitoring-plugin/server/telemetry_collection/register_monitoring_telemetry_collection.js:227:33)
[00:00:06]           │ proc [kibana]     at CollectorSet.fetchCollector (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/usage-collection-plugin/server/collector/collector_set.js:141:24)
[00:00:06]           │ proc [kibana]     at fetch_monitoringTelemetry (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/usage-collection-plugin/server/collector/collector_set.js:175:103) {"service":{"node":{"roles":["background_tasks","ui"]}}}

[mashhurs]

@elasticmachine merge upstream

[afharo]

LGTM! This is great! Thanks for such an effort!

[mashhurs]

LGTM! This is great! Thanks for such an effort!

Thank you so much @afharo. This happened because of your huge help, appreciate!

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 37f3912

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
monitoring	18	20	+2

Total ESLint disabled count

id	before	after	diff
monitoring	25	27	+2

History

• 💚 Build #208859 succeeded 22b51e8
• 💔 Build #208800 failed 1701038
• 💔 Build #208539 failed f86444c
• 💔 Build #208309 failed af3c267
• 💔 Build #208301 failed 73160e1
• 💔 Build #207998 failed 540b7ce

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mashhurs

[mashhurs]

@afharo, @neptunian can we please backport this change to upcoming 8.14.x releases?

[afharo]

I've added the appropriate label to back port this PR to the previous minor.

Did the same with #182857

Hopefully, our kibanamachine bot backports them for us.

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.14

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[miltonhultgren]

@afharo Thank you so much for sharing all your knowledge here and getting this to done!