[Stack Monitoring] Add support for beats datastream patterns

[klacabane]

Summary

Closes #146686

Update apm and beats queries to read from the Beat package data streams created in elastic/integrations#4708. Also updates health API to fetch from the data streams.

API tests follow up in #147755

Testing

That's a fairly heavy workflow. I'm investigating ways to make that easier

• build beat package with elastic-package build. The package is not published yet so you'll need to pull [beat] Create beat package integrations#4708 locally
• start a stack elastic-package stack up -v -d --version 8.7.0-SNAPSHOT. Make sure you do this within integrations repo so it picks up the previously built package
• start a beat service with elastic-package service up -v --variant metricbeat_8.7.0
• install both elasticsearch and beat packages from the kibana started by the stack command (https://localhost:5601). Nothing shows up in SM if we don't have elasticsearch data.
  • both packages are prerelease versions and we need to explicitly tell Integrations plugin to show them up[1]
  • elasticsearch hosts is https://elasticsearch:9200 and beat should be http://elastic-package-service_beat_1:5066 but it may differ depending on your docker version
• start a Kibana with this branch connected to the elasticsearch instance from the stack up command. see howto
• navigate to Stack Monitoring and verify the metricbeat is properly monitored
• start an apm server with elastic-package service up -v --variant apm_8.7.0
• navigate to Stack Monitoring and verify apm-server is properly monitored

[1]

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[miltonhultgren]

If you select a very large time range the graphs go blank and the metrics are N/A, this applied to ES views as well.

The API responds with no data.

[miltonhultgren]

LGTM 👍🏼 Tested both types locally

[miltonhultgren]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 5558528

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	61	67	+6
osquery	109	115	+6
securitySolution	439	445	+6
total			+20

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	70	76	+6
osquery	110	117	+7
securitySolution	516	522	+6
total			+21

History

• 💚 Build #95476 succeeded dd9ac34
• 💛 Build #95415 was flaky c4ea25a
• 💚 Build #92643 succeeded 2b350df
• 💚 Build #92127 succeeded a25a237
• 💚 Build #91932 succeeded 315e3ef
• 💚 Build #90740 succeeded 6cc4ba5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @klacabane

[klacabane]

Will add api tests in #147755

[klacabane]

Ack value stays flat at 0 which is caused by missing assets used to store apm data:

{"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-apm.error] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-apm.error] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}

I'll create a follow up to install apm integration as part of the apm setup script