[Dataset quality] using msearch to speed up degradedDocs query

[yngrdyn]

Relates to #179227.

After gathering some numbers around possible tweaks of the current degradedDocs query (more information), I decided to move forward and split the query to reduce the time taken by elastic search aggregating on data streams.

This PR contains the following changes:

• mSearch method was added to DatasetQualityESClient to allow the usage of multi search.
• degradedDocsRt was changed to now include not only the amount of degradedDocs but also the total docs for the datastreams within the timerange selected

Nothing visible has changed in terms of functionality

Screen.Recording.2024-05-09.at.12.33.53.mov

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mohamedhamed-ahmed]

do we actually need this query to get the total docs? don't you automatically get the total doc counts from bucket.doc_count?

my thoughts was the we just need to add 1 more prop and assign the value to it.

[yngrdyn]

No, in the first query we just get the total number of documents with _ignored not null. A bucket in the first query will look like

{
  "key": {
    "dataset": "apm.error",
    "namespace": "default"
  },
  "doc_count": 102
},

Notice that we are not doing the nested aggregation anymore, which I have the theory is the most expensive one. And yes, we do need the total documents in the timerange to get the ratio (percentages).

[mohamedhamed-ahmed]

LGTM! Thanks for the quick change 🚀

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: dd6ef15
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-183023-dd6ef1545cfb
• Observability Deployment

Metrics [docs]

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
module count	-	5407	+5407
total size	-	8.8MB	+8.8MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
datasetQuality	36.2KB	36.3KB	+19.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @yngrdyn