[Infra UI] Terms aggregation query for modules and datasets times out

[skh]

(found by: @Kerry350 )

This query started to time out on the 8.0.0 Observability cluster: https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/infra/server/lib/adapters/fields/framework_fields_adapter.ts#L53

Due to this, the Infra and Logs UI can't be loaded. The problem can be reproduced in the Kibana Dev Tools Console with the following query:

GET metricbeat-*/_search
{ 
  "aggs": {
    "modules": {
      "terms": {
        "field": "event.module",
        "size": 1000
      }
    },
    "dataSets": {
      "terms": {
        "field": "event.dataset",
        "size": 1000
      }
    }
  }
}

After some discussions in slack, the following ideas came up to improve this:

• add size: 0 to the query to make it eligible for caching
• restrict the time range
• use a composite query instead

[elasticmachine]

Pinging @elastic/infra-logs-ui

[jasonrhodes]

Separate but related: it looks like we are possibly using the wrong field name here
https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/infra/server/lib/adapters/fields/framework_fields_adapter.ts#L65

"event.modules"

vs

https://www.elastic.co/guide/en/ecs/current/ecs-event.html

event.module

[simianhacker]

The quick wins for this are:

• convert the query from a "large" terms aggregation into a composite aggregation (that paginates).
• set size: 0 on the query

Restricting the time range gets trickier because this call is happening as part of the source request which doesn't send any time range information. What makes this challenging is that they way the source works today, there is a hook that is used in the context. The call to load the source from the server is triggered inside a useEffect (inside a hook) in the source "container". Changing how this works has sweeping consequences since not every page that calls this service has an available time range.

Since we can't guarantee that every request to the source endpoint has a valid time range (and considering the current design), we could limit it to the last 24 hours. There are 2 possible approaches:

• Limit the query with { query: { range: { "@timestamp": { gte: "now-24h", lte: "now" } } } } which will limit the results to the last 24 hours (relative to the server). Level of effort is as easy as adding the aforementioned query to the request.
• Send the current browser timestamp with (every) source query and then use that to limit the request to the last 24 hours (relative to the browser). Level of effort is easy but will require changing the GraphQL endpoint along with the request to send the current browser time.

@jasonrhodes Thoughts?