[Stack Monitoring] separate out mb and legacy index patterns from requests

[neptunian]

Addresses slow loading in the ES overview page #117885.

In 7.14, the shard activity for getting the latest recovery was changed to have 2 separate requests in order to accommodate the introduction of metricbeat-* index pattern. One for metricbeat-* and the other for .monitoring-*. The metricbeat docs are different and the query needs to get 10k documents, but it also still includes the .monitoring index patterns in its search which isn't necessary. The same holds true for the legacy requests querying metricbeat-* index. This separates them out. So a user who has no metricbeat-* indices isn't suddenly going to have a performance issue.

legacy query params before

{
  "index": ".monitoring-es-6-*,.monitoring-es-7-*,metricbeat-*",
  "size": 1,
  "ignore_unavailable": true,
  "body": {
    "_source": [
      "index_recovery.shards"
    ],
    "sort": {
      "timestamp": {
        "order": "desc",
        "unmapped_type": "long"
      }
    },
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "term": {
                    "type": "index_recovery"
                  }
                },
                {
                  "term": {
                    "metricset.name": "index_recovery"
                  }
                }
              ]
            }
          },
          {
            "term": {
              "cluster_uuid": "sVkxxtOdQe-TNy3ECQ0bmQ"
            }
          },
          {
            "range": {
              "timestamp": {
                "format": "epoch_millis",
                "gte": 1638979303889,
                "lte": 1638980203889
              }
            }
          }
        ]
      }
    }
  }
}

legacy query params after

{
  "index": ".monitoring-es-6-*,.monitoring-es-7-*",
  "size": 1,
  "ignore_unavailable": true,
  "body": {
    "_source": [
      "index_recovery.shards"
    ],
    "sort": {
      "timestamp": {
        "order": "desc",
        "unmapped_type": "long"
      }
    },
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "term": {
                    "type": "index_recovery"
                  }
                },
                {
                  "term": {
                    "metricset.name": "index_recovery"
                  }
                }
              ]
            }
          },
          {
            "term": {
              "cluster_uuid": "sVkxxtOdQe-TNy3ECQ0bmQ"
            }
          },
          {
            "range": {
              "timestamp": {
                "format": "epoch_millis",
                "gte": 1638995534235,
                "lte": 1638996434235
              }
            }
          }
        ]
      }
    }
  }
}

metricbeat query params before

{
  "index": ".monitoring-es-6-*,.monitoring-es-7-*,metricbeat-*",
  "size": 10000,
  "ignore_unavailable": true,
  "body": {
    "_source": [
      "elasticsearch.index.recovery",
      "@timestamp"
    ],
    "sort": {
      "timestamp": {
        "order": "desc",
        "unmapped_type": "long"
      }
    },
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "term": {
                    "type": "index_recovery"
                  }
                },
                {
                  "term": {
                    "metricset.name": "index_recovery"
                  }
                }
              ]
            }
          },
          {
            "term": {
              "cluster_uuid": "sVkxxtOdQe-TNy3ECQ0bmQ"
            }
          },
          {
            "range": {
              "timestamp": {
                "format": "epoch_millis",
                "gte": 1638979270889,
                "lte": 1638980170889
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "max_timestamp": {
        "max": {
          "field": "@timestamp"
        }
      }
    }
  }
}

metricbeat query params after

{
  "index": "metricbeat-*",
  "size": 10000,
  "ignore_unavailable": true,
  "body": {
    "_source": [
      "elasticsearch.index.recovery",
      "@timestamp"
    ],
    "sort": {
      "timestamp": {
        "order": "desc",
        "unmapped_type": "long"
      }
    },
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "term": {
                    "type": "index_recovery"
                  }
                },
                {
                  "term": {
                    "metricset.name": "index_recovery"
                  }
                }
              ]
            }
          },
          {
            "term": {
              "cluster_uuid": "sVkxxtOdQe-TNy3ECQ0bmQ"
            }
          },
          {
            "range": {
              "timestamp": {
                "format": "epoch_millis",
                "gte": 1638995534235,
                "lte": 1638996434235
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "max_timestamp": {
        "max": {
          "field": "@timestamp"
        }
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 6f10616
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #12289 succeeded b540506

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @neptunian

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	8.0
✅	7.16

The backport PRs will be merged automatically after passing CI.