[alerting] sorted limit of groups in index threshold alert

[pmuellr]

The watcher index threshold alert which the new Kibana alerting index threshold alert is based on, has an option to limit the number of "groups" returned (when using groupField). The Kibana alert supports this, but the watcher one labels it as "Top n of ...", implying that the groups are somehow sorted before limiting, presumably showing you the most relevant groups.

It's not quite clear how this works, given all the aggregation functions. I think for count, average max and sum, you'd basically want to pick the groups that the highest values being processed. For min, you'd want the lowest. For between? And I added a "notBetween" to the Kibana alert. I think maybe we just don't sort for those. note: between is a comparator, not an aggregation

We'll need to figure out how to work this into our query DSL that we are sending. I could see some sorting done with the size limiter, not quite sure if that's still applicable given we're doing a different query than watcher did, but seems like a start.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

Where this gets interesting is when the aggregation function changes - count() avg() min() max() sum(). The query for count() is different from the others, as we don't need a metric aggregation, ES provides the doc counts in the date range buckets.

Somehow near the size: 42 part of the request, we'll want to apply some kind of ordering. This seems to be the best reference on this:

• es reference - terms aggregation - order

request/response with count() over top 42 host.name.keyword

request

{
    "index": [
        "es-apm-sys-sim"
    ],
    "body": {
        "size": 0,
        "query": {
            "bool": {
                "filter": {
                    "range": {
                        "@timestamp": {
                            "gte": "2020-03-12T17:57:36.229Z",
                            "lt": "2020-03-12T17:58:26.229Z",
                            "format": "strict_date_time"
                        }
                    }
                }
            }
        },
        "aggs": {
            "groupAgg": {
                "terms": {
                    "field": "host.name.keyword",
                    "size": 42
                },
                "aggs": {
                    "dateAgg": {
                        "date_range": {
                            "field": "@timestamp",
                            "ranges": [
                                {
                                    "from": "2020-03-12T17:57:36.229Z",
                                    "to": "2020-03-12T17:58:26.229Z"
                                }
                            ]
                        }
                    }
                }
            }
        }
    },
    "ignoreUnavailable": true,
    "allowNoIndices": true,
    "ignore": [
        404
    ]
}

response

{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 196,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    },
    "aggregations": {
        "groupAgg": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
                {
                    "key": "host-A",
                    "doc_count": 49,
                    "dateAgg": {
                        "buckets": [
                            {
                                "key": "2020-03-12T17:57:36.229Z-2020-03-12T17:58:26.229Z",
                                "from": 1584035856229,
                                "from_as_string": "2020-03-12T17:57:36.229Z",
                                "to": 1584035906229,
                                "to_as_string": "2020-03-12T17:58:26.229Z",
                                "doc_count": 49
                            }
                        ]
                    }
                },
                {
                    "key": "host-B",
                    "doc_count": 49,
                    "dateAgg": {
                        "buckets": [
                            {
                                "key": "2020-03-12T17:57:36.229Z-2020-03-12T17:58:26.229Z",
                                "from": 1584035856229,
                                "from_as_string": "2020-03-12T17:57:36.229Z",
                                "to": 1584035906229,
                                "to_as_string": "2020-03-12T17:58:26.229Z",
                                "doc_count": 49
                            }
                        ]
                    }
                },
                {
                    "key": "host-C",
                    "doc_count": 49,
                    "dateAgg": {
                        "buckets": [
                            {
                                "key": "2020-03-12T17:57:36.229Z-2020-03-12T17:58:26.229Z",
                                "from": 1584035856229,
                                "from_as_string": "2020-03-12T17:57:36.229Z",
                                "to": 1584035906229,
                                "to_as_string": "2020-03-12T17:58:26.229Z",
                                "doc_count": 49
                            }
                        ]
                    }
                },
                {
                    "key": "host-D",
                    "doc_count": 49,
                    "dateAgg": {
                        "buckets": [
                            {
                                "key": "2020-03-12T17:57:36.229Z-2020-03-12T17:58:26.229Z",
                                "from": 1584035856229,
                                "from_as_string": "2020-03-12T17:57:36.229Z",
                                "to": 1584035906229,
                                "to_as_string": "2020-03-12T17:58:26.229Z",
                                "doc_count": 49
                            }
                        ]
                    }
                }
            ]
        }
    }
}

request/response with avg(system.cpu.total.norm.pct) over top 42 host.name.keyword

request

{
    "index": [
        "es-apm-sys-sim"
    ],
    "body": {
        "size": 0,
        "query": {
            "bool": {
                "filter": {
                    "range": {
                        "@timestamp": {
                            "gte": "2020-03-12T18:04:00.650Z",
                            "lt": "2020-03-12T18:04:50.650Z",
                            "format": "strict_date_time"
                        }
                    }
                }
            }
        },
        "aggs": {
            "groupAgg": {
                "terms": {
                    "field": "host.name.keyword",
                    "size": 42
                },
                "aggs": {
                    "dateAgg": {
                        "date_range": {
                            "field": "@timestamp",
                            "ranges": [
                                {
                                    "from": "2020-03-12T18:04:00.650Z",
                                    "to": "2020-03-12T18:04:50.650Z"
                                }
                            ]
                        },
                        "aggs": {
                            "metricAgg": {
                                "avg": {
                                    "field": "system.cpu.total.norm.pct"
                                }
                            }
                        }
                    }
                }
            }
        }
    },
    "ignoreUnavailable": true,
    "allowNoIndices": true,
    "ignore": [
        404
    ]
}

response

{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 196,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    },
    "aggregations": {
        "groupAgg": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
                {
                    "key": "host-A",
                    "doc_count": 49,
                    "dateAgg": {
                        "buckets": [
                            {
                                "key": "2020-03-12T18:04:00.650Z-2020-03-12T18:04:50.650Z",
                                "from": 1584036240650,
                                "from_as_string": "2020-03-12T18:04:00.650Z",
                                "to": 1584036290650,
                                "to_as_string": "2020-03-12T18:04:50.650Z",
                                "doc_count": 49,
                                "metricAgg": {
                                    "value": 0.8295918362481254
                                }
                            }
                        ]
                    }
                },
                {
                    "key": "host-B",
                    "doc_count": 49,
                    "dateAgg": {
                        "buckets": [
                            {
                                "key": "2020-03-12T18:04:00.650Z-2020-03-12T18:04:50.650Z",
                                "from": 1584036240650,
                                "from_as_string": "2020-03-12T18:04:00.650Z",
                                "to": 1584036290650,
                                "to_as_string": "2020-03-12T18:04:50.650Z",
                                "doc_count": 49,
                                "metricAgg": {
                                    "value": 0.608163266765828
                                }
                            }
                        ]
                    }
                },
                {
                    "key": "host-C",
                    "doc_count": 49,
                    "dateAgg": {
                        "buckets": [
                            {
                                "key": "2020-03-12T18:04:00.650Z-2020-03-12T18:04:50.650Z",
                                "from": 1584036240650,
                                "from_as_string": "2020-03-12T18:04:00.650Z",
                                "to": 1584036290650,
                                "to_as_string": "2020-03-12T18:04:50.650Z",
                                "doc_count": 49,
                                "metricAgg": {
                                    "value": 0.44183673238267707
                                }
                            }
                        ]
                    }
                },
                {
                    "key": "host-D",
                    "doc_count": 49,
                    "dateAgg": {
                        "buckets": [
                            {
                                "key": "2020-03-12T18:04:00.650Z-2020-03-12T18:04:50.650Z",
                                "from": 1584036240650,
                                "from_as_string": "2020-03-12T18:04:00.650Z",
                                "to": 1584036290650,
                                "to_as_string": "2020-03-12T18:04:50.650Z",
                                "doc_count": 49,
                                "metricAgg": {
                                    "value": 0.11428571690102013
                                }
                            }
                        ]
                    }
                }
            ]
        }
    }
}

Here's where to instrument the code to get these kind of data dumps:

kibana/x-pack/plugins/alerting_builtins/server/alert_types/index_threshold/lib/time_series_query.ts

Lines 107 to 116 in 73d1013

	logger.debug(`${logPrefix} call: ${JSON.stringify(esQuery)}`);
	try {
	esResult = await callCluster('search', esQuery);
	} catch (err) {
	logger.warn(`${logPrefix} error: ${JSON.stringify(err.message)}`);
	throw new Error('error running search');
	}
	logger.debug(`${logPrefix} result: ${JSON.stringify(esResult)}`);

[pmuellr]

Here are some relevant text from the doc on using order with a terms aggregation:

It is also possible to order the buckets based on a "deeper" aggregation in the hierarchy. This is supported as long as the aggregations path are of a single-bucket type, where the last aggregation in the path may either be a single-bucket one or a metrics one.

In our case, the aggs path includes a multi-bucket date_range agg, so we can't reference the existing "leaf" metric agg this way. We'd need to create a new agg over the entire date range, independent of the date_range agg.

Warning: Sorting by ascending _count or by sub aggregation is discouraged as it increases the error on document counts. ... errors are unbounded. One particular case that could still be useful is sorting by min or max aggregation: counts will not be accurate but at least the top buckets will be correctly picked.

So that means the sort would work fine for min() max() and count() (the default sort criteria anyway), leaving avg() and sum() as being "error" prone.

[pmuellr]

Seems worth noting as well that we're using the same query in the alert executor AND the time series query to render the viz graph in the alert ui. It's far more important to get the former right, than the latter.

The alert executor only ends up with a single date_range bucket, so seems like we could optimize on that, or not optimize and use the whole date range. Add a new agg to get the calculated metric with a single bucket agg based on the last/only date range being requested (or whole date range), and then reference that agg in the order param.

For count(), we don't need to do anything. No new agg required, no order required. Should work as is currently coded. For min(), we'd want to sort ascending, for all the other's we'll sort descending (the default).

This doesn't take into account the comparators < <= > >= between notbetween, and you'd kind think that it might ... should these two alerts sort differently?

avg(some-field) < threshold-value
avg(some-field) > threshold-value

Maybe? like you'd want to sort ascending for the first, but descending for the second. That leads to quandries like how would you sort

avg(some-field) between (threshold-value-1, threshold-value-2)

That would involve sorting by the distance of bucket's average from a total average, or something? Seems hard-to-impossible, and beyond the scope of just getting basic ordering in, so will defer working on that part, for the first PR to address this issue.