[Security Solution] Threshold rule performance fixes

[madirey]

There are 2 known issues which can impact performance of a threshold rule over large indices. I've outlined some steps below that should be addressed ASAP:

• Limit fields in Threshold top_hits aggregation to only include necessary fields (or none at all if we can get away with it)
• Ensure that the user does not try to count the cardinality of a field that's included in the Threshold terms agg in rule configuration... this will always result in a cardinality of 1 which is not useful, and thus wasteful
• Initialize shard_min_doc_count appropriately (I think the default value of 0 may be appropriate here... will review further)
• Performance testing to find allowed number of "group by" fields: [Security Solution][Detections] Allow selection of more than three 'Group by' fields #122490
• Investigate use of multi_terms and composite instead of nested terms aggs ([Security Solution][Alerts] Explore using multi_terms or composite aggs for threshold rules #125703)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[MadameSheema]

@peluja1012 @banderror can you please add this bug to a project (if needed), add an impact and remove the triage label? Thanks!

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[madirey]

Performance of 3, 4, 5, and 10 threshold aggregation fields:

[madirey]

From the above, the only thing that is really affected by increasing the max number of aggregation fields is the average query time. The current max (3) results in a maximum query time of 4.5 seconds. This increases to 6 seconds when 4 fields are used, and goes up to about 8 seconds for 5 and 10 fields. Index time is not affected.

Tests were run using a modified version of kbn-alert-load against stack version 8.1.2, using 50 threshold rules running every minute over approximately 600,000 total events. Metrics were collected for 5 minutes.

[madirey]

The query time increases almost linearly for lower values of threshold fields, but this affect appears to attenuate over time, as the query time for both 5 and 10 threshold fields is nearly identical. This shows that there will be a cost associated with increasing the number of allowed threshold fields, but if we DO wish to increase it, perhaps we shouldn't limit the value at all.

[marshallmain]

@madirey is it possible to get numbers for a multi_terms based threshold implementation for comparison?

[madirey]

multi_terms is consistently about 10x slower than a nested terms agg. The documentation notes that multi_terms will be slower than nested terms aggregations except in a few specific cases. We can expect that our search results may contain many results, so it's expected that multi_terms with its execution hint of map will not perform as well.

[madirey]

composite aggs are consistently slower than our nested terms aggs by about 3x. I'm trying an optimization that was recommended by the ES team to see if that helps. multi_terms likely won't fit our use case... it's great for getting the top N buckets, but we'd like to get ALL buckets.

That said, our current implementation (nested terms) is limited in that we can only get 10k (max is 65k, so we could bump this up) top-level buckets. Which means we could get significantly less than that after the cardinality of the child buckets are multiplied in. We mitigate this by sorting the buckets by cardinality (desc), but this could still result in false negatives. The advantage of composite is that we could page over all the results and we won't miss anything. (But again, how useful is it going to be for an operator to have to sort through >10k alerts...?)

[madirey]

The optimization mentioned above actually increased the rule runtime by about 3x. So far, it looks like nested terms aggs are by far the best option. Conclusion: we could consider increasing the number of threshold fields allowed at the expense of increasing potential heap size of the ES cluster.

[madirey]

@marshallmain @spong ^^