When using the Index Threshold rule type, add ability to check the count of buckets

[mcpate]

Describe the feature:
I'd like to use the Index Threshold rule type to bucket on a particular field, then I'd like to check that the count of buckets matches a particular number.

Describe a specific use case for the feature:
We have several Watches that we'd like to move over to Kibana Alerting. These Watches query an index with a set of filters, then they aggregate those results, then the condition within the Watch verifies that the count of buckets matches a particular number.

For a concrete example: We have a service that sends data to 3 cloud providers - AWS, GCP, Azure. When it sends data it logs some info about the data it is sending. This info includes a field called cloud.provider. We'd like to make sure that, when looking at the last few hours of logs, we have logs indicating that we are sending data to all 3 cloud providers. We do this by bucketing by cloud.provider and then we make sure that we have 3 buckets.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

Wondering how we'd structure this in terms of the params. The easiest thing would be to add a new "AggType":

kibana/x-pack/plugins/triggers_actions_ui/server/data/lib/core_query_types.ts

Line 93 in 5a97b47

const AggTypes = new Set(['count', 'avg', 'min', 'max', 'sum']);

Maybe call it, count-buckets? We already do some special processing for count now, so it seems like this might be the best place for it. It is a little weird, in that it makes the "AggType" even less similar to agg-y things in ES, but as mentioned count is already a little weird.

It would like count in that we wouldn't actually prompt for an additional field to do the agg over, since that's not needed.

Alternatively, would there be something more direct involving cardinality? So, add a cardinality agg with the existing aggs, then for this use case you wouldn't group over, just use that agg with the field you want. That sort of feels better, and would allow you to use it WITH grouping as well, if needed (presumably cardinality of a different field you're grouping over).

[pmuellr]

These Watches query an index with a set of filters

The index threshold rule type currently doesn't support any filters, though there are issues open to add this.

Which makes me wonder if there's another rule type - presumably in o11y - that might already do this out of the box. It looks like maybe the Metrics Threshold could? I think the potential problem with this is that the o11y rules typically configure the indices searched via Kibana config (probably per-space), and so it could be clumsy to set this up for some unique indices, and then could stop working if someone resets the config.

[mcpate]

All great info @pmuellr. Note that I am also currently looking at how I might be able to use Transforms to structure the data in a way that might allow using the Elasticsearch Query rule type. (In which case maybe what is needed is an update to the documentation on how to do this.)

[pmuellr]

Transforms seem like a good approach to making data from an index easier to handle from the alerting side. I think doc-wise, we'd just suggest this is a possibility, since presumably the work entails creating the transform, then building the rule to operate against the new index created by the transform. Are you thinking some per-ruleType help, on how to structure such transforms for use with a particular rule type?

[mcpate]

I did some research last week and it does look like Transforms could provide a mechanism to get the number of buckets for a particular field. (I used a "latest" transform and specified the field I wanted bucketed as a "unique key". The Transform seemed to keep the latest document for each unique key. I was then able to create a rule that looked at the Transform index going back several hours, and verified that the number of documents was X.)

I think doc-wise, we'd just suggest this is a possibility, since presumably the work entails creating the transform, then building the rule to operate against the new index created by the transform.

This makes sense and could be useful for folks versed in what is possible in Watcher, but who might not know all the possibilities Elasticsearch offers.

Are you thinking some per-ruleType help, on how to structure such transforms for use with a particular rule type?

I'm not exactly sure? I'm not sure if per-ruleType is needed (that might be too detailed). For my particular case, I just needed to know that Transforms existed and that they could be used to manipulate the data to a form better suited to my rule. It might be that the best place for a blurb about Transforms is in the section that talks about Watcher vs. Kibana Alerting? (Since, afaik, we don't have a migration guide.)

[pmuellr]

Ya, I'm on-board with some section in the docs talking about other things you can do to make your existing data fit rules - transforms being one tool, another example would be using a filtered alias if the rule type doesn't otherwise have filters, etc.

A cardinality agg seems simple enough to add - which makes me think we should split this into two issues; one for that, another for extending the docs.

[pmuellr]

cc: @shanisagiv1