[Metrics UI] Alert preview produces confusing results

[Zacqary]

UPDATE: This is an R&D ticket to document why the alert preview was returning inaccurate and/or confusing results before we removed it in 7.15, in this ticket: #107946

AC:

• Document the problems with the "Test Alert Condition" preview functionality and what is causing these problems
• File a new ticket to fix the problems and/or create an RFC with @elastic/kibana-alerting-services

ORIGINAL TICKET DESCRIPTION BELOW:

From #90070 (review)

I have 10 hosts reporting an average around 12-20%, then I have one host that is pushing 100% (which averages out to about 50% (ish). When I set the threshold to 50% I would expect the preview to say something like

There were 60 instances that satisfied the conditions of this alert across 1 kubernetes.pod.uids in the last hour.

but instead it says

There were 60 instances that satisfied the conditions of this alert across 10 kubernetes.pod.uids in the last hour.

It feels like we are counting too many UUIDs for the match. When the alert runs, it only matches on one UUID.

This isn't so much a bug as a UX issue. The alert preview is designed to tell the user that it's looking at 10 pod UIDs total, but it doesn't tell you how many of those alerted.

We should try to achieve something like:

There were 60 instances that satisfied the conditions of this alert on 1 out of 10 kubernetes.pod.uids in the last hour.

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[jasonrhodes]

In 7.15 we are going to remove this feature from the alert flyout until we understand it better. I'd love for @Zacqary and @simianhacker to sync up when you are both around and figure out what exactly is going wrong with it, so we have it documented how it would be fixed. We can then decide if it's a good idea to re-enable it here or to try to submit this as an RFC to the alerting framework.

[smith]

Not planning on implementing this.