[Spike] [Log threshold rule] Explore options to fix "too many buckets" error in alert details page

[benakansara]

Explore the options to fix alerts history chart "too many buckets" error described in #173020 and come up with a suitable solution.

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[benakansara]

I think the issue will be resolved once we filter the alerts history chart by group fields in #176718. I checked the same query that results in too_many_buckets_exception with filters applied. With filters, it didn't throw the exception.

We still need to do stress testing to see if there are many alerts for a single group field (for example, host-0 has 1000+ alerts), does it work or not. Based on that we can evaluate if #176718 would be enough to address the underlying issue or do we need to change the implementation of alerts history chart.

[jasonrhodes]

I think you're right that #176718 will fix the majority of cases where this error could occur. Can we use this issue to figure out how we avoid that "Oops" error even if a rule creates too many buckets?

At the very least, let's log a telemetry event any time a user encounters this error.

[jasonrhodes]

@elastic/obs-bi-team is there a currently "approved" way for us to log some kind of telemetry event in a certain error scenario, so we can track how many times that scenario appears for customers? I think that would be a sufficient outcome for this ticket at this point (do you agree, @benakansara?)

[almudenasanz]

@jasonrhodes you could use EBT with a custom context
https://docs.elastic.dev/telemetry/collection/event-based-telemetry#how-does-an-event-look-like

[benakansara]

@jasonrhodes Since we are replacing current history chart with AlertSummaryWidget kind of chart, this issue will become obsolete. The new chart will be common for all alert detail pages, so it will automatically appear on Log threshold alert details page too, and we can remove current history chart. Once that is implemented, we can close this ticket. Wdyt?

[jasonrhodes]

@benakansara yes, I agree, this was another reason I was excited by that other direction.

Closed in favor of #181475