Include alert mute state and reason when persisting triggered alerts

[rhr323]

Describe the feature:

For reporting purposes, it would be highly beneficial to have the capability to filter out muted alerts—those which did not trigger any actions.

The current data structure within the .alerts-stack.alerts-default index lacks information about whether an alert was in a muted state at the time it was triggered. Therefore, a feature enhancement is requested to persist information regarding the alert's state upon triggering.

Additionally, introducing fields that detail the reasons why an alert did not run its actions, such as being in a muted state for various reasons, would also be helpful.

Describe a specific use case for the feature:

The primary use case for this request stems from the need for more precise and effective reporting capabilities with regards to alerting. Being able to distinguish between alerts that were actively suppressed and those that triggered actions allows for a more accurate assessment of alert "noise".

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

I guess one of the odd things about this will be if the "muted" state / reason changes during the lifetime of an alert. Do we update it each time? Just record the first one? Presumably we will have a list of changes here from the alerting "audit trail" once that's going.

[pmuellr]

Thinking about it a little more ...

My read from the OC use case is as a a kind of meta-analysis: 'Being able to distinguish between alerts that were actively suppressed and those that triggered actions allows for a more accurate assessment of alert "noise".'

For that purpose, if you're trying to tune your rules/actions/conditions, it would be good to record the "mute state/reason" on alert creation - how it changes over time may be of less importance. Guessing this will simplify adding the fields here, as we wouldn't have to update the alert doc ...

Though I can also read that as a "point in time" kind of analysis, which would point to keeping it updated.

If it's not too much work, perhaps we can record both - the initial "mute state/reason", and the "current mute state/reason".

[pmuellr]

Initial requester for this indicated they were more interested in the initial muting state, as opposed to tracking the current muting state, in the alert docs.

[ymao1]

cc @shanisagiv1