[Discuss][Actions] Should actions remain "at most once"?

[gmmorris]

When the actions plugin was introduced we chose to go with an "at most once" approach, meaning we wanted to ensure actions don't accidentally fire more than once.

We achieved this by setting action tasks to maxAttempts to 1, which means actions are never retried.
The thinking at the time was that running an action twice was worse than not running it at all, as we had a history of failed tasks in the .kibana_task_manager index and we had concrete plans of building an Event Log UI before GA.

Best laid plans of 🐭 and 👷‍♀️ and all , we're long past GA without an Event Log UI and following this PR these failed tasks are cleaned up every 5m minutes.

This means that even though actions are still "at most once", we no longer have the mitigation of users being able to investigate failed tasks. Failed actions are still logged in the Event Log, so it's not that there is no record, but we have no concrete plans for an Event Log UI and that tasks themselves are deleted so it isn't easy to reproduce the failure case.

This feels like a risk to me, and I think we should either reconsider the "at most once" guidance, or we should consider some kind of record of the failed tasks that can't cause migration failures further down the line.

Discuss :)

cc @elastic/kibana-alerting-services @stacey-gammon @kobelb

[mikecote]

cc @arisonl for product awareness on what we should do from a product perspective.

[kobelb]

I'm largely putting on a product hat here, but I think we should allow our users to choose between "at most once" and "at least once" delivery for actions on a per-action basis. Depending on the situation, I anticipate users wanting the ability to customize how this works for each different alerting rule.

[gmmorris]

This relates to the long running actions issue as well: #113424

[mikecote]

Closing in favour of #143046