Retry policy for an action in mistral workflow can lead to orphaned action execution on failure

[m4dcoder]

Say there is a st2 retry policy on an action and the action is being invoked in a mistral workflow. If the action fails, the mistral workflow will fail but st2 will still retry the execution of the action.

[m4dcoder]

Initial research on the fix here is to move the policy evaluation from the notifier (https://github.com/StackStorm/st2/blob/master/st2actions/st2actions/notifier/notifier.py#L256) to post_run of the runner. This allows for the appropriate action runner to handle callback and post run sequences appropriately. Currently, for action execution of mistral tasks, the post_run which callback mistral to post execution result and policy application are done separately. The goal is to not post action completion in the mistral callback until the st2 retry policy is applied/exhausted. The mistral callback still needs to update the context of its counterpart in mistral with the details of the new action execution being re-run.

[Kami]

I've discussed this with @m4dcoder on Slack and here is a recap (of my thoughts on the approach and what we decided to do).

I think it doesn't make much sense to move stuff into post_run and implement it for Mistral workflows. Moving it into post_run breaks the existing abstraction and model for applying policies (current model is based on the notifier and it's easy to follow and understand). At the same time, similar thing can be achieved for Mistral workflows using primitives available in Mistral directly inside Mistral workflows.

So doing that we would basically do one off in the code for Mistral which is, imo not worth it. It breaks the existing model, increases complexity, harder to understand and maintain, etc.

Having said that, I do think we should at least update the existing policy code and only allow it to be executed on simple (non workflow and action-chain) actions. This way we avoid the Mistral orphan issues.

We should do this on two levels:

1. At policy register / load time.
2. At run-time - sadly because action definitions, etc. can change, we still need to do the same check during run-time of the policy application as well.

[m4dcoder]

Fix at #3140