[Actions] Treat failures as successes for Task Manager #109655

chrisronline · 2021-08-23T13:47:15Z

Relates to #55340

This PR changes how action execution failures are handled for Task Manager. Prior to this PR, if action execution threw an exception or returned an error, Task Manager would attempt to retry the task until the maxAttempts for the task definition was reached. If unable to retry, the task would be marked as Failed and stored forever as such.

Persisting these forever is problematic, especially during upgrades when migrations are ran across every saved object. The amount of failure action task documents can grow exponentially based on the rule and number of actions assigned (especially considering that some rules allow "group by" where a single rule execution can spawn many action tasks). In addition to this, action_task_params saved objects are created alongside action task saved objects and are not removed if actions fail.

In #96971, we added a task that will clean up these failed action tasks, but we should really stop persisting these if we plan to just clean them up later anyways.

This PR aims to do that by telling Task Manager that the action was successful so it will remove it.

This does solve the problem as described above, but it also introduces new challenges, mainly that the task manager health api will report all failed actions as successful now which we will need to note in our documentation.

Testing

You need to create a scenario where actions are failing and ensure that failed actions aren't persisted in TM:

POST .kibana_task_manager*/_search
{
  "size": 0,
  "aggs": {
    "NAME": {
      "terms": {
        "field": "task.status",
        "size": 100
      }
    }
  }
}

gmmorris · 2021-09-02T09:02:43Z

This does solve the problem as described above, but it also introduces new challenges, mainly that the task manager health api will report all failed actions as successful now which we will need to note in our documentation.

To preempt a question which will likely come up in review:
As mentioned here we will address this in the future by:

Rolling this change back, as the ideal behaviour is for failed actions to appear as failed tasks
Extend Task Manager to allow tasks to fail in such a manner that Task Manager cleans up the task document despite having failed, thereby avoiding the migration issue.

elasticmachine · 2021-09-02T13:14:42Z

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

chrisronline · 2021-09-02T17:59:38Z

@elasticmachine merge upstream

ymao1

LGTM! Verified that failed actions no longer leave task manager documents or action_task_params documents. Just a few comments about documentation and log messages. Nice work!

ymao1 · 2021-09-02T17:48:32Z

docs/user/production-considerations/task-manager-health-monitoring.asciidoc

@@ -123,6 +124,14 @@ The root `status` indicates the `status` of the system overall.

 The Runtime `status` indicates whether task executions have exceeded any of the <<task-manager-configuring-health-monitoring,configured health thresholds>>. An `OK` status means none of the threshold have been exceeded. A `Warning` status means that at least one warning threshold has been exceeded. An `Error` status means that at least one error threshold has been exceeded.

+[IMPORTANT]
+==============================================
+Some tasks (such as <<action-types,connectors>>) will incorrectly report their status as successful even if the task failed.


I wonder if this part should be more generic since task manager can run any type of task, and we should add a section to the Actions and Connectors docs that specifically reference the event log.

What are you thinking it should say? actions are the only known culprit of this and it does say Some tasks to make it seem like it's not an only action thing. WDYT?

I'm just wondering if it might lead people reading these docs to try to use the event log to look up failures for other tasks.

@gchaps Any suggestions for wording?

I need more context to provide wording.

Oh sorry, I missed this. I saw you added feedback. Did you still need more context?

x-pack/plugins/actions/server/lib/task_runner_factory.ts

docs/management/action-types.asciidoc

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

YulNaumenko

LGTM

gchaps

lgtm

chrisronline · 2021-09-09T14:16:41Z

@elasticmachine merge upstream

kibanamachine · 2021-09-09T16:34:53Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`actions`	122	123	+1

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`actions`	7	8	+1

Unknown metric groups

API count

id	before	after	diff
`actions`	122	123	+1

History

💚 Build #151605 succeeded bb234ea
💔 Build #151602 failed 07ce53c
💛 Build #150999 was flaky 975ad85
💚 Build #150551 succeeded a2b7506
💚 Build #150363 succeeded a89fd86

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Support retry with email as an example * Fix tests * Add logic to treat as failure if there is a retry * Handle retry better * Make this optional * Tweaks * Remove unnecessary code * Fix existing tests * Add some unit tests * Add test * Add doc note * More docs * PR feedback * Update docs/management/action-types.asciidoc Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/management/action-types.asciidoc Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/management/action-types.asciidoc Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/management/action-types.asciidoc Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/management/action-types.asciidoc Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…1769) * Support retry with email as an example * Fix tests * Add logic to treat as failure if there is a retry * Handle retry better * Make this optional * Tweaks * Remove unnecessary code * Fix existing tests * Add some unit tests * Add test * Add doc note * More docs * PR feedback * Update docs/management/action-types.asciidoc Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/management/action-types.asciidoc Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/management/action-types.asciidoc Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/management/action-types.asciidoc Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/management/action-types.asciidoc Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Support retry with email as an example

c9ddb78

chrisronline force-pushed the actions/failed_tasks branch from 676c013 to c9ddb78 Compare August 24, 2021 20:26

chrisronline added 4 commits August 25, 2021 09:31

Fix tests

366af39

Add logic to treat as failure if there is a retry

195944b

Handle retry better

ac9e9c6

Make this optional

705fd7a

chrisronline mentioned this pull request Aug 26, 2021

Failed tasks and action type execution param objects remain as saved objects forever #55340

Closed

chrisronline added 8 commits August 26, 2021 13:11

Tweaks

e9c036d

Merge remote-tracking branch 'elastic/master' into actions/failed_tasks

3150ce9

Remove unnecessary code

70df414

Merge remote-tracking branch 'elastic/master' into actions/failed_tasks

53e369f

Fix existing tests

58d8946

Add some unit tests

df80d91

Add test

b3b316a

Add doc note

a89fd86

chrisronline changed the title ~~Only throw exceptions for failed tasks if retries are supported~~ [Actions] Treat failures as successes for Task Manager Sep 1, 2021

More docs

a2b7506

chrisronline marked this pull request as ready for review September 2, 2021 13:13

chrisronline requested a review from a team as a code owner September 2, 2021 13:13

chrisronline added release_note:skip Skip the PR/issue when compiling release notes review Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v7.16.0 v8.0.0 labels Sep 2, 2021

chrisronline added the Feature:Actions/Framework Issues related to the Actions Framework label Sep 2, 2021

chrisronline linked an issue Sep 2, 2021 that may be closed by this pull request

Failed tasks and action type execution param objects remain as saved objects forever #55340

Closed

ymao1 approved these changes Sep 3, 2021

View reviewed changes

chrisronline added 2 commits September 3, 2021 09:29

Merge remote-tracking branch 'elastic/master' into actions/failed_tasks

69f1e3c

PR feedback

975ad85

chrisronline force-pushed the actions/failed_tasks branch from 0e57e76 to 975ad85 Compare September 3, 2021 15:12

gchaps reviewed Sep 3, 2021

View reviewed changes

chrisronline and others added 6 commits September 7, 2021 13:50

Update docs/management/action-types.asciidoc

ac96c26

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

Update docs/management/action-types.asciidoc

07ce53c

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

Update docs/management/action-types.asciidoc

5ddf97c

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

Update docs/management/action-types.asciidoc

1b9eb7f

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

Update docs/management/action-types.asciidoc

1ce8251

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

Merge remote-tracking branch 'elastic/master' into actions/failed_tasks

bb234ea

chrisronline requested review from ymao1 and gchaps September 7, 2021 18:01

ymao1 approved these changes Sep 8, 2021

View reviewed changes

YulNaumenko approved these changes Sep 8, 2021

View reviewed changes

gchaps approved these changes Sep 9, 2021

View reviewed changes

Merge branch 'master' into actions/failed_tasks

1ad1073

chrisronline merged commit b9e6f93 into elastic:master Sep 9, 2021

chrisronline deleted the actions/failed_tasks branch September 9, 2021 16:51

chrisronline mentioned this pull request Sep 9, 2021

[7.x] [Actions] Treat failures as successes for Task Manager (#109655) #111769

Merged

chrisronline mentioned this pull request Sep 29, 2021

[Actions] Better enqueue test #112434

Merged

pmuellr mentioned this pull request Oct 26, 2021

[alerting] rule tasks that fail 3 times are never run again, with no indication in the rule #116321

Closed

mikecote mentioned this pull request Feb 16, 2023

Failed Task Manager task documents are never cleaned up bloating the index #79977

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Actions] Treat failures as successes for Task Manager #109655

[Actions] Treat failures as successes for Task Manager #109655

chrisronline commented Aug 23, 2021 •

edited

Loading

gmmorris commented Sep 2, 2021

elasticmachine commented Sep 2, 2021

chrisronline commented Sep 2, 2021

ymao1 left a comment

ymao1 Sep 2, 2021

chrisronline Sep 3, 2021

chrisronline Sep 3, 2021

ymao1 Sep 3, 2021

gchaps Sep 3, 2021 •

edited

Loading

chrisronline Sep 8, 2021

YulNaumenko left a comment

gchaps left a comment

chrisronline commented Sep 9, 2021

kibanamachine commented Sep 9, 2021

API count

[Actions] Treat failures as successes for Task Manager #109655

[Actions] Treat failures as successes for Task Manager #109655

Conversation

chrisronline commented Aug 23, 2021 • edited Loading

Testing

gmmorris commented Sep 2, 2021

elasticmachine commented Sep 2, 2021

chrisronline commented Sep 2, 2021

ymao1 left a comment

Choose a reason for hiding this comment

ymao1 Sep 2, 2021

Choose a reason for hiding this comment

chrisronline Sep 3, 2021

Choose a reason for hiding this comment

chrisronline Sep 3, 2021

Choose a reason for hiding this comment

ymao1 Sep 3, 2021

Choose a reason for hiding this comment

gchaps Sep 3, 2021 • edited Loading

Choose a reason for hiding this comment

chrisronline Sep 8, 2021

Choose a reason for hiding this comment

YulNaumenko left a comment

Choose a reason for hiding this comment

gchaps left a comment

Choose a reason for hiding this comment

chrisronline commented Sep 9, 2021

kibanamachine commented Sep 9, 2021

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

API count

History

chrisronline commented Aug 23, 2021 •

edited

Loading

gchaps Sep 3, 2021 •

edited

Loading