Deleting a Kibana space containing running alerting rules does not delete the rule's task Saved Objects

[pmuellr]

Currently running alerting rules have two Saved Objects (SOs) associated with them - a rule and task SO:

• the rule SO is space specific, and contains the "business logic" of the alerting rule
• the task SO is space agnostic, and is used by task manager to coordinate running the alerting rule

When a user deletes a Kibana space with running alerting rules, the rule SO's will be deleted, but the task SO's will not. The alerting framework will end up trying to run the task eventually (based on the task SO), and fail indicating that it can't find the rule SO, after which it marks the task as failed will not be tried again.

This is causing "expected" failures that we would like to avoid, as it's noisy log messages to the customers. It is currently affecting our "health status" (we move into bad health states with lots of failures), and so causing some false alarms regarding Kibana status. And it's a performance hit running tasks that are guaranteed to fail, and consume a task slot while doing so.

Ideally, when deleting an alerting rule in any situation, the delete should be done via the alerting client and not the saved object client (or saved object plugin itself, at a lower level). Somehow we need to get the "delete space" API to work in this mode.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)