[RAM] Update rule status

[XavierM]

Summary

Resolves the parent issue: #136039

Also resolves the subtasks:

• [RAM] Update rules client APIs to support new consolidated statuses #143461
• [RAM] Update task runner to support new consolidated statuses  #143460

This is the backend portion of the consolidated rule status feature. It mainly contains changes to the rules_client.ts and task_runner.ts to support the new consolidated rule statuses.

This PR added a new property: lastRun to the rules saved object to hold the new rule outcome statuses (succeeded, warning, and failed) as the new simplified rule status over the existing executionStatus property. However, we are keeping the old executionStatus so we can slowly migrate the rest of the application to use the new lastRun outcomes.

In addition, we have enriched the monitoring property to be the source of truth for metrics related to the last run (as well as new fields that other plugins will find useful). We also added a monitoring service that allows other plugins to easily add data to the monitoring field.

To test this PR, please use #144466 since it has both the frontend and backend changes.

Checklist

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[XavierM]

let's delete running, we are not using it and too scare to make another call to SO. Let's wait and see

[mdefazio]

This was going to be my question actually...if I create a rule, what is the status prior to the first run? Or if I disable it before its run?

[XavierM]

we were thinking that we can tell an approximation time when the rule is going to run. if you disable it before its run, we won't show anything, do you have a better idea?

[mdefazio]

After chatting, we opted to simply show -- if the rule has never run. My initial concern was if I create a rule and see nothing in this column, I would wonder if the rule is starting or not. So for this scenario, we'll also show the -- but do a similar treatment as we do for Stat loading indicator. We felt this was the simplest solution at this time and still indicated to the user that something is happening with the rule after I create it.

[ymao1]

Did initial code review only of the task runner. Still need to look at the rules client and migration and pull it down to run it. Looking good tho!

[ymao1]

When a rule is created, before running, it looks like monitoring is set to this:

"monitoring": {
  "run": {
    "history": [],
    "calculated_metrics": {
      "success_ratio": 0
    },
    "last_run": {
      "timestamp": "2022-11-03T13:06:31.486Z",
      "metrics": {
      "duration": 0,
        "total_search_duration_ms": null,
        "total_indexing_duration_ms": null,
        "total_alerts_detected": null,
        "total_alerts_created": null,
        "gap_duration_s": null
      }
    }
  }
}

Should those initial duration be null since we don't know it? Also is setting last_run.timestamp to the current date confusing since it hasn't run yet?

I'm thinking of a case where a rule is created in an overloaded Kibana cluster, so it might take longer than expected for the task to get claimed and run. In the UI, this last_run.timestamp would show up and give the impression that the rule has already run? Maybe I'm overthinking :)

[JiaweiWu]

That is a good point, having 0 for the duration is definitely misleading

As for the timestamp, we're essentially mirroring existing executionStatus -> lastExecutionDate field with the lastRun -> timestamp field. So I guess setting the timestamp during create was existing behavior but we could fix it for both of the existing behavior if it doesn't make sense.

[XavierM]

That make sense!

[ymao1]

After 1 execution, my rule saved object looks like:

{
	... other fields
	"executionStatus": {
		"status": "active",
		"lastExecutionDate": "2022-11-03T13:09:15.575Z",
		"error": null,
		"warning": null,
		"lastDuration": 7270
	},
	"monitoring": {
		"run": {
			"history": [{
				"duration": 7270,
				"success": true,
				"timestamp": 1667480955574
			}],
			"calculated_metrics": {
				"success_ratio": 1,
				"p99": 7270,
				"p50": 7270,
				"p95": 7270
			},
			"last_run": {
				"timestamp": "2022-11-03T13:09:15.574Z",
				"metrics": {
					"duration": 0,
					"total_search_duration_ms": null,
					"total_indexing_duration_ms": null,
					"total_alerts_detected": null,
					"total_alerts_created": null,
					"gap_duration_s": null
				}
			}
		}
	},
	"lastRun": {
		"alertsCount": {
			"new": 5,
			"ignored": 0,
			"recovered": 0,
			"active": 5
		},
		"outcomeMsg": null,
		"warning": null,
		"outcome": "succeeded"
	},
}

The monitoring.run.last_run.duration field doesn't look like it's been updated to the last duration.

[ymao1]

I'm sure this was covered in the RFC and I'm behind, but why a separate warning and outcomeMsg field? The outcome can be succeeded | failed | warning so I would expect any warnings to show up as outcome: 'warning', outcomeMsg: <warning message>?

[JiaweiWu]

Ah yes good catch WRT the duration, will fix that.

From the RFC, the warning field is described to act sort of like a decorator to the outcome. My understanding is the outcome could be successful but there might have been a special flag raised in the rule run that should be noted. So in this case the warning field gets mapped to reasons. @XavierM could correct me if I'm wrong here.

[XavierM]

++

[JiaweiWu]

@elasticmachine merge upstream

[ymao1]

LGTM!

[JiaweiWu]

@elasticmachine merge upstream

[JiaweiWu]

@elasticmachine merge upstream

[JiaweiWu]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 367b3d8

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
alerting	379	406	+27

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
alerting	26	27	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
alerting	38.7KB	39.2KB	+443.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id	before	after	diff
alert	74	95	+21

Unknown metric groups

API count

id	before	after	diff
alerting	388	415	+27

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
alerting	68	70	+2
enterpriseSearch	19	21	+2
fleet	59	65	+6
osquery	108	113	+5
securitySolution	441	447	+6
total			+21

References to deprecated APIs

id	before	after	diff
alerting	84	96	+12

Total ESLint disabled count

id	before	after	diff
alerting	69	71	+2
enterpriseSearch	20	22	+2
fleet	67	73	+6
osquery	109	115	+6
securitySolution	518	524	+6
total			+22

History

• 💔 Build #87656 failed 49d07a1
• 💔 Build #87623 failed c8e1da5
• 💔 Build #87562 failed 1869e17
• 💛 Build #87357 was flaky bbef4eb
• 💔 Build #86893 failed 897ec3f
• 💔 Build #86772 failed f0b1c66

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream