[ResponseOps][BE] Alert creation delay based on user definition #174657

doakalexi · 2024-01-11T00:09:41Z

Related to #173009

Summary

This is the first of two PRs and only focuses on the backend implementation. This PR adds a new notificationDelay field to the Rule object. With the delay the rule will run X times and has to match the threshold X times before triggering actions. It won't affect the alert recovery, but it can be expanded on easily if we want to include recovered alerts in the future.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

To verify

Use Dev Tools to create a rule with the notificationDelay

POST kbn:/api/alerting/rule
{
  "params": {
    "searchType": "esQuery",
    "timeWindowSize": 5,
    "timeWindowUnit": "m",
    "threshold": [
      -1
    ],
    "thresholdComparator": ">",
    "size": 100,
    "esQuery": """{
    "query":{
      "match_all" : {}
    }
  }""",
    "aggType": "count",
    "groupBy": "all",
    "termSize": 5,
    "excludeHitsFromPreviousRun": false,
    "sourceFields": [],
    "index": [
      ".kibana-event-log*"
    ],
    "timeField": "@timestamp"
  },
  "consumer": "stackAlerts",
  "schedule": {
    "interval": "1m"
  },
  "tags": [],
  "name": "test",
  "rule_type_id": ".es-query",
  "actions": [
    {
      "group": "query matched",
      "id": "${ACTION_ID}",
      "params": {
        "level": "info",
        "message": """Elasticsearch query rule '{{rule.name}}' is active:

- Value: {{context.value}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}"""
      },
      "frequency": {
        "notify_when": "onActionGroupChange",
        "throttle": null,
        "summary": false
      }
    }
  ],
  "notification_delay": {
    "active": 3
  }
}

Verify that the rule will not trigger actions until it has matched the delay threshold. It might be helpful to look at rule details page and add the Triggered actions column to easily see the action was triggered after X consecutive active alerts

- Verify that the delay does not affect recovered alerts

doakalexi · 2024-01-11T00:23:08Z

/ci

doakalexi · 2024-01-12T18:26:54Z

/ci

doakalexi · 2024-01-12T19:14:25Z

/ci

doakalexi · 2024-01-12T19:15:02Z

/ci

…bana into alerting/notification-delay

doakalexi · 2024-01-12T21:30:09Z

/ci

doakalexi · 2024-01-16T18:25:31Z

/ci

elasticmachine · 2024-01-16T19:40:47Z

Pinging @elastic/response-ops (Team:ResponseOps)

ymao1 · 2024-01-22T16:47:23Z

Should the update API support the notification delay parameter as well?

doakalexi · 2024-01-22T16:49:52Z

Should the update API support the notification delay parameter as well?

I added that code in the ui PR, I can add it here instead if you think that is better

ymao1 · 2024-01-22T16:53:06Z

When I migrate an existing rule to this branch, and let it run with an active alert, I see the activeCount get set to 0 the first run, and then it stays zero in subsequent runs (in each run, the alert is active). Should the activeCount be getting incremented? The rule has no notification delay setting but that doesn't seem like it should impact the activeCount parameter?

ymao1 · 2024-01-22T16:53:41Z

I added that code in the ui PR, I can add it here instead if you think that is better

I think that's fine...I was just trying to update an existing rule with a notification delay parameter :)

doakalexi · 2024-01-22T16:57:09Z

I added that code in the ui PR, I can add it here instead if you think that is better

I think that's fine...I was just trying to update an existing rule with a notification delay parameter :)

ohh okay, I can add that code here if needed!

doakalexi · 2024-01-22T17:00:55Z

When I migrate an existing rule to this branch, and let it run with an active alert, I see the activeCount get set to 0 the first run, and then it stays zero in subsequent runs (in each run, the alert is active). Should the activeCount be getting incremented? The rule has no notification delay setting but that doesn't seem like it should impact the activeCount parameter?

If there is not a notificationDelay field it will be reset in the execution handler: https://github.com/elastic/kibana/pull/174657/files#diff-8656f6dc8acf27d0e712d9c277de92a693b6e2ee3bcdae9f08f8395141f7b38cR641
I can change this code so it's incrementing, I had decided that if the user did not want to delay the alert then we didn't need to track it.

ymao1 · 2024-01-22T17:02:39Z

If there is not a notificationDelay field it will be reset in the execution handler:
Gotcha 👍

kibana-ci · 2024-01-22T18:09:49Z

💚 Build Succeeded

Buildkite Build
Commit: 61ae6f6

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`alerting`	799	803	+4

Unknown metric groups

API count

id	before	after	diff
`alerting`	830	834	+4

References to deprecated APIs

id	before	after	diff
`alerting`	114	116	+2

History

💚 Build #187759 succeeded 5809967
💛 Build #187746 was flaky f655c35
💛 Build #187325 was flaky 394c10d
💔 Build #187302 failed 5ba8271
💔 Build #187294 failed 275520b
💔 Build #186912 failed 1b69332

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

ersin-erdal

LGTM

Tested locally, works as expected.
Just not sure if the notification should be scheduled on the third or fourth active alert when the notificationDelay is 3.

alert.getActiveCount() < this.rule.notificationDelay.active
or
alert.getActiveCount() <= this.rule.notificationDelay.active

other than this, good job :)

doakalexi · 2024-01-23T17:16:45Z

Tested locally, works as expected. Just not sure if the notification should be scheduled on the third or fourth active alert when the notificationDelay is 3.

alert.getActiveCount() < this.rule.notificationDelay.active or alert.getActiveCount() <= this.rule.notificationDelay.active

I am not 100% sure which is correct, but I implemented this similar to how we are doing the pending recovered count for flapping.
if (alert.getPendingRecoveredCount() < flappingSettings.statusChangeThreshold) {

@timestamp

…tic#174657) Related to elastic#173009 ## Summary This is the first of two PRs and only focuses on the backend implementation. This PR adds a new `notificationDelay` field to the `Rule` object. With the delay the rule will run X times and has to match the threshold X times before triggering actions. It won't affect the alert recovery, but it can be expanded on easily if we want to include recovered alerts in the future. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `notificationDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "notification_delay": { "active": 3 } } ``` - Verify that the rule will not trigger actions until it has matched the delay threshold. It might be helpful to look at rule details page and add the Triggered actions column to easily see the action was triggered after X consecutive active alerts <img width="1420" alt="Screen Shot 2024-01-16 at 1 18 52 PM" src="https://github.com/elastic/kibana/assets/109488926/85d8ceef-042c-4a52-950e-24492dc0e79f"> - Verify that the delay does not affect recovered alerts

@timestamp

…tic#174657) Related to elastic#173009 ## Summary This is the first of two PRs and only focuses on the backend implementation. This PR adds a new `notificationDelay` field to the `Rule` object. With the delay the rule will run X times and has to match the threshold X times before triggering actions. It won't affect the alert recovery, but it can be expanded on easily if we want to include recovered alerts in the future. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `notificationDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "notification_delay": { "active": 3 } } ``` - Verify that the rule will not trigger actions until it has matched the delay threshold. It might be helpful to look at rule details page and add the Triggered actions column to easily see the action was triggered after X consecutive active alerts <img width="1420" alt="Screen Shot 2024-01-16 at 1 18 52 PM" src="https://github.com/elastic/kibana/assets/109488926/85d8ceef-042c-4a52-950e-24492dc0e79f"> - Verify that the delay does not affect recovered alerts

@timestamp

…tic#174657) Related to elastic#173009 ## Summary This is the first of two PRs and only focuses on the backend implementation. This PR adds a new `notificationDelay` field to the `Rule` object. With the delay the rule will run X times and has to match the threshold X times before triggering actions. It won't affect the alert recovery, but it can be expanded on easily if we want to include recovered alerts in the future. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `notificationDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "notification_delay": { "active": 3 } } ``` - Verify that the rule will not trigger actions until it has matched the delay threshold. It might be helpful to look at rule details page and add the Triggered actions column to easily see the action was triggered after X consecutive active alerts <img width="1420" alt="Screen Shot 2024-01-16 at 1 18 52 PM" src="https://github.com/elastic/kibana/assets/109488926/85d8ceef-042c-4a52-950e-24492dc0e79f"> - Verify that the delay does not affect recovered alerts

Adding the notification delay

1b69332

doakalexi added 2 commits January 12, 2024 10:22

Adding AAD test

89cc726

Removing changes to rule registry

275520b

Change activeCount to active

2931666

Merge branch 'main' into alerting/notification-delay

5ba8271

doakalexi added 2 commits January 12, 2024 13:28

Fixing tests

4f160be

Merge branch 'alerting/notification-delay' of github.com:doakalexi/ki…

394c10d

…bana into alerting/notification-delay

Adding jest tests

f655c35

doakalexi added Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.13.0 release_note:skip Skip the PR/issue when compiling release notes labels Jan 16, 2024

doakalexi changed the title ~~Adding the notification delay~~ [ResponseOps][BE] Alert creation delay based on user definition Jan 16, 2024

doakalexi marked this pull request as ready for review January 16, 2024 19:40

doakalexi requested a review from a team as a code owner January 16, 2024 19:40

Merge branch 'main' into alerting/notification-delay

5809967

Merge branch 'main' into alerting/notification-delay

61ae6f6

ersin-erdal approved these changes Jan 23, 2024

View reviewed changes

doakalexi merged commit 80640cf into elastic:main Jan 23, 2024
33 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Jan 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ResponseOps][BE] Alert creation delay based on user definition #174657

[ResponseOps][BE] Alert creation delay based on user definition #174657

doakalexi commented Jan 11, 2024 •

edited

Loading

doakalexi commented Jan 11, 2024

doakalexi commented Jan 12, 2024

doakalexi commented Jan 12, 2024

doakalexi commented Jan 12, 2024

doakalexi commented Jan 12, 2024

doakalexi commented Jan 16, 2024

elasticmachine commented Jan 16, 2024

ymao1 commented Jan 22, 2024

doakalexi commented Jan 22, 2024

ymao1 commented Jan 22, 2024

ymao1 commented Jan 22, 2024

doakalexi commented Jan 22, 2024

doakalexi commented Jan 22, 2024

ymao1 commented Jan 22, 2024

kibana-ci commented Jan 22, 2024

API count

References to deprecated APIs

ersin-erdal left a comment

doakalexi commented Jan 23, 2024

[ResponseOps][BE] Alert creation delay based on user definition #174657

[ResponseOps][BE] Alert creation delay based on user definition #174657

Conversation

doakalexi commented Jan 11, 2024 • edited Loading

Summary

Checklist

To verify

doakalexi commented Jan 11, 2024

doakalexi commented Jan 12, 2024

doakalexi commented Jan 12, 2024

doakalexi commented Jan 12, 2024

doakalexi commented Jan 12, 2024

doakalexi commented Jan 16, 2024

elasticmachine commented Jan 16, 2024

ymao1 commented Jan 22, 2024

doakalexi commented Jan 22, 2024

ymao1 commented Jan 22, 2024

ymao1 commented Jan 22, 2024

doakalexi commented Jan 22, 2024

doakalexi commented Jan 22, 2024

ymao1 commented Jan 22, 2024

kibana-ci commented Jan 22, 2024

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

References to deprecated APIs

History

ersin-erdal left a comment

Choose a reason for hiding this comment

doakalexi commented Jan 23, 2024

doakalexi commented Jan 11, 2024 •

edited

Loading