[ResponseOps] Alert creation delay based on user definition #175851

doakalexi · 2024-01-29T23:10:18Z

Resolves #173009

Summary

This PR:

Changes the field name from notification_delay to alert_delay
Updates the alerts client and rule registry to index new alert docs on a delay
Updates the framework code to delay the creation of an alert

Checklist

Unit or functional tests were updated or added to match the most common scenarios

To verify

Use Dev Tools to create a rule with the alertDelay

POST kbn:/api/alerting/rule
{
  "params": {
    "searchType": "esQuery",
    "timeWindowSize": 5,
    "timeWindowUnit": "m",
    "threshold": [
      -1
    ],
    "thresholdComparator": ">",
    "size": 100,
    "esQuery": """{
    "query":{
      "match_all" : {}
    }
  }""",
    "aggType": "count",
    "groupBy": "all",
    "termSize": 5,
    "excludeHitsFromPreviousRun": false,
    "sourceFields": [],
    "index": [
      ".kibana-event-log*"
    ],
    "timeField": "@timestamp"
  },
  "consumer": "stackAlerts",
  "schedule": {
    "interval": "1m"
  },
  "tags": [],
  "name": "test",
  "rule_type_id": ".es-query",
  "actions": [
    {
      "group": "query matched",
      "id": "${ACTION_ID}",
      "params": {
        "level": "info",
        "message": """Elasticsearch query rule '{{rule.name}}' is active:

- Value: {{context.value}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}"""
      },
      "frequency": {
        "notify_when": "onActionGroupChange",
        "throttle": null,
        "summary": false
      }
    }
  ],
  "alert_delay": {
    "active": 3
  }
}

Verify that the alert will not be created until it has matched the delay threshold.
Verify that the delay does not affect recovered alerts

…bana into alerting/notification-delay

…cation-delay-fe

doakalexi · 2024-01-29T23:10:36Z

/ci

doakalexi · 2024-01-30T21:47:54Z

/ci

x-pack/plugins/triggers_actions_ui/public/application/sections/rule_form/rule_form.tsx

doakalexi · 2024-01-31T00:22:27Z

/ci

elasticmachine · 2024-01-31T16:41:11Z

Pinging @elastic/response-ops (Team:ResponseOps)

…/kibana into alerting/notification-delay-fe

ymao1 · 2024-02-01T16:13:34Z

I guess it's safe to assume that no rules will have been created with the notificationDelay config in serverless since it was API only? Otherwise they will be skipped due to validation error

[2024-02-01T11:12:21.408-05:00][WARN ][plugins.taskManager] Task (alerting:.es-query/85750824-12ec-4954-acfb-ec9b65899fc0) has a validation error in its indirect params: [notificationDelay]: definition for this key is missing

doakalexi · 2024-02-01T16:23:29Z

I guess it's safe to assume that no rules will have been created with the notificationDelay config in serverless since it was API only? Otherwise they will be skipped due to validation error
[2024-02-01T11:12:21.408-05:00][WARN ][plugins.taskManager] Task (alerting:.es-query/85750824-12ec-4954-acfb-ec9b65899fc0) has a validation error in its indirect params: [notificationDelay]: definition for this key is missing

Yes, it was API only. I don't think anyone should have added a notification delay in this case.

kibana-ci · 2024-02-05T16:13:56Z

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

✅ unchanged

History

💚 Build #190627 succeeded ec1d0d9
💔 Build #190606 failed 26b6a56
💛 Build #190538 was flaky 7082d03
💚 Build #190323 succeeded 134fae2
💔 Build #190312 failed 468f94d
💔 Build #190067 failed 9f3575a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

ymao1

LGTM! Verified using an ES query rule type and a custom threshold rule type and alert delay works as described. Also tried upgrading to this PR branch from main with existing rules and verified that they continue running successfully.

* main: (224 commits) [Http] Replace `buildNr` with `buildSha` in static asset paths (#175898) [Ops] Fix GCS bucket access for future buildkite agents (#174756) [api-docs] 2024-02-07 Daily api_docs build (#176362) skip flaky suite (#176002) skip failing es promotion suite (#176359) [Cloud Security] [Grouping] Add URL Params support to the grouping components (#175749) chore(NA): update versions after v8.12.2 bump (#176309) chore(NA): update versions after v7.17.19 bump (#176313) skip failing test suite (#176352) [SLO] Enable burn rate alert by default during creation via UI (#176317) [Fleet] Add the uptime capability to observability projects (#176285) [Security Solution][Endpoint] Fix Manifest Manger so that it works with large (>10k) (#174411) [ResponseOps] Alert creation delay based on user definition (#175851) [data views] Default field formatters based on field meta values (#174973) [Cloud Security]Detection Rules counter on Rules Flyout (#176041) [Security Solution] Data Quality Dashboard persistence (#175673) [Ent Search] Connector client copy cleanup (#176290) [ML] Anomaly Detection: Adds actions menu to anomaly markers in Single Metric Viewer chart. (#175556) [ML] Anomaly Detection: Fix `values-dots` colors (#176303) [Fleet] Logstash Output - being compliant to RFC-952 (#176298) ...

@timestamp

…175851) Resolves elastic#173009 ## Summary This PR: - Changes the field name from `notification_delay` to `alert_delay` - Updates the alerts client and rule registry to index new alert docs on a delay - Updates the framework code to delay the creation of an alert ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `alertDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "alert_delay": { "active": 3 } } ``` - Verify that the alert will not be created until it has matched the delay threshold. - Verify that the delay does not affect recovered alerts

@timestamp

…175851) Resolves elastic#173009 ## Summary This PR: - Changes the field name from `notification_delay` to `alert_delay` - Updates the alerts client and rule registry to index new alert docs on a delay - Updates the framework code to delay the creation of an alert ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `alertDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "alert_delay": { "active": 3 } } ``` - Verify that the alert will not be created until it has matched the delay threshold. - Verify that the delay does not affect recovered alerts

@timestamp

…175851) Resolves elastic#173009 ## Summary This PR: - Changes the field name from `notification_delay` to `alert_delay` - Updates the alerts client and rule registry to index new alert docs on a delay - Updates the framework code to delay the creation of an alert ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `alertDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "alert_delay": { "active": 3 } } ``` - Verify that the alert will not be created until it has matched the delay threshold. - Verify that the delay does not affect recovered alerts

@timestamp

…175851) Resolves elastic#173009 ## Summary This PR: - Changes the field name from `notification_delay` to `alert_delay` - Updates the alerts client and rule registry to index new alert docs on a delay - Updates the framework code to delay the creation of an alert ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### To verify - Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to create a rule with the `alertDelay` ``` POST kbn:/api/alerting/rule { "params": { "searchType": "esQuery", "timeWindowSize": 5, "timeWindowUnit": "m", "threshold": [ -1 ], "thresholdComparator": ">", "size": 100, "esQuery": """{ "query":{ "match_all" : {} } }""", "aggType": "count", "groupBy": "all", "termSize": 5, "excludeHitsFromPreviousRun": false, "sourceFields": [], "index": [ ".kibana-event-log*" ], "timeField": "@timestamp" }, "consumer": "stackAlerts", "schedule": { "interval": "1m" }, "tags": [], "name": "test", "rule_type_id": ".es-query", "actions": [ { "group": "query matched", "id": "${ACTION_ID}", "params": { "level": "info", "message": """Elasticsearch query rule '{{rule.name}}' is active: - Value: {{context.value}} - Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}} - Timestamp: {{context.date}} - Link: {{context.link}}""" }, "frequency": { "notify_when": "onActionGroupChange", "throttle": null, "summary": false } } ], "alert_delay": { "active": 3 } } ``` - Verify that the alert will not be created until it has matched the delay threshold. - Verify that the delay does not affect recovered alerts

doakalexi and others added 13 commits January 10, 2024 16:08

Adding the notification delay

1b69332

Adding AAD test

89cc726

Removing changes to rule registry

275520b

Change activeCount to active

2931666

Merge branch 'main' into alerting/notification-delay

5ba8271

Fixing tests

4f160be

Merge branch 'alerting/notification-delay' of github.com:doakalexi/ki…

394c10d

…bana into alerting/notification-delay

Adding jest tests

f655c35

Adding notification delay to the front end

95ddef5

Updating tests

324b151

Merge branch 'main' of github.com:elastic/kibana into alerting/notifi…

3058513

…cation-delay-fe

Renaming to alert delay

8aec35a

Updating the alert creation delay

9f3575a

Updating tests

468f94d

doakalexi changed the title ~~Alerting/notification delay fe~~ [ResponseOps] Alert creation delay based on user definition Jan 30, 2024

doakalexi commented Jan 30, 2024

View reviewed changes

x-pack/plugins/triggers_actions_ui/public/application/sections/rule_form/rule_form.tsx Outdated Show resolved Hide resolved

doakalexi commented Jan 30, 2024

View reviewed changes

x-pack/plugins/triggers_actions_ui/public/application/sections/rule_form/rule_form.tsx Outdated Show resolved Hide resolved

Fixing test failures

134fae2

doakalexi marked this pull request as ready for review January 31, 2024 16:40

doakalexi requested review from a team as code owners January 31, 2024 16:40

doakalexi added release_note:enhancement v8.13.0 Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels Jan 31, 2024

doakalexi and others added 2 commits January 31, 2024 08:48

Merge branch 'main' into alerting/notification-delay-fe

7082d03

Remove event log changes

8bfda46

doakalexi added 3 commits January 31, 2024 12:39

Merge branch 'alerting/notification-delay-fe' of github.com:doakalexi…

a77187e

…/kibana into alerting/notification-delay-fe

Remove ui changes

75a4e50

Remove new line

26b6a56

doakalexi added release_note:skip Skip the PR/issue when compiling release notes and removed release_note:enhancement labels Jan 31, 2024

Merge branch 'main' into alerting/notification-delay-fe

ec1d0d9

Merge branch 'main' into alerting/notification-delay-fe

c3fdfec

ymao1 approved these changes Feb 6, 2024

View reviewed changes

doakalexi merged commit 31fbc86 into elastic:main Feb 6, 2024
39 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Feb 6, 2024

doakalexi mentioned this pull request Feb 13, 2024

[ResponseOps][FE] Alert creation delay based on user definition #176346

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ResponseOps] Alert creation delay based on user definition #175851

[ResponseOps] Alert creation delay based on user definition #175851

doakalexi commented Jan 29, 2024 •

edited

Loading

doakalexi commented Jan 29, 2024

doakalexi commented Jan 30, 2024

doakalexi commented Jan 31, 2024

elasticmachine commented Jan 31, 2024

ymao1 commented Feb 1, 2024

doakalexi commented Feb 1, 2024 •

edited

Loading

kibana-ci commented Feb 5, 2024

ymao1 left a comment

[ResponseOps] Alert creation delay based on user definition #175851

[ResponseOps] Alert creation delay based on user definition #175851

Conversation

doakalexi commented Jan 29, 2024 • edited Loading

Summary

Checklist

To verify

doakalexi commented Jan 29, 2024

doakalexi commented Jan 30, 2024

doakalexi commented Jan 31, 2024

elasticmachine commented Jan 31, 2024

ymao1 commented Feb 1, 2024

doakalexi commented Feb 1, 2024 • edited Loading

kibana-ci commented Feb 5, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

History

ymao1 left a comment

Choose a reason for hiding this comment

doakalexi commented Jan 29, 2024 •

edited

Loading

doakalexi commented Feb 1, 2024 •

edited

Loading