Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ResponseOps] Include alert creation delay in the event log #175941

Closed
doakalexi opened this issue Jan 30, 2024 · 1 comment · Fixed by #176348
Closed

[ResponseOps] Include alert creation delay in the event log #175941

doakalexi opened this issue Jan 30, 2024 · 1 comment · Fixed by #176348
Assignees
@doakalexi
Labels
Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@doakalexi
Copy link
Contributor

Related to #173009

We should document in the event log when alert creation is delayed for a rule run.
@doakalexi doakalexi added the Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) label Jan 30, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@doakalexi doakalexi self-assigned this Jan 31, 2024
This was referenced Feb 2, 2024
Merged
Merged
doakalexi added a commit that referenced this issue Feb 12, 2024
@doakalexi
[ResponseOps] Include alert creation delay in the event log (#176348)
d0dfc33 
Resolves #175941

## Summary

Adds a new field, `number_of_delayed_alerts`, to the event log for a
rule run.
It's a count for all the delayed alerts, I opted to go this route
instead of counting the number of times each alert was delayed. Pls let
me know if you would like to go another way or would like to add any
other metrics :)


### To verify
-Go to Dev Tools
- Create a rule with the alert delay
```
POST kbn:/api/alerting/rule
{
  "params": {
    "searchType": "esQuery",
    "timeWindowSize": 5,
    "timeWindowUnit": "m",
    "threshold": [
      -1
    ],
    "thresholdComparator": ">",
    "size": 100,
    "esQuery": """{
    "query":{
      "match_all" : {}
    }
  }""",
    "aggType": "count",
    "groupBy": "all",
    "termSize": 5,
    "excludeHitsFromPreviousRun": false,
    "sourceFields": [],
    "index": [
      ".kibana-event-log*"
    ],
    "timeField": "@timestamp"
  },
  "consumer": "stackAlerts",
  "schedule": {
    "interval": "1m"
  },
  "tags": [],
  "name": "test",
  "rule_type_id": ".es-query",
  "actions": [
    {
      "group": "query matched",
      "id": "${ACTION_ID}",
      "params": {
        "level": "info",
        "message": """Elasticsearch query rule '{{rule.name}}' is active:

- Value: {{context.value}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}"""
      },
      "frequency": {
        "notify_when": "onActionGroupChange",
        "throttle": null,
        "summary": false
      }
    }
  ],
  "alert_delay": {
    "active": 3
  }
}
```
- Let the rule run and then run the following to look at the event log
- Verify that there is a new field `number_of_delayed_alerts` and that
the counts what you would would expect for a rule running with the
alert_delay
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@doakalexi @CoenWarmer
[ResponseOps] Include alert creation delay in the event log (elastic#…
8e70c03 
…176348)

Resolves elastic#175941

## Summary

Adds a new field, `number_of_delayed_alerts`, to the event log for a
rule run.
It's a count for all the delayed alerts, I opted to go this route
instead of counting the number of times each alert was delayed. Pls let
me know if you would like to go another way or would like to add any
other metrics :)


### To verify
-Go to Dev Tools
- Create a rule with the alert delay
```
POST kbn:/api/alerting/rule
{
  "params": {
    "searchType": "esQuery",
    "timeWindowSize": 5,
    "timeWindowUnit": "m",
    "threshold": [
      -1
    ],
    "thresholdComparator": ">",
    "size": 100,
    "esQuery": """{
    "query":{
      "match_all" : {}
    }
  }""",
    "aggType": "count",
    "groupBy": "all",
    "termSize": 5,
    "excludeHitsFromPreviousRun": false,
    "sourceFields": [],
    "index": [
      ".kibana-event-log*"
    ],
    "timeField": "@timestamp"
  },
  "consumer": "stackAlerts",
  "schedule": {
    "interval": "1m"
  },
  "tags": [],
  "name": "test",
  "rule_type_id": ".es-query",
  "actions": [
    {
      "group": "query matched",
      "id": "${ACTION_ID}",
      "params": {
        "level": "info",
        "message": """Elasticsearch query rule '{{rule.name}}' is active:

- Value: {{context.value}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}"""
      },
      "frequency": {
        "notify_when": "onActionGroupChange",
        "throttle": null,
        "summary": false
      }
    }
  ],
  "alert_delay": {
    "active": 3
  }
}
```
- Let the rule run and then run the following to look at the event log
- Verify that there is a new field `number_of_delayed_alerts` and that
the counts what you would would expect for a rule running with the
alert_delay
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@doakalexi @CoenWarmer
[ResponseOps] Include alert creation delay in the event log (elastic#…
603e9a6 
…176348)

Resolves elastic#175941

## Summary

Adds a new field, `number_of_delayed_alerts`, to the event log for a
rule run.
It's a count for all the delayed alerts, I opted to go this route
instead of counting the number of times each alert was delayed. Pls let
me know if you would like to go another way or would like to add any
other metrics :)


### To verify
-Go to Dev Tools
- Create a rule with the alert delay
```
POST kbn:/api/alerting/rule
{
  "params": {
    "searchType": "esQuery",
    "timeWindowSize": 5,
    "timeWindowUnit": "m",
    "threshold": [
      -1
    ],
    "thresholdComparator": ">",
    "size": 100,
    "esQuery": """{
    "query":{
      "match_all" : {}
    }
  }""",
    "aggType": "count",
    "groupBy": "all",
    "termSize": 5,
    "excludeHitsFromPreviousRun": false,
    "sourceFields": [],
    "index": [
      ".kibana-event-log*"
    ],
    "timeField": "@timestamp"
  },
  "consumer": "stackAlerts",
  "schedule": {
    "interval": "1m"
  },
  "tags": [],
  "name": "test",
  "rule_type_id": ".es-query",
  "actions": [
    {
      "group": "query matched",
      "id": "${ACTION_ID}",
      "params": {
        "level": "info",
        "message": """Elasticsearch query rule '{{rule.name}}' is active:

- Value: {{context.value}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}"""
      },
      "frequency": {
        "notify_when": "onActionGroupChange",
        "throttle": null,
        "summary": false
      }
    }
  ],
  "alert_delay": {
    "active": 3
  }
}
```
- Let the rule run and then run the following to look at the event log
- Verify that there is a new field `number_of_delayed_alerts` and that
the counts what you would would expect for a rule running with the
alert_delay
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@doakalexi @fkanout
[ResponseOps] Include alert creation delay in the event log (elastic#…
818e1f9 
…176348)

Resolves elastic#175941

## Summary

Adds a new field, `number_of_delayed_alerts`, to the event log for a
rule run.
It's a count for all the delayed alerts, I opted to go this route
instead of counting the number of times each alert was delayed. Pls let
me know if you would like to go another way or would like to add any
other metrics :)


### To verify
-Go to Dev Tools
- Create a rule with the alert delay
```
POST kbn:/api/alerting/rule
{
  "params": {
    "searchType": "esQuery",
    "timeWindowSize": 5,
    "timeWindowUnit": "m",
    "threshold": [
      -1
    ],
    "thresholdComparator": ">",
    "size": 100,
    "esQuery": """{
    "query":{
      "match_all" : {}
    }
  }""",
    "aggType": "count",
    "groupBy": "all",
    "termSize": 5,
    "excludeHitsFromPreviousRun": false,
    "sourceFields": [],
    "index": [
      ".kibana-event-log*"
    ],
    "timeField": "@timestamp"
  },
  "consumer": "stackAlerts",
  "schedule": {
    "interval": "1m"
  },
  "tags": [],
  "name": "test",
  "rule_type_id": ".es-query",
  "actions": [
    {
      "group": "query matched",
      "id": "${ACTION_ID}",
      "params": {
        "level": "info",
        "message": """Elasticsearch query rule '{{rule.name}}' is active:

- Value: {{context.value}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}"""
      },
      "frequency": {
        "notify_when": "onActionGroupChange",
        "throttle": null,
        "summary": false
      }
    }
  ],
  "alert_delay": {
    "active": 3
  }
}
```
- Let the rule run and then run the following to look at the event log
- Verify that there is a new field `number_of_delayed_alerts` and that
the counts what you would would expect for a rule running with the
alert_delay
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@doakalexi doakalexi

Labels
Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
No open projects
AppEx: ResponseOps - Execution & Connectors
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ResponseOps] Include alert creation delay in the event log doakalexi/kibana
2 participants
@elasticmachine @doakalexi