I have an event definition setup as follows:
- Filter & Aggregation
- Search Query: [...]
- Search within the last: 30 minutes
- Execute search every: 30 minutes
- Notifications
- Script Notification
- Grace Period: 6 hours
- Message Backlog: 1
However, when the alert condition is met, the script is executed every search interval (30 minutes) despite the grace period being set to 6 hours. I can't find a way to audit in graylog when an alert is fired and sends an alert VS not sending one because of the grace period. However, because this script triggers a pushover i can see exactly when an alert was executed.
Whats interesting is the graylog stream for events logs an event once per minute, despite the Execute search every being set to 30 minutes:
Pushover set the following alerts:
- 11:17 AM
- 11:47 AM (ignored grace period)
- 12:17 PM (ignored grace period)
- 12:47 PM (ignored grace period)
Expected Behavior
Grace period is obeyed.
Current Behavior
Grace period is ignored.
Possible Solution
Steps to Reproduce (for bugs)
I was able to reproduce using the following Event Definition:
- Filter & Aggregation:
- Condition Type: Filter & Aggregation
- Search Query:
_exists_:timestamp
- Search within the last: 5 minutes
- Execute search ever: 5 minutes
- Notifications
- Script Notification [Enterprise]
- Grace Period: 1 hour
Logging shows as follows:
2022-12-19T14:52:35.950-06:00 DEBUG [EventDefinitionHandler] Created scheduler job definition <63a0cf133b871f7f68f99cdc/A Test - Duplicate Alerts ignoring cooldown> for event definition <6384e92d86ca562e159decd9/A Test - Duplicate Alerts ignoring cooldown>
2022-12-19T14:52:35.964-06:00 DEBUG [EventDefinitionHandler] Created job trigger <63a0cf133b871f7f68f99cdd> for job definition <63a0cf133b871f7f68f99cdc/A Test - Duplicate Alerts ignoring cooldown> and event definition <6384e92d86ca562e159decd9/A Test - Duplicate Alerts ignoring cooldown>
2022-12-19T14:52:36.793-06:00 DEBUG [EventProcessorEngine] Executing event processor <A Test - Duplicate Alerts ignoring cooldown/6384e92d86ca562e159decd9/aggregation-v1>
2022-12-19T14:52:40.861-06:00 DEBUG [DBEventProcessorStateService] Update event processor state for <6384e92d86ca562e159decd9> with min processed timestamp of <2022-12-19T20:47:35.927Z> max processed timestamp of <2022-12-19T20:52:35.927Z>
5 minutes later (next time event definition executes)
2022-12-19T14:57:36.183-06:00 DEBUG [EventProcessorEngine] Executing event processor <A Test - Duplicate Alerts ignoring cooldown/6384e92d86ca562e159decd9/aggregation-v1>
2022-12-19T14:57:36.227-06:00 DEBUG [EventProcessorExecutionJob] Event processor <A Test - Duplicate Alerts ignoring cooldown/6384e92d86ca562e159decd9> couldn't be executed because of a failed precondition (retry in 5000 ms)
2022-12-19T14:57:41.260-06:00 DEBUG [EventProcessorEngine] Executing event processor <A Test - Duplicate Alerts ignoring cooldown/6384e92d86ca562e159decd9/aggregation-v1>
2022-12-19T14:57:47.520-06:00 DEBUG [DBEventProcessorStateService] Update event processor state for <6384e92d86ca562e159decd9> with min processed timestamp of <2022-12-19T20:52:35.927Z> max processed timestamp of <2022-12-19T20:57:35.927Z>
Context
A grace period is crucial to prevent excessive alerts and alert fatique.
Your Environment
- Graylog Version: Graylog 5.0.1
- Java Version: 17.0.5
- Elasticsearch Version: OpenSearch 2.4.1
- MongoDB Version: 5.0.14
- Operating System: Ubuntu Server 20.04 LTS
- Browser version: Chrome 108.0.5359.124
I have an event definition setup as follows:
However, when the alert condition is met, the script is executed every search interval (30 minutes) despite the grace period being set to 6 hours. I can't find a way to audit in graylog when an alert is fired and sends an alert VS not sending one because of the grace period. However, because this script triggers a pushover i can see exactly when an alert was executed.
Whats interesting is the graylog stream for events logs an event once per minute, despite the
Execute search everybeing set to 30 minutes:Pushover set the following alerts:
Expected Behavior
Grace period is obeyed.
Current Behavior
Grace period is ignored.
Possible Solution
Steps to Reproduce (for bugs)
I was able to reproduce using the following Event Definition:
_exists_:timestampLogging shows as follows:
5 minutes later (next time event definition executes)
Context
A grace period is crucial to prevent excessive alerts and alert fatique.
Your Environment