Event Notification Backlog Issue

[klesher]

Enabling and setting a Message Backlog > 1 in the Notification settings appears to have no effect on the resulting e-mail notification. This occurs even when creating a new Event Definition and Notification in Graylog v3.3.9. No error messages are thrown in the ElasticSearch or Graylog application logs.

Expected Behavior

A number of messages returned by the event filter, up to a count of $message_backlog, should be included in the e-mail notification.

Current Behavior

A single event matched by the filter is included in the e-mail notification.

Possible Solution

Steps to Reproduce (for bugs)

1. Create a new event definition which could have multiple matches in a given time period. Ensure that in the "Notifications" section, Message Backlog is enabled and set to a value greater than 1 (we used 50).
2. Create an e-mail notification and associate it with this event. The default Body Template as follows was used:

--- [Event Definition] ---------------------------
Title:       ${event_definition_title}
Description: ${event_definition_description}
Type:        ${event_definition_type}
--- [Event] --------------------------------------
Timestamp:            ${event.timestamp}
Message:              ${event.message}
Source:               ${event.source}
Key:                  ${event.key}
Priority:             ${event.priority}
Alert:                ${event.alert}
Timestamp Processing: ${event.timestamp}
Timerange Start:      ${event.timerange_start}
Timerange End:        ${event.timerange_end}
Fields:
${foreach event.fields field}  ${field.key}: ${field.value}
${end}
${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
${message}
${end}
${end}

3. Wait for Event to trigger an e-mail notification.

4. A single backlog message is included in the e-mail:

--- [Event Definition] ---------------------------
Title:       Testing
Description:
Type:        aggregation-v1
--- [Event] --------------------------------------
Timestamp:            2020-12-01T17:32:44.000Z
Message:              Testing
Source:               graylog-prod
Key:                 
Priority:             2
Alert:                true
Timestamp Processing: 2020-12-01T17:32:44.000Z
Timerange Start:     
Timerange End:       
Fields:


--- [Backlog] ------------------------------------
Last messages accounting for this alert:

{index=rsyslog_97, message=<redacted>, timestamp=2020-12-01T17:32:44.000Z, source=<redacted>>, stream_ids=[<redacted>], fields={gl2_accounted_message_size=374, level=5, gl2_remote_ip=<redacted>, gl2_remote_port=<redacted>, gl2_message_id=<redacted>, gl2_source_node=<redacted>, gl2_source_input=<redacted>, facility=local0}, id=<redacted>}

Context

This issue makes it a more difficult to properly respond to alerts. We either need to accept a flood of e-mail alerts each time an event is matched (I.E. a single e-mail per event), or manually search to see if there are other events when a notification fires. The ability to see that there are multiple matched alerts in a single event notification would provide additional welcome context as to potential severity or impact of an Event that a single match might not.

Your Environment

• Graylog Version: 3.3.9
• Java Version: 1.7.0
• Elasticsearch Version:
• MongoDB Version: 4.0.21
• Operating System: Amazon Linux
• Browser version: Chrome 87.0.4280.67

Thanks & happy to provide any additional information/perform any troubleshooting needed!

[chalfling]

And if your Alert Query match a message that is not was used to trigger the Alert, but was received after(is last message in query match) then your Backlog will be sent with the wrong content.

I described in this issue: #8828