Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Notify when aggregation search throws an exception #14967

Merged
merged 7 commits into from Mar 22, 2023

Conversation

patrickmann
Copy link
Contributor

resolves #14746

An event definition with aggregation will fail silently (with just a log message) when encountering an ES/OS error.
We now also generate a notification to alert the admin that events are getting lost.

@patrickmann patrickmann changed the title first draft Notify when aggregation search throws an exception Mar 20, 2023
@patrickmann patrickmann requested a review from a team March 20, 2023 07:39
@@ -144,9 +149,18 @@ public AggregationResult doSearch() throws EventProcessorException {
});

// If we have only EmptyParameterErrors, just return an empty Result
if (! (errors.stream().filter(e -> !(e instanceof EmptyParameterError)).count() > 1)) {
if (errors.stream().allMatch(e -> e instanceof EmptyParameterError)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might consider backporting this fix by itself

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll check with triage

@patrickmann patrickmann requested a review from mpfz0r March 21, 2023 15:52
.addType(Notification.Type.SEARCH_ERROR)
.addSeverity(Notification.Severity.NORMAL)
.addTimestamp(DateTime.now(DateTimeZone.UTC))
.addKey(searchJob.getId())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to use the eventdefintion id here.
Otherwise we might get a new notification for each failed attempt, which might be a lot.

.addSeverity(Notification.Severity.NORMAL)
.addTimestamp(DateTime.now(DateTimeZone.UTC))
.addKey(searchJob.getId())
.addDetail("title", "Aggregation search failed")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also add the title/id of the event definition here

@patrickmann patrickmann merged commit afc32fb into master Mar 22, 2023
1 check passed
@patrickmann patrickmann deleted the notifyAggregationSearchError branch March 22, 2023 12:30
ousmaneo pushed a commit that referenced this pull request Mar 29, 2023
* first draft

* cleanup

* linter fix

* fix tests

* review feedback

* improve notification information
R4103N pushed a commit to R4103N/graylog2-server that referenced this pull request Mar 31, 2023
* first draft

* cleanup

* linter fix

* fix tests

* review feedback

* improve notification information
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Event fails silently on huge dataset from search provider
2 participants