Exclude non-message streams from search in aggregation events #17087
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
patrickmann
requested review from
a team and
thll
and removed request for
a team
thll
requested changes
Nov 7, 2023
| @@ -219,7 +220,7 @@ private Set<String> getStreams(AggregationEventProcessorParameters parameters) { | |||
|
|
|||
| private void filterSearch(EventFactory eventFactory, AggregationEventProcessorParameters parameters, | |||
| EventConsumer<List<EventWithContext>> eventsConsumer) throws EventProcessorException { | |||
| final Set<String> streams = getStreams(parameters); | |||
| final Set<String> streams = eventStreamService.buildEventSourceStreams(getStreams(parameters), Collections.emptySet()); | |||
There was a problem hiding this comment.
I think this is the wrong method, because, as the name suggests, it is supposed to operate on the results of a search. I think we should use the same method that we use for aggregations here.
It appears as if that one will also take care of filtering out the investigation streams, if the enterprise plugin is present. Otherwise we'll end up using different stream filters on the search queries for aggregation and filter events.
thll
approved these changes
Nov 10, 2023
maxiadlovskii
pushed a commit
that referenced
this pull request
Nov 14, 2023
* exclude non-message streams from default search * update CL * update CL * apply filtering to search query * use PermittedStreams for filter search
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relates to Graylog2/graylog-plugin-enterprise#6042
When no source stream is given for an event definition we use all streams in the ES/OS search query. We now exclude all non-message streams.
Motivation and Context
Users cannot select non-message streams as a source stream in the event definition. But these were still being included in the search when no source stream had been explicitly defined. This is unnecessary, inconsistent, and hurts performance. It may also push the search URI over the length limit, when there is a large number of streams.
Types of changes
Checklist: