forked from elastic/kibana
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Terms query for Indicator Match rule (elastic#144511)
## Terms query for Indicator Match rule TODO: [] need more unit/integrations tests, but ready for review The indicator match rule will use terms query when it is possible to search for matches for threat-first-search and for events-first-search. ## How the match query worked: Example for threat-first-search. If we have matching conditions like: `host.ip ==== indicator.host.ip` or (`source.name === indicator.source.name` AND `host.name === indicator.host.name`) It will generate queries like: ``` match: {host.ip: "1"}, or match: {host.ip: "2"} or match: {host.ip: "3"} or (match: {source.name: "1"} and match: {host.name: "1"}) or (match: {source.name: "2"} and match: {host.name: "2"}) or (match: {source.name: "3"} and match: {host.name: "3"}) ``` Each match will also have `_name` fields like: `${threatId}_${threatIndex}_${threatFields}_${sourceField}` So and because it's 1:1 relation between match and response, later at enrichment stage will be clear which threat matches which event. ## Terms query. We do fetch info about mapping for fields which use for match conditions of the IM rule. Terms query doesn't support all field types, this is why there is some allowed list which field types. Terms query not applied for AND conditions. For example: Fields types host.ip - `ip` user.name - `keyword` user.description - `text` indicator.host.ip_range - `ip_range` `host.ip === indicator.host.ip` or `host.ip_range === indicator.host.ip` or (`source.name === indicator.source.name` AND `host.name === indicator.host.name`) It will generate queries like: ``` terms: {host.ip: ["1","2","3"]}, or match: {host.ip_range: "1"} // terms query support range fields, but it will be difficult later to understand which threat match which event, because we can have more than 1 response for this condition or match: {host.ip_range: "2"} or (match: {source.name: "1"} and match: {host.name: "1"}) or (match: {source.name: "2"} and match: {host.name: "2"}) or (match: {source.name: "3"} and match: {host.name: "3"}) ``` For terms query, we don't know which response matches with events, this is why we do match it back in the code. ## Other changes Threat-first-search - will do one extra request to have all matched threats. For example: The threat index has 1.000.000 documents. IM rule gets the first batch of 9.000 threats and builds a query to the events index. It returns 100 events (max_signal = 100). Then it tries to enrich those 100 events with threat info. The problem is that the original implementation will enrich with the only threats from this 9.000 batch. And it will ignore other matches in 1.000.000 threats. This way we do one extra request in the end from potential alerts to threat index. # Tests performance In the best case, it can improve performance by around 3x times. [Base](elastic#149113) Threat Indicators - 1.500.000 documents Source - 1.000.000 documents. 1 field for match condition <img width="557" alt="213484531-3ab68c61-c3f5-4e28-b2c4-c1e90a5b1775" src="https://user-images.githubusercontent.com/7609147/215526984-ff027ba1-2f64-49fe-8fe8-a23ff4eda4dc.png"> This PR: <img width="537" alt="Screenshot 2023-01-30 at 20 20 32" src="https://user-images.githubusercontent.com/7609147/215575128-730514ac-a186-4ab8-87fd-af2ea8f79cec.png"> --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
- Loading branch information
Showing
16 changed files
with
1,232 additions
and
161 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.