diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 45a9174607..5876f17865 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -88,6 +88,12 @@ Detection rules:: Background tasks that run periodically and produce alerts when suspicious activity is detected. +[[term-sec-indices]] +{es-sec} indices:: +Indices containing host and network source events (such as +`packetbeat-*`, `log-*`, and `winlogbeat-*`). Detection rules use these +indices to generate alerts when suspicious activity is detected. + Endpoint exceptions:: <> added to both rules and Endpoint agents on hosts. Endpoint exceptions can only be added when: @@ -103,6 +109,11 @@ alerts. External alerts:: Alerts {es-sec} receives from external systems, such as Suricata. +Threat indices:: +Indices containing suspect field values. Threat-match detection rules use these +indices to compare their field values with source event values contained in +<>. + [float] [[detections-permissions]] == Detections configuration and index privilege prerequisites diff --git a/docs/detections/detections-ui-exceptions.asciidoc b/docs/detections/detections-ui-exceptions.asciidoc index 7e1ebde02a..a277dd979c 100644 --- a/docs/detections/detections-ui-exceptions.asciidoc +++ b/docs/detections/detections-ui-exceptions.asciidoc @@ -2,8 +2,14 @@ [role="xpack"] == Rule exceptions and value lists -To prevent the creation of unwanted alerts, you can add exceptions to detection -rules. Exceptions contain the source event conditions that determine when +To prevent the creation of unwanted alerts, you can add exceptions to these +detection rule types: + +* Custom query +* EQL +* Threat match + +Exceptions contain the source event conditions that determine when alerts are not generated. They provide a convenient way of allowing trusted processes and network activity to function without producing unnecessary noise. @@ -42,7 +48,6 @@ NOTE: All values in the file must be of the same {es} type. . Click *Upload value lists*. + The *Upload value lists* window opens. - [role="screenshot"] image::images/upload-lists-ui.png[] @@ -61,15 +66,23 @@ To view, delete, or export existing lists: [float] [[detection-rule-exceptions]] -=== Add detection exceptions to a rule +=== Add exceptions to a rule You can add exceptions to a rule via the Rule details page or the Alerts table. When you add an exception, you can also close all alerts that meet the exception's criteria. -IMPORTANT: When you select to close all alerts that meet the exception's -criteria, all matching alerts are closed, *including* alerts generated by other -rules. +[IMPORTANT] +============== +Be careful when adding exceptions to EQL sequence rules. Exceptions are +evaluated against every event in the sequence and when the exception matches any +event(s) in the sequence, alerts are not generated. To exclude values from a +specific event in the sequence, update the rule's EQL statement. For example: + +`sequence + [file where file.extension == "exe" and file.name != "app-name.exe"] + [process where true and process.name != "process-name.exe"]` +============== . To add an exception via the Rule details page: .. Go to the Rule details page of the rule to which you want to add the @@ -86,7 +99,6 @@ The *Add Exception* window opens (via Alerts table). + [role="screenshot"] image::images/add-exception-ui.png[] - . Add conditions that define when the exception prevents alerts. You can define multiple conditions with `OR` and `AND` relationships. In the example above, the exception prevents the rule from generating alerts when the @@ -105,7 +117,6 @@ values in a list with `is in list` and `is not in list` operators: + [role="screenshot"] image::images/exceptions-ui-list.png[] - NOTE: When using a list, all exception statements must use `is in list` and `is not in list` operators. @@ -115,6 +126,10 @@ NOTE: When using a list, all exception statements must use `is in list` and is only available when adding exceptions via the Alerts table. * _Close all alerts that match this exception, including alerts generated by other rules_: Closes all alerts that match the exception's conditions. ++ +IMPORTANT: When you select to close all alerts that meet the exception's +criteria, all matching alerts are closed, *including* alerts generated by other +rules. . Click *Add Exception*. @@ -156,7 +171,6 @@ The *Add Endpoint Exception* window opens (via Alerts table). + [role="screenshot"] image::images/endpoint-add-exp.png[] - . If required, modify the conditions. + NOTE: <> describes when nested conditions are required. @@ -234,4 +248,4 @@ correctly: Creates an exception that excludes all LFC-signed trusted processes: [role="screenshot"] -image::images/nested-exp.png[] \ No newline at end of file +image::images/nested-exp.png[] diff --git a/docs/detections/images/create-new-rule.png b/docs/detections/images/create-new-rule.png index 5488cd8780..07b7f4dcbd 100644 Binary files a/docs/detections/images/create-new-rule.png and b/docs/detections/images/create-new-rule.png differ diff --git a/docs/detections/images/eql-rule-query-example.png b/docs/detections/images/eql-rule-query-example.png new file mode 100644 index 0000000000..7c71f2610d Binary files /dev/null and b/docs/detections/images/eql-rule-query-example.png differ diff --git a/docs/detections/images/threat-math-rule-query-example.png b/docs/detections/images/threat-math-rule-query-example.png new file mode 100644 index 0000000000..2040ffd6c6 Binary files /dev/null and b/docs/detections/images/threat-math-rule-query-example.png differ diff --git a/docs/detections/rules-ui-create.asciidoc b/docs/detections/rules-ui-create.asciidoc index 0321e0591b..8cd9de882a 100644 --- a/docs/detections/rules-ui-create.asciidoc +++ b/docs/detections/rules-ui-create.asciidoc @@ -29,6 +29,17 @@ generated for each value. For example, if the threshold `field` is `source.ip` and its `value` is `10`, an alert is generated for every source IP address that appears in at least 10 of the rule's search results. +* *EQL*: Searches the defined indices and creates an alert when results match an +{ref}/eql.html[Event Query Language (EQL)] query. +* *Threat match*: Creates an alert when {es-sec} index field values match +field values defined in the specified threat indices. For example, you +can create a threat index for IP addresses and use this index to create an alert +whenever an event's `destination.ip` equals a value in the index. Threat index +field mappings should be {ecs-ref}[ECS-compliant]. For information on creating +{es} indices and field types, see +{ref}/getting-started-index.html[Index some documents], +{ref}/indices-create-index.html[Create index API] and +{ref}/mapping-types.html[Field data types]. When creating or modifying rules, you can add exceptions that prevent a rule from generating an alert even when its criteria are met. This is useful for @@ -36,6 +47,9 @@ reducing noise, such as preventing alerts from trusted processes and internal IP addresses. <> describes how to add exceptions to a rule. +NOTE: You can add exceptions to custom query, EQL, and threat-matching rule +types. + For both prebuilt and custom rules, you can use the {kib} {kibana-ref}/alerting-getting-started.html[Alerting and Actions] feature to send notifications when alerts are created. Notifications can be sent via @@ -136,6 +150,55 @@ per rule execution: ** _Field_: Leave blank ** _Threshold_: `10` +* To create a new EQL rule, select _Event Correlation_ and then: +.. Define which {es} indices the rule searches for alerts. +.. Add an {ref}/eql-syntax.html[EQL statement] used to detect alerts. ++ +For example the following rule detects when `msxsl.exe` makes an outbound +network connection: + +** _Index patterns_: `winlogbeat-*` ++ +Winlogbeat ship Windows events to {es-sec}. +** _EQL query_: `sequence by process.entity_id [process where event.type in ("start", "process_started") and process.name == "msxsl.exe"] [network where event.type == "connection" and process.name == "msxsl.exe" and network.direction == "outgoing"]` ++ +Searches the `winlogbeat-*` indices for sequences of a `msxsl.exe` process start +event followed by an outbound network connection event that was started by the +`msxsl.exe` process. ++ +[role="screenshot"] +image::images/eql-rule-query-example.png[] + +* To create a threat-match rule that searches the specified threat indices for +field values, select _Threat Match_ and then fill in these fields: +.. _Index patterns_: The {es-sec} event indices on which the rules runs. +.. _Custom query_: The query and filters used to retrieve the required results from +the {es-sec} event indices. For example, if you only need to check +`destination.ip` event values, add `destination.ip : *`. ++ +TIP: If you want the rule to check every field in the indices, use this +wildcard expression: `*:*`. +.. _Threat index patterns_: The threat indices containing field values for which +you want to generate alerts. +.. _Threat index query_: The query and filters used to filter the fields from +the threat indices. +.. _Threat Mapping_: Compares the values of the specified event and threat field +values. When the field values are identical, an alert is generated. To define +which field values are compared from the indices: +** _Field_: The field used for comparing values in the {es-sec} event +indices. +** _Threat index field_: The field used for comparing values in the threat +indices. +.. You can add `AND` and `OR` clauses to define when alerts are generated. ++ +For example, to create a rule that generates alerts when `host.name` *and* +`destination.ip` field values in the `logs-*` or `packetbeat-*` {es-sec} indices +are identical to the corresponding field values in the `ip-threat-list` threat +index: ++ +[role="screenshot"] +image::images/threat-math-rule-query-example.png[] + . Select the Timeline template used when you investigate an alert created by the rule in Timeline (optional). +