Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/detections/detection-engine-intro.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,12 @@ Detection rules::
Background tasks that run periodically and produce alerts when suspicious
activity is detected.

[[term-sec-indices]]
{es-sec} indices::
Indices containing host and network source events (such as
`packetbeat-*`, `log-*`, and `winlogbeat-*`). Detection rules use these
indices to generate alerts when suspicious activity is detected.

Endpoint exceptions::
<<term-exceptions, Exceptions>> added to both rules and Endpoint agents on
hosts. Endpoint exceptions can only be added when:
Expand All @@ -103,6 +109,11 @@ alerts.
External alerts::
Alerts {es-sec} receives from external systems, such as Suricata.

Threat indices::
Indices containing suspect field values. Threat-match detection rules use these
indices to compare their field values with source event values contained in
<<term-sec-indices, {es-sec} indices>>.

[float]
[[detections-permissions]]
== Detections configuration and index privilege prerequisites
Expand Down
36 changes: 25 additions & 11 deletions docs/detections/detections-ui-exceptions.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,14 @@
[role="xpack"]
== Rule exceptions and value lists

To prevent the creation of unwanted alerts, you can add exceptions to detection
rules. Exceptions contain the source event conditions that determine when
To prevent the creation of unwanted alerts, you can add exceptions to these
detection rule types:

* Custom query
* EQL
* Threat match
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Donnater @jmikell821 We recently renamed Threat Match to Indicator Match. I think you're already aware, but leaving this as a reminder. :)


Exceptions contain the source event conditions that determine when
alerts are not generated. They provide a convenient way of allowing trusted
processes and network activity to function without producing unnecessary noise.

Expand Down Expand Up @@ -42,7 +48,6 @@ NOTE: All values in the file must be of the same {es} type.
. Click *Upload value lists*.
+
The *Upload value lists* window opens.

[role="screenshot"]
image::images/upload-lists-ui.png[]

Expand All @@ -61,15 +66,23 @@ To view, delete, or export existing lists:

[float]
[[detection-rule-exceptions]]
=== Add detection exceptions to a rule
=== Add exceptions to a rule

You can add exceptions to a rule via the Rule details page or the Alerts table.
When you add an exception, you can also close all alerts that meet the
exception's criteria.

IMPORTANT: When you select to close all alerts that meet the exception's
criteria, all matching alerts are closed, *including* alerts generated by other
rules.
[IMPORTANT]
==============
Be careful when adding exceptions to EQL sequence rules. Exceptions are
evaluated against every event in the sequence and when the exception matches any
event(s) in the sequence, alerts are not generated. To exclude values from a
specific event in the sequence, update the rule's EQL statement. For example:

`sequence
[file where file.extension == "exe" and file.name != "app-name.exe"]
[process where true and process.name != "process-name.exe"]`
==============

. To add an exception via the Rule details page:
.. Go to the Rule details page of the rule to which you want to add the
Expand All @@ -86,7 +99,6 @@ The *Add Exception* window opens (via Alerts table).
+
[role="screenshot"]
image::images/add-exception-ui.png[]

. Add conditions that define when the exception prevents alerts. You can define
multiple conditions with `OR` and `AND` relationships. In the example above,
the exception prevents the rule from generating alerts when the
Expand All @@ -105,7 +117,6 @@ values in a list with `is in list` and `is not in list` operators:
+
[role="screenshot"]
image::images/exceptions-ui-list.png[]

NOTE: When using a list, all exception statements must use `is in list` and
`is not in list` operators.

Expand All @@ -115,6 +126,10 @@ NOTE: When using a list, all exception statements must use `is in list` and
is only available when adding exceptions via the Alerts table.
* _Close all alerts that match this exception, including alerts generated by other rules_:
Closes all alerts that match the exception's conditions.
+
IMPORTANT: When you select to close all alerts that meet the exception's
criteria, all matching alerts are closed, *including* alerts generated by other
rules.

. Click *Add Exception*.

Expand Down Expand Up @@ -156,7 +171,6 @@ The *Add Endpoint Exception* window opens (via Alerts table).
+
[role="screenshot"]
image::images/endpoint-add-exp.png[]

. If required, modify the conditions.
+
NOTE: <<ex-nested-conditions>> describes when nested conditions are required.
Expand Down Expand Up @@ -234,4 +248,4 @@ correctly:
Creates an exception that excludes all LFC-signed trusted processes:

[role="screenshot"]
image::images/nested-exp.png[]
image::images/nested-exp.png[]
Binary file modified docs/detections/images/create-new-rule.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
63 changes: 63 additions & 0 deletions docs/detections/rules-ui-create.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,27 @@ generated for each value.
For example, if the threshold `field` is `source.ip` and its `value` is `10`, an
alert is generated for every source IP address that appears in at least 10 of
the rule's search results.
* *EQL*: Searches the defined indices and creates an alert when results match an
{ref}/eql.html[Event Query Language (EQL)] query.
* *Threat match*: Creates an alert when {es-sec} index field values match
field values defined in the specified threat indices. For example, you
can create a threat index for IP addresses and use this index to create an alert
whenever an event's `destination.ip` equals a value in the index. Threat index
field mappings should be {ecs-ref}[ECS-compliant]. For information on creating
{es} indices and field types, see
{ref}/getting-started-index.html[Index some documents],
{ref}/indices-create-index.html[Create index API] and
{ref}/mapping-types.html[Field data types].

When creating or modifying rules, you can add exceptions that prevent a rule
from generating an alert even when its criteria are met. This is useful for
reducing noise, such as preventing alerts from trusted processes and internal
IP addresses. <<detections-ui-exceptions>> describes how to add exceptions to a
rule.

NOTE: You can add exceptions to custom query, EQL, and threat-matching rule
types.

For both prebuilt and custom rules, you can use the
{kib} {kibana-ref}/alerting-getting-started.html[Alerting and Actions] feature
to send notifications when alerts are created. Notifications can be sent via
Expand Down Expand Up @@ -136,6 +150,55 @@ per rule execution:
** _Field_: Leave blank
** _Threshold_: `10`

* To create a new EQL rule, select _Event Correlation_ and then:
.. Define which {es} indices the rule searches for alerts.
.. Add an {ref}/eql-syntax.html[EQL statement] used to detect alerts.
+
For example the following rule detects when `msxsl.exe` makes an outbound
network connection:

** _Index patterns_: `winlogbeat-*`
+
Winlogbeat ship Windows events to {es-sec}.
Comment on lines +160 to +162
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a convention for visually distinguishing an idea from its description? I think that's what these two lines are, but I (and presumably other readers) would benefit from some visual categorization/reinforcement here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rylnd: Since I'm taking this over from Ben, I'm not sure which two ideas we're trying to distinguish. Are you hoping for a short flow of how the EQL query from winlogbeat moves from the process to the network?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Donnater ! Sorry, I was unclear there. I think the content is good, my question was centered around formatting:

Creating_detection_rules___Elastic_Security_Solution__master____Elastic

I was having trouble parsing that text and I think it's because the rule parameter and its accompanying description aren't visually distinct. If that's standard formatting for something like this, though, I'm all for consistency 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Donnater aside from Ryland's comment around formatting, I think there is a typo in this line. I think it's supposed to say Winlogbeat ships as opposed to Winlogbeat ship.

** _EQL query_: `sequence by process.entity_id [process where event.type in ("start", "process_started") and process.name == "msxsl.exe"] [network where event.type == "connection" and process.name == "msxsl.exe" and network.direction == "outgoing"]`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to make this a multiline code block? It's a bit hard to read as it is. Something like

sequence by process.entity_id
  [process 
    where event.type in ("start", "process_started") 
    and process.name == "msxsl.exe"]
  [network 
    where event.type == "connection" 
    and process.name == "msxsl.exe"
    and network.direction == "outgoing"]

Not sure if that's the "correct" EQL formatting, but you get the idea 🤷‍♂️

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rylnd: I think so, let me explore this one.

+
Searches the `winlogbeat-*` indices for sequences of a `msxsl.exe` process start
event followed by an outbound network connection event that was started by the
`msxsl.exe` process.
+
[role="screenshot"]
image::images/eql-rule-query-example.png[]

* To create a threat-match rule that searches the specified threat indices for
field values, select _Threat Match_ and then fill in these fields:
.. _Index patterns_: The {es-sec} event indices on which the rules runs.
.. _Custom query_: The query and filters used to retrieve the required results from
the {es-sec} event indices. For example, if you only need to check
`destination.ip` event values, add `destination.ip : *`.
+
TIP: If you want the rule to check every field in the indices, use this
wildcard expression: `*:*`.
.. _Threat index patterns_: The threat indices containing field values for which
you want to generate alerts.
.. _Threat index query_: The query and filters used to filter the fields from
the threat indices.
.. _Threat Mapping_: Compares the values of the specified event and threat field
values. When the field values are identical, an alert is generated. To define
which field values are compared from the indices:
** _Field_: The field used for comparing values in the {es-sec} event
indices.
** _Threat index field_: The field used for comparing values in the threat
indices.
.. You can add `AND` and `OR` clauses to define when alerts are generated.
+
For example, to create a rule that generates alerts when `host.name` *and*
`destination.ip` field values in the `logs-*` or `packetbeat-*` {es-sec} indices
are identical to the corresponding field values in the `ip-threat-list` threat
index:
+
[role="screenshot"]
image::images/threat-math-rule-query-example.png[]

. Select the Timeline template used when you investigate an alert created by
the rule in Timeline (optional).
+
Expand Down