Skip to content

Commit 302b1c0

Browse files
nastasha-solomonmergify[bot]
nastasha-solomon
authored and
mergify[bot]
committed
[8.17] Document impact of using logsDB for security users (#6272)
* First draft * Removes serverless content * Minor change * Removes unnecessary words * Fixes casing for all mentions of logsdb ESS and Serverless * Update docs/detections/detections-logsdb-impact.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/detections/detections-logsdb-impact.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/serverless/rules/detections-logsdb-impact.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Update docs/serverless/rules/detections-logsdb-impact.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> * Adds licensing info --------- Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> (cherry picked from commit 61b1f1b) # Conflicts: # docs/serverless/rules/detection-engine-overview.asciidoc # docs/serverless/rules/detections-logsdb-impact.asciidoc
1 parent 49d476c commit 302b1c0

File tree

5 files changed

+285
-0
lines changed

5 files changed

+285
-0
lines changed

‎docs/detections/detection-engine-intro.asciidoc‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,3 +167,9 @@ and you should contact your {kib} administrator.
167167
NOTE: For *self-managed* {stack} deployments only, this message may be
168168
displayed when the <<detections-permissions, `xpack.security.enabled`>>
169169
setting is not enabled in the `elasticsearch.yml` file. For more information, refer to <<detections-on-prem-requirements>>.
170+
171+
[discrete]
172+
[[detections-logsdb-index-mode]]
173+
== Using logsdb index mode
174+
175+
To learn how your rules and alerts are affected by using the {ref}/logs-data-stream.html[logsdb index mode], refer to <<detections-logsdb-index-mode-impact>>.

‎docs/detections/detections-index.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@ include::detection-engine-intro.asciidoc[]
22

33
include::detections-req.asciidoc[leveloffset=+1]
44

5+
include::detections-logsdb-impact.asciidoc[leveloffset=+1]
6+
57
include::about-rules.asciidoc[]
68

79

‎docs/detections/detections-logsdb-impact.asciidoc‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
[[detections-logsdb-index-mode-impact]]
2+
= Using logsdb index mode with {elastic-sec}
3+
4+
NOTE: To use the {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`] feature, you must have the appropriate subscription. Refer to the subscription page for https://www.elastic.co/subscriptions/cloud[Elastic Cloud] and {subscriptions}[Elastic Stack/self-managed] for the breakdown of available features and their associated subscription tiers.
5+
6+
This topic explains the impact of using logsdb index mode with {elastic-sec}.
7+
8+
With logsdb index mode, the original `_source` field is not stored in the index but can be reconstructed using {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`].
9+
10+
When the `_source` is reconstructed, {ref}/mapping-source-field.html#synthetic-source-modifications[modifications] are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted.
11+
12+
Continue reading to find out how this affects specific {elastic-sec} components.
13+
14+
[discrete]
15+
[[logsdb-alerts]]
16+
== Alerts
17+
18+
When alerts are generated, the `_source` event is copied into the alert to retain the original data. When the logsdb index mode is applied, the `_source` event stored in the alert is reconstructed using synthetic `_source`.
19+
20+
If you're switching to use logsdb index mode, the `_source` field stored in the alert might look different in certain situations:
21+
22+
* {ref}/mapping-source-field.html#synthetic-source-modifications-leaf-arrays[Arrays can be reconstructed differently or deduplicated]
23+
* {ref}/mapping-source-field.html#synthetic-source-modifications-field-names[Field names]
24+
* `geo_point` data fields (refer to {ref}/mapping-source-field.html#synthetic-source-modifications-ranges[Representation of ranges] and {ref}/mapping-source-field.html#synthetic-source-precision-loss-for-point-types[Reduced precision of `geo_point` values] for more information)
25+
26+
Alerts generated by the following rule types could be affected:
27+
28+
* Custom query
29+
* Event correlation (non-sequence only)
30+
* Non-aggregate rule types (for example, {esql} rules that use non-aggregating queries)
31+
32+
Alerts that are generated by threshold, {ml}, and event correlation sequence rules are not affected since they do not contain copies of the original source.
33+
34+
[discrete]
35+
[[logsdb-rule-actions]]
36+
== Rule actions
37+
38+
While we do not recommend using `_source` for actions, in cases where the action relies on the `_source`, the same limitations and changes apply.
39+
40+
If you send alert notifications by enabling {kibana-ref}/alerting-getting-started.html#alerting-concepts-actions[actions] to the external systems that have workflows or automations based on fields formatted from the original source, they may be affected. In particular, this can happen when the fields used are arrays of objects.
41+
42+
We recommend checking and adjusting the rule actions using `_source` before switching to logsdb index mode.
43+
44+
[discrete]
45+
[[logsdb-runtime-fields]]
46+
== Runtime fields
47+
48+
Runtime fields that reference `_source` may be affected. Some runtime fields might not work and need to be adjusted. For example, if an event was indexed with the value of `agent.name` in the dot-notation form, it will be returned in the nested form and might not work.
49+
50+
The following is an example of accessing `_source` that works with the logsdb index mode enabled:
51+
52+
[source,console]
53+
----
54+
"source": """ emit(params._source.agent.name + "_____" + doc['agent.name'].value ); """
55+
"source": """ emit(params._source['agent']['name'] + "_____" + doc['agent.name'].value ); """
56+
"source": """ emit(field('agent.name').get(null) + "_____" + doc['agent.name'].value ); """
57+
"source": """ emit($('agent.name', null) + "_____" + doc['agent.name'].value ); """
58+
----
59+
60+
The following will not work with synthetic source (logsdb index mode enabled):
61+
62+
[source,console]
63+
----
64+
"source": """ emit(params._source['agent.name'] + "_____" + doc['agent.name'].value ); """
65+
----

‎docs/serverless/rules/detection-engine-overview.asciidoc‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
[[security-detection-engine-overview]]
2+
= Detection engine overview
3+
4+
// :description: Learn about the detection engine and its features.
5+
// :keywords: serverless, security, overview
6+
7+
8+
Use the detection engine to create and manage rules and view the alerts
9+
these rules create. Rules periodically search indices (such as `logs-*` and
10+
`filebeat-*`) for suspicious source events and create alerts when a rule's
11+
conditions are met. When an alert is created, its status is `Open`. To help
12+
track investigations, an alert's <<detection-alert-status,status>> can be set as
13+
`Open`, `Acknowledged`, or `Closed`.
14+
15+
[role="screenshot"]
16+
image::images/detection-engine-overview/-detections-alert-page.png[Alerts page]
17+
18+
In addition to creating <<security-rules-create,your own rules>>, enable
19+
<<load-prebuilt-rules,Elastic prebuilt rules>> to immediately start detecting
20+
suspicious activity. For detailed information on all the prebuilt rules, see the <<security-prebuilt-rules,Prebuilt rules reference>>. Once the prebuilt rules are loaded and
21+
running, <<security-tune-detection-signals,Tune detection rules>> and <<security-add-exceptions,Add and manage exceptions>> explain
22+
how to modify the rules to reduce false positives and get a better set of
23+
actionable alerts. You can also use exceptions and value lists when creating or
24+
modifying your own rules.
25+
26+
There are two special prebuilt rules you need to know about:
27+
28+
// Links to prebuilt rule pages temporarily removed for initial serverless docs.
29+
30+
* **Endpoint Security**:
31+
Automatically creates an alert from all incoming Elastic Endpoint alerts. To
32+
receive Elastic Endpoint alerts, you must install the Endpoint agent on your
33+
hosts (see <<security-install-edr,Install and configure the {elastic-defend} integration>>).
34+
+
35+
When this rule is enabled, the following Endpoint events are displayed as
36+
detection alerts:
37+
+
38+
** Malware Prevention Alert
39+
** Malware Detection Alert
40+
+
41+
[NOTE]
42+
====
43+
When you load the prebuilt rules, this is the only rule that is enabled
44+
by default.
45+
====
46+
47+
// Links to prebuilt rule pages temporarily removed for initial serverless docs.
48+
49+
* **External Alerts**: Automatically creates an alert for
50+
all incoming third-party system alerts (for example, Suricata alerts).
51+
52+
If you want to receive notifications via external systems, such as Slack or
53+
email, when alerts are created, use the {kibana-ref}/alerting-getting-started.html[Alerting and Actions] framework.
54+
55+
After rules have started running, you can monitor their executions to verify
56+
they are functioning correctly, as well as view, manage, and troubleshoot
57+
alerts (see <<security-alerts-manage,Manage detection alerts>> and <<security-alerts-ui-monitor,Monitor and troubleshoot rule executions>>).
58+
59+
You can create and manage rules and alerts via the UI or the {security-guide}/rule-api-overview.html[Detections API].
60+
61+
// Link to classic docs until serverless API docs are available.
62+
63+
[IMPORTANT]
64+
====
65+
To make sure you can access Detections and manage rules, see
66+
<<security-detections-requirements,Detections prerequisites and requirements>>.
67+
====
68+
69+
[discrete]
70+
[[support-indicator-rules]]
71+
== Limited support for indicator match rules
72+
73+
Indicator match rules provide a powerful capability to search your security data; however, their queries can consume significant deployment resources. When creating an <<create-indicator-rule,indicator match rule>>, we recommend limiting the time range of the indicator index query to the minimum period necessary for the desired rule coverage. For example, the default indicator index query `@timestamp > "now-30d/d"` searches specified indicator indices for indicators ingested during the past 30 days and rounds the query start time down to the nearest day (resolves to UTC `00:00:00`). Without this limitation, the rule will include all of the indicators in your indicator indices, which may extend the time it takes for the indicator index query to complete.
74+
75+
In addition, indicator match rules with an additional look-back time value greater than 24 hours are not supported.
76+
77+
[discrete]
78+
[[detections-permissions]]
79+
== Detections configuration and prerequisites
80+
81+
<<security-detections-requirements,Detections requirements>> provides detailed information on all the
82+
permissions required to initiate and use the Detections feature.
83+
84+
[discrete]
85+
[[malware-prevention]]
86+
== Malware prevention
87+
88+
Malware, short for malicious software, is any software program designed to damage or execute unauthorized actions on a
89+
computer system. Examples of malware include viruses, worms, Trojan horses, adware, scareware, and spyware. Some
90+
malware, such as viruses, can severely damage a computer's hard drive by deleting files or directory information. Other
91+
malware, such as spyware, can obtain user data without their knowledge.
92+
93+
Malware may be stealthy and appear as legitimate executable code, scripts, active content, and other software. It is also
94+
often embedded in non-malicious files, non-suspicious websites, and standard programs — sometimes making the root
95+
source difficult to identify. If infected and not resolved promptly, malware can cause irreparable damage to a computer
96+
network.
97+
98+
For information on how to enable malware protection on your host, see <<malware-protection,Malware Protection>>.
99+
100+
[discrete]
101+
[[machine-learning-model]]
102+
=== Machine learning model
103+
104+
To determine if a file is malicious or benign, a machine learning model looks for static attributes of files (without executing
105+
the file) that include file structure, layout, and content. This includes information such as file header data, imports, exports,
106+
section names, and file size. These attributes are extracted from millions of benign and malicious file samples, which then
107+
are passed to a machine-learning algorithm that distinguishes a benign file from a malicious one. The machine learning
108+
model is updated as new data is procured and analyzed.
109+
110+
[discrete]
111+
[[security-detection-engine-overview-threshold]]
112+
=== Threshold
113+
114+
A malware threshold determines the action the agent should take if malware is detected. The Elastic Agent uses a recommended threshold level that generates a balanced number of alerts with a low probability of undetected malware. This threshold also minimizes the number of false positive alerts.
115+
116+
[discrete]
117+
[[ransomware-prevention]]
118+
== Ransomware prevention
119+
120+
Ransomware is computer malware that installs discreetly on a user's computer and encrypts data until a specified amount of money (ransom) is paid. Ransomware is usually similar to other malware in its delivery and execution, infecting systems
121+
through spear-phishing or drive-by downloads. If not resolved immediately, ransomware can cause irreparable damage to an entire computer network.
122+
123+
Behavioral ransomware prevention on the Elastic Endpoint detects and stops ransomware attacks on Windows systems by analyzing data from low-level system processes, and is effective across an array of widespread ransomware families — including those targeting the system’s master boot record.
124+
125+
For information on how to enable ransomware protection on your host, see <<ransomware-protection,Ransomware protection>>.
126+
127+
[discrete]
128+
[[security-detection-engine-overview-resolve-ui-error-messages]]
129+
=== Resolve UI error messages
130+
131+
Depending on your user role privileges and whether detection system indices have already been created, you might get one of these error messages when you
132+
open the **Alerts** or **Rules** page:
133+
134+
* **`Let’s set up your detection engine`**
135+
+
136+
If you get this message, a user with specific privileges must visit the
137+
**Alerts** or **Rules** page before you can view detection alerts and rules.
138+
Refer to <<enable-detections-ui,Enable and access detections>> for a list of all the requirements.
139+
* **`Detection engine permissions required`**
140+
+
141+
If you get this message, you do not have the
142+
<<detections-permissions,required privileges>> to view the **Detections** feature,
143+
and you should contact your project administrator.
144+
145+
[discrete]
146+
[[detections-logsdb-index-mode]]
147+
== Using logsdb index mode
148+
149+
logsdb is enabled by default for Elastic serverless. Refer to <<detections-logsdb-index-mode-impact>> to learn more.

‎docs/serverless/rules/detections-logsdb-impact.asciidoc‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
[[detections-logsdb-index-mode-impact]]
2+
= Using logsdb index mode with {sec-serverless}
3+
4+
Logsdb is enabled by default for {serverless-full}. This topic explains the impact of using logsdb index mode with {sec-serverless}.
5+
6+
With logsdb index mode, the original `_source` field is not stored in the index but can be reconstructed using {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`].
7+
8+
When the `_source` is reconstructed, {ref}/mapping-source-field.html#synthetic-source-modifications[modifications] are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted.
9+
10+
Continue reading to find out how this affects specific {sec-serverless} components.
11+
12+
[discrete]
13+
[[logsdb-alerts]]
14+
== Alerts
15+
16+
When alerts are generated, the `_source` event is copied into the alert to retain the original data. When the logsdb index mode is applied, the `_source` event stored in the alert is reconstructed using synthetic `_source`.
17+
18+
If you're switching to use logsdb index mode, the `_source` field stored in the alert might look different in certain situations:
19+
20+
* {ref}/mapping-source-field.html#synthetic-source-modifications-leaf-arrays[Arrays can be reconstructed differently or deduplicated]
21+
* {ref}/mapping-source-field.html#synthetic-source-modifications-field-names[Field names]
22+
* `geo_point` data fields (refer to {ref}/mapping-source-field.html#synthetic-source-modifications-ranges[Representation of ranges] and {ref}/mapping-source-field.html#synthetic-source-precision-loss-for-point-types[Reduced precision of `geo_point` values] for more information)
23+
24+
Alerts generated by the following rule types could be affected:
25+
26+
* Custom query
27+
* Event correlation (non-sequence only)
28+
* Non-aggregate rule types (for example, {esql} rules that use non-aggregating queries)
29+
30+
Alerts that are generated by threshold, {ml}, and event correlation sequence rules are not affected since they do not contain copies of the original source.
31+
32+
[discrete]
33+
[[logsdb-rule-actions]]
34+
== Rule actions
35+
36+
While we do not recommend using `_source` for actions, in cases where the action relies on the `_source`, the same limitations and changes apply.
37+
38+
If you send alert notifications by enabling {kibana-ref}/alerting-getting-started.html#alerting-concepts-actions[actions] to the external systems that have workflows or automations based on fields formatted from the original source, they may be affected. In particular, this can happen when the fields used are arrays of objects.
39+
40+
We recommend checking and adjusting the rule actions using `_source` before switching to logsdb index mode.
41+
42+
[discrete]
43+
[[logsdb-runtime-fields]]
44+
== Runtime fields
45+
46+
Runtime fields that reference `_source` may be affected. Some runtime fields might not work and need to be adjusted. For example, if an event was indexed with the value of `agent.name` in the dot-notation form, it will be returned in the nested form and might not work.
47+
48+
The following is an example of accessing `_source` that works with the logsdb index mode enabled:
49+
50+
[source,console]
51+
----
52+
"source": """ emit(params._source.agent.name + "_____" + doc['agent.name'].value ); """
53+
"source": """ emit(params._source['agent']['name'] + "_____" + doc['agent.name'].value ); """
54+
"source": """ emit(field('agent.name').get(null) + "_____" + doc['agent.name'].value ); """
55+
"source": """ emit($('agent.name', null) + "_____" + doc['agent.name'].value ); """
56+
----
57+
58+
The following will not work with synthetic source (logsdb index mode enabled):
59+
60+
[source,console]
61+
----
62+
"source": """ emit(params._source['agent.name'] + "_____" + doc['agent.name'].value ); """
63+
----

0 commit comments

Comments
 (0)