Skip to content

Security machine learning rule documentation improvements #2240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Jul 25, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
ac60c1d
restructure ad page, add tips for linking between these pages
jmcarlock Jul 23, 2025
38cafb9
fix links
jmcarlock Jul 23, 2025
062ba28
fix formatting and link
jmcarlock Jul 23, 2025
cecfcd1
wording improvment
jmcarlock Jul 23, 2025
9de21e3
simplify wording
jmcarlock Jul 23, 2025
a42cd0c
fix link
jmcarlock Jul 23, 2025
d91b845
fix link
jmcarlock Jul 23, 2025
c3e2295
formatting improvement
jmcarlock Jul 23, 2025
e95d67e
fix description
jmcarlock Jul 23, 2025
e204ae3
Update solutions/security/advanced-entity-analytics/anomaly-detection.md
jmcarlock Jul 24, 2025
f97cff2
Update solutions/security/detect-and-alert/create-detection-rule.md
jmcarlock Jul 24, 2025
bd150fd
Update solutions/security/advanced-entity-analytics/anomaly-detection.md
jmcarlock Jul 24, 2025
9e7c925
Update solutions/security/advanced-entity-analytics/anomaly-detection.md
jmcarlock Jul 24, 2025
9f716aa
Update solutions/security/advanced-entity-analytics/anomaly-detection.md
jmcarlock Jul 24, 2025
6819c4f
Update solutions/security/detect-and-alert/create-detection-rule.md
jmcarlock Jul 24, 2025
3c9b322
fix link
jmcarlock Jul 24, 2025
bc6ad79
add screenshots
jmcarlock Jul 24, 2025
a261360
improve image alt
jmcarlock Jul 24, 2025
6ea9f4f
fix tab spacing
jmcarlock Jul 24, 2025
aa387ca
fix how to select entity details
jmcarlock Jul 24, 2025
1dc0dda
add annotations to images
jmcarlock Jul 24, 2025
fabd6f7
improve wording
jmcarlock Jul 24, 2025
f30078a
move note, add correct image
jmcarlock Jul 24, 2025
add0d24
crop for clarity
jmcarlock Jul 24, 2025
f267cc9
move note
jmcarlock Jul 24, 2025
ecd6a37
add note on managing alerts
jmcarlock Jul 24, 2025
419259f
fix description of ootb ml jobs page
jmcarlock Jul 24, 2025
69a47db
minor tweaks
natasha-moore-elastic Jul 25, 2025
cdd0d6c
Merge branch 'main' into security-ml-rules-improvements
jmcarlock Jul 25, 2025
18b28dd
Merge branch 'main' into security-ml-rules-improvements
jmcarlock Jul 25, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added solutions/images/security-host-anomalies.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,16 @@ products:
# Anomaly detection
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it would make sense to link to it from the Data tables sections on those pages.



::::{note}
[{{ml-cap}}](/explore-analyze/machine-learning/anomaly-detection.md) functionality is available when you have the appropriate role, subscription, are using a [cloud deployment](https://cloud.elastic.co/registration?page=docs&placement=docs-body), or are testing out a **Free Trial**. Refer to [Machine learning job and rule requirements](/solutions/security/advanced-entity-analytics/machine-learning-job-rule-requirements.md) for more information.
::::

You can view the details of detected anomalies within the `Anomalies` table widget shown on the Hosts, Network, and associated details pages, or even narrow to the specific date range of an anomaly from the `Max anomaly score by job` field in the overview of the details pages for hosts and IPs. These interfaces also offer the ability to drag and drop details of the anomaly to Timeline, such as the `Entity` itself, or any of the associated `Influencers`.
Anomaly detection jobs identify anomalous events or patterns in your data. In a security context, they’re typically used with detection rules to trigger alerts when behavior deviates from baseline activity.


::::{tip}
Refer to [{{ml-cap}}: Anomaly detection](/explore-analyze/machine-learning/anomaly-detection.md) and [About detection rules](/solutions/security/detect-and-alert/about-detection-rules.md) for more background.
::::


## Manage {{ml}} jobs [manage-jobs]
Expand Down Expand Up @@ -47,6 +54,10 @@ You can also check the status of {{ml}} detection rules, and start or stop their
:screenshot:
:::

::::{tip}
* For instructions on creating {{ml}} rules, refer to [Create a machine learning rule](/solutions/security/detect-and-alert/create-detection-rule.md#create-ml-rule).
* Alerts generated by {{ml}} rules are displayed on the **Alerts** page. For more information, refer to [Manage detection alerts](/solutions/security/detect-and-alert/manage-detection-alerts.md).
::::


### Prebuilt jobs [included-jobs]
Expand All @@ -63,18 +74,47 @@ Or

* You install one or more of the [Advanced Analytics integrations](/solutions/security/advanced-entity-analytics/behavioral-detection-use-cases.md#ml-integrations).

[](/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md) describes all available {{ml}} jobs and lists which ECS fields are required on your hosts when you are not using {{beats}} or the {{agent}} to ship your data. For information on tuning anomaly results to reduce the number of false positives, see [Optimizing anomaly results](/solutions/security/advanced-entity-analytics/optimizing-anomaly-results.md).
[](/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md) describes all available {{ml}} jobs and lists their requirements. For information on tuning anomaly results to reduce the number of false positives, see [Optimizing anomaly results](/solutions/security/advanced-entity-analytics/optimizing-anomaly-results.md).

::::{note}
Machine learning jobs look back and analyze two weeks of historical data prior to the time they are enabled. After jobs are enabled, they continuously analyze incoming data. When jobs are stopped and restarted within the two-week time frame, previously analyzed data is not processed again.
::::



## View detected anomalies [view-anomalies]

To view the `Anomalies` table widget and `Max Anomaly Score By Job` details, the user must have the `machine_learning_admin` or `machine_learning_user` role.
View details of detected anomalies in the **Anomalies** data tables, available on the **Hosts**, **Network**, or **Users** pages. You can access these pages from the navigation menu or with the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).

::::{note}
To view the **Anomalies** table and **Max Anomaly Score By Job** details, the user must have the `machine_learning_admin` or `machine_learning_user` role.
::::

:::{image} /solutions/images/security-host-anomalies.png
:alt: Host anomalies
:screenshot:
:::

::::{note}
To adjust the `score` threshold that determines which anomalies are shown, you can modify the `securitySolution:defaultAnomalyScore` [advanced setting](/solutions/security/get-started/configure-advanced-settings.md).
::::

In the **Anomalies** table, you can add entity details, like the entity or any associated influencers, into Timeline.

:::{image} /solutions/images/security-host-anomaly-add-to-timeline.png
:alt: Host anomalies
:screenshot:
:::

When you click into a specific host, IP, or user name (depending on the page), you can narrow the time range to a specific anomaly. To do this, click the info icon next to a maximum anomaly score and select **Narrow to this date range**.

:::{image} /solutions/images/security-host-max-anomaly-score-by-job.png
:alt: Filter by Max anomaly score by job
:screenshot:
:::

On this page, you can also add fields to Timeline by hovering over a field name and selecting the Add to timeline icon.

:::{image} /solutions/images/security-host-add-to-timeline.png
:alt: Filter by Max anomaly score by job
:screenshot:
:::
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,10 @@ To create or edit {{ml}} rules, you need:
* The appropriate [{{stack}} subscription](https://www.elastic.co/pricing) or [{{serverless-short}} project tier](../../../deploy-manage/deploy/elastic-cloud/project-settings.md).
* The [`machine_learning_admin`](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) in {{stack}} or the appropriate [user role](/deploy-manage/users-roles/cloud-organization/user-roles.md) in {{serverless-short}}.
* The selected {{ml}} job to be running for the rule to function correctly.
::::

::::{tip}
For an overview of using {{ml}} with {{elastic-sec}}, refer to [Anomaly detection](/solutions/security/advanced-entity-analytics/anomaly-detection.md).
::::


Expand Down Expand Up @@ -120,6 +123,9 @@ To create or edit {{ml}} rules, you need:

5. Click **Continue** to [configure basic rule settings](/solutions/security/detect-and-alert/create-detection-rule.md#rule-ui-basic-params).

::::{tip}
To filter noisy {{ml}} rules, use [rule exceptions](/solutions/security/detect-and-alert/rule-exceptions.md).
::::

## Create a threshold rule [create-threshold-rule]

Expand Down
2 changes: 1 addition & 1 deletion solutions/security/explore/hosts-page.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Beneath the KPI charts are data tables, categorized by individual tabs, which ar
* **Events**: All host events. To display alerts received from external monitoring tools, scroll down to the Events table and select **Show only external alerts** on the right.
* **All hosts**: High-level host details.
* **Uncommon processes**: Uncommon processes running on hosts.
* **Anomalies**: Anomalies discovered by machine learning jobs.
* **Anomalies**: Anomalies discovered by [{{ml}} jobs](/solutions/security/advanced-entity-analytics/anomaly-detection.md).
* **Host risk**: The latest recorded host risk score for each host, and its host risk classification. In {{stack}}, this feature requires a [Platinum subscription](https://www.elastic.co/pricing) or higher. In serverless, this feature requires the Security Analytics Complete [project feature](/deploy-manage/deploy/elastic-cloud/project-settings.md). Click **Enable** on the **Host risk** tab to get started. To learn more, refer to our [entity risk scoring documentation](/solutions/security/advanced-entity-analytics/entity-risk-scoring.md).
* **Sessions**: Linux process events that you can open in [Session View](/solutions/security/investigate/session-view.md), an investigation tool that allows you to examine Linux process data at a hierarchal level.

Expand Down
2 changes: 1 addition & 1 deletion solutions/security/explore/users-page.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Beneath the KPI charts are data tables, which are useful for viewing and investi
* **Events**: Ingested events that contain the `user.name` field. You can stack by the `event.action`, `event.dataset`, or `event.module` field. To display alerts received from external monitoring tools, scroll down to the Events table and select **Show only external alerts** on the right.
* **All users**: A chronological list of unique user names, when they were last active, and the associated domains.
* **Authentications**: A chronological list of user authentication events and associated details, such as the number of successes and failures, and the host name of the last successful destination.
* **Anomalies**: Unusual activity discovered by machine learning jobs that contain user data.
* **Anomalies**: Unusual activity discovered by [{{ml}} jobs](/solutions/security/advanced-entity-analytics/anomaly-detection.md) that contain user data.
* **User risk**: The latest recorded user risk score for each user, and its user risk classification. In {{stack}}, this feature requires a [Platinum subscription](https://www.elastic.co/pricing) or higher. In serverless, this feature requires the Security Analytics Complete [project feature](/deploy-manage/deploy/elastic-cloud/project-settings.md). Click **Enable** on the **User risk** tab to get started. To learn more, refer to our [entity risk scoring documentation](/solutions/security/advanced-entity-analytics/entity-risk-scoring.md).

The Events table includes inline actions and several customization options. To learn more about what you can do with the data in these tables, refer to [*Manage detection alerts*](/solutions/security/detect-and-alert/manage-detection-alerts.md).
Expand Down
Loading