[Lateral Movement Detection] Add dashboard and release to production #4974

sodhikirti07 · 2023-01-11T22:19:27Z

What does this PR do?

This PR adds the following enhancements in the Lateral Movement Detection package:

Dashboard
Updated README with instructions to add dashboard
Updated changelog and manifest yml files: added a new condition property for premium subscription, changed package version for production release and added license.
Added ML tags and bumped version in security rules

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

https://github.com/elastic/security-ml/issues/203

Screenshots

Refer to below comment for screenshots!

sodhikirti07 · 2023-01-11T22:26:30Z

Lateral Movement Detection package updated for version 0.0.2 with dashboard asset:

Anomaly detection jobs created and started:

Detection rules enabled and started without warnings:

ML tag added to the detection rules using machine learning jobs:

Dashboard asset installed and available without any errors:

packages/lmd/kibana/dashboard/lmd-17fea180-8c4c-11ed-bb03-41a73f349362.json

szeitlin

Looks great! Ship it!

Great work getting this done so fast! 🎉

elasticmachine · 2023-01-11T22:32:18Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-20T16:24:01.852+0000
Duration: 15 min 43 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1
Skipped	0
Total	1

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

…in dashboard

susan-shu-c

🚀

sodhikirti07 · 2023-01-18T14:48:49Z

Screenshot of dashboard with data:

packages/lmd/manifest.yml

peteharverson · 2023-01-18T17:28:53Z

packages/lmd/docs/README.md

+Before going to the dashboard, make sure you have the following settings configured in your Kibana. 
+1. You have started the above anomaly detection jobs.
+2. After enabling the jobs, go to **Management > Stack Management > Kibana > Data Views**. 
+3. Click on **Create data view** button and create a data view with an **Index pattern** of **.ml-anomalies***. Make sure you assign the same value to **Name** and **Custom data view ID** under **Show advanced settings**.


As discussed with @sodhikirti07, note that anyone viewing the dashboard will also need read access to the .ml-anomalies* index to be able to see the visualizations which query that index directly. To create the data view you also need to enable Allow hidden and system indices inside the Advanced settings.

Thank you for catching this. I've made the changes.

andrewkroh · 2023-01-18T19:35:49Z

packages/lmd/manifest.yml

@@ -2,7 +2,8 @@ format_version: 2.2.0
 name: lmd
 title: "Lateral Movement Detection"
 version: 1.0.0
-license: "Elastic-2.0"
+source:
+  license: "Elastic-2.0"
 description: "ML package to detect lateral movement based on file transfer activity. Requires a Platinum subscription."


The package-spec [^1] appears to allow you to codify this with:

conditions: elastic: subscription: platinum

I'm not sure what the behavior is, but thought I would mention it.

https://github.com/elastic/package-spec/blob/60560379ad6ed5ffa6fc3beb13ec22ad89bdddc4/spec/integration/manifest.spec.yml#L42-L59

ajosh0504 · 2023-01-18T19:21:37Z

packages/lmd/docs/README.md

+
+The **Lateral Movement Detection Dashboard** is available under **Analytics > Dashboard**. This dashboard gives an overview of anomalies triggered for the lateral movement detection package.
+
+Before going to the dashboard, make sure you have the following settings configured in your Kibana. 


Suggested change

Before going to the dashboard, make sure you have the following settings configured in your Kibana.

For the dashboard to work as expected, the following settings need to be configured in Kibana.

ajosh0504 · 2023-01-18T19:26:21Z

packages/lmd/docs/README.md

+1. You have started the above anomaly detection jobs.
+2. You have **read** access to **.ml-anomalies*** index or are assigned **machine_learning_user** role. For more information on roles, please refer to [Built-in roles in Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-roles.html).
+3. After enabling the jobs, go to **Management > Stack Management > Kibana > Data Views**. 
+4. Click on **Create data view** button and create a data view with an **Index pattern** of **.ml-anomalies***. Make sure you assign the same value to **Name** and **Custom data view ID** under **Show advanced settings**. You also need to enable **Allow hidden and system indices** inside the **Advanced settings**.


Allow hidden and system indices should happen before they can proceed to naming the pattern, so maybe that should be the first sentence in the bullet.

Also maybe explicitly mention what the Name and Data View ID must be, since I'm guessing the dashboard is relying on a specific ID.

ajosh0504 · 2023-01-18T19:34:41Z

packages/lmd/kibana/dashboard/lmd-17fea180-8c4c-11ed-bb03-41a73f349362.json

+    },
+    "references":  [
+    {
+      "id": ".ml-anomalies*",


Consider making this index pattern more specific. I believe the job results go to .ml-anomalies-shared by default, unless you're specifying custom indices.

Thanks for pointing this out! Changed the dashboard to include .ml-anomalies-shared index-pattern.

packages/lmd/kibana/security_rule/2577e749-ed63-4b58-ac32-4f8a94d73e60.json

packages/lmd/kibana/security_rule/282cd7e6-9d51-421e-93b4-ebc2d422eb44.json

ajosh0504 · 2023-01-18T19:36:46Z

packages/lmd/kibana/security_rule/2577e749-ed63-4b58-ac32-4f8a94d73e60.json

@@ -16,7 +16,8 @@
        "tags": [
            "Elastic",
            "Lateral Movement Detection",
-            "host"
+            "host",
+            "ML"
        ],
        "type": "machine_learning",
        "version": 1


Bump the version number since you're updating the rule.

packages/lmd/kibana/security_rule/aa00d5db-7732-440a-97fc-8f9a3dc5314b.json

packages/lmd/kibana/security_rule/e5ebb55f-97f0-4815-b05b-59b223b1d4df.json

sodhikirti07 · 2023-01-19T15:37:42Z

@andrewkroh Thanks for mentioning this. Added the platinum subscription condition in manifest.yml and removed it from package description. @ajosh0504 @susan-shu-c We should do it for other packages as well.

andrewkroh

manifest.yml changes LGTM,

szeitlin

lgtm!

packages/lmd/docs/README.md

szeitlin · 2023-01-19T16:37:00Z

packages/lmd/manifest.yml

+version: 1.0.0
+source:
+  license: "Elastic-2.0"
+description: "ML package to detect lateral movement based on file transfer activity."


I see Requires a Platinum subscription got removed <-- is this no longer true?

It is true for our package but I added it under condition property in manifest.yml. So, it will appear on right hand side of the package's main page (see the screenshot above).

Thanks, I dislike how github doesn't have a way to insert screenshots into the code review view, so I was just looking at the code when I wrote that.

FWIW, in the future this is the kind of thing you could put in the description of the PR (it's ok to update that as you go along).

Co-authored-by: Sam Zeitlin <samantha.zeitlin@elastic.co>

droberts195 · 2023-01-19T17:10:57Z

packages/lmd/docs/README.md

+
+For the dashboard to work as expected, the following settings need to be configured in Kibana. 
+1. You have started the above anomaly detection jobs.
+2. You have **read** access to **.ml-anomalies*** index or are assigned the **machine_learning_user** role. For more information on roles, please refer to [Built-in roles in Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-roles.html).


Directly granting access to machine learning results indices has implications for using spaces to control ML job visibility. I think this should be made clear. Something along the lines of this:

Suggested change

2. You have **read** access to **.ml-anomalies*** index or are assigned the **machine_learning_user** role. For more information on roles, please refer to [Built-in roles in Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-roles.html).

2. You have **read** access to **.ml-anomalies*** index or are assigned the **machine_learning_user** role. For more information on roles, please refer to [Built-in roles in Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-roles.html). Please be aware that a user who has access to the underlying machine learning results indices can see the results of _all_ jobs in _all_ spaces. Do not grant these permissions if you use Kibana spaces to control which users can see which machine learning results. For more information on machine learning privileges see [here](https://www.elastic.co/guide/en/machine-learning/current/setup.html#setup-privileges).

ajosh0504

Looking great! Nice work! 🚢

packages/lmd/docs/README.md

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

droberts195

The ML permissions documentation LGTM.

elasticmachine · 2023-01-23T14:11:36Z

Package lmd - 1.0.0 containing this change is available at https://epr.elastic.co/search?package=lmd

sodhikirti07 added 2 commits January 11, 2023 16:45

Added dashboards,ML tag,changed version and release to ga

6f34aba

Changed changelog and manifest yml

8d91f0d

sodhikirti07 requested a review from a team as a code owner January 11, 2023 22:19

Updated changelog with PR link

a110819

sodhikirti07 requested review from susan-shu-c, ajosh0504 and szeitlin January 11, 2023 22:29

sodhikirti07 commented Jan 11, 2023

View reviewed changes

packages/lmd/kibana/dashboard/lmd-17fea180-8c4c-11ed-bb03-41a73f349362.json Outdated Show resolved Hide resolved

szeitlin approved these changes Jan 11, 2023

View reviewed changes

Changed Kibana version in manifest.yml corresponding to version used …

21c3af6

…in dashboard

susan-shu-c approved these changes Jan 13, 2023

View reviewed changes

sodhikirti07 requested review from sophiec20 and stevedodson January 16, 2023 18:05

andrewkroh added the Integration:lmd Lateral Movement Detection label Jan 18, 2023

andrewkroh changed the title ~~Add dashboard and release Lateral Movement Detection package in production~~ [Lateral Movement Detection] Add dashboard and release to production Jan 18, 2023

andrewkroh reviewed Jan 18, 2023

View reviewed changes

packages/lmd/manifest.yml Outdated Show resolved Hide resolved

packages/lmd/manifest.yml Outdated Show resolved Hide resolved

peteharverson reviewed Jan 18, 2023

View reviewed changes

sodhikirti07 added 4 commits January 18, 2023 12:42

Added read access condition to .ml-anomalies index

bf5e1b2

Remove release tag, changed format and package version

f7e7349

Changed version in changelog

047c2aa

Added source to license

50a31db

andrewkroh reviewed Jan 18, 2023

View reviewed changes

ajosh0504 suggested changes Jan 18, 2023

View reviewed changes

sodhikirti07 added 4 commits January 19, 2023 10:04

Updated README

64e44cc

Changed the index pattern to .ml-anomalies-shared

4a2d56d

Bumped rule versions

460fe46

Added elastic subscription condition to manifest

d05cff5

Removed require platinum subscription from package description

0ecdfd9

sodhikirti07 requested a review from a team as a code owner January 19, 2023 15:33

sodhikirti07 requested review from peteharverson, ajosh0504 and andrewkroh January 19, 2023 15:40

andrewkroh approved these changes Jan 19, 2023

View reviewed changes

szeitlin approved these changes Jan 19, 2023

View reviewed changes

Update packages/lmd/docs/README.md

405ac8e

Co-authored-by: Sam Zeitlin <samantha.zeitlin@elastic.co>

droberts195 reviewed Jan 19, 2023

View reviewed changes

ajosh0504 approved these changes Jan 19, 2023

View reviewed changes

packages/lmd/docs/README.md Outdated Show resolved Hide resolved

sodhikirti07 and others added 4 commits January 19, 2023 12:13

Update packages/lmd/docs/README.md

fc851e6

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

Update README.md

40dedb6

Update README.md

0b346bd

Update README.md

368844a

sodhikirti07 requested a review from droberts195 January 19, 2023 19:22

Update README.md

6895ccf

droberts195 approved these changes Jan 23, 2023

View reviewed changes

sodhikirti07 merged commit b87df11 into main Jan 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Lateral Movement Detection] Add dashboard and release to production #4974

[Lateral Movement Detection] Add dashboard and release to production #4974

sodhikirti07 commented Jan 11, 2023 •

edited

Loading

sodhikirti07 commented Jan 11, 2023

szeitlin left a comment •

edited

Loading

elasticmachine commented Jan 11, 2023 •

edited

Loading

Build stats

Test stats 🧪

susan-shu-c left a comment

sodhikirti07 commented Jan 18, 2023 •

edited

Loading

peteharverson Jan 18, 2023

sodhikirti07 Jan 18, 2023

andrewkroh Jan 18, 2023

sodhikirti07 Jan 19, 2023

ajosh0504 Jan 18, 2023

sodhikirti07 Jan 19, 2023

ajosh0504 Jan 18, 2023

sodhikirti07 Jan 19, 2023

ajosh0504 Jan 18, 2023

sodhikirti07 Jan 19, 2023

ajosh0504 Jan 18, 2023

sodhikirti07 Jan 19, 2023

sodhikirti07 commented Jan 19, 2023

andrewkroh left a comment

szeitlin left a comment

szeitlin Jan 19, 2023

sodhikirti07 Jan 19, 2023

szeitlin Jan 19, 2023

szeitlin Jan 19, 2023

droberts195 Jan 19, 2023

ajosh0504 left a comment

droberts195 left a comment

elasticmachine commented Jan 23, 2023


		The Lateral Movement Detection Dashboard is available under Analytics > Dashboard. This dashboard gives an overview of anomalies triggered for the lateral movement detection package.

		Before going to the dashboard, make sure you have the following settings configured in your Kibana.

	Before going to the dashboard, make sure you have the following settings configured in your Kibana.
	For the dashboard to work as expected, the following settings need to be configured in Kibana.

[Lateral Movement Detection] Add dashboard and release to production #4974

[Lateral Movement Detection] Add dashboard and release to production #4974

Conversation

sodhikirti07 commented Jan 11, 2023 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

sodhikirti07 commented Jan 11, 2023

szeitlin left a comment • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Jan 11, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

susan-shu-c left a comment

Choose a reason for hiding this comment

sodhikirti07 commented Jan 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sodhikirti07 commented Jan 19, 2023

andrewkroh left a comment

Choose a reason for hiding this comment

szeitlin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajosh0504 left a comment

Choose a reason for hiding this comment

droberts195 left a comment

Choose a reason for hiding this comment

elasticmachine commented Jan 23, 2023

sodhikirti07 commented Jan 11, 2023 •

edited

Loading

szeitlin left a comment •

edited

Loading

elasticmachine commented Jan 11, 2023 •

edited

Loading

sodhikirti07 commented Jan 18, 2023 •

edited

Loading