[m365_defender] Add support for Incident data-stream with new Security Graph API #4435

vinit-chauhan · 2022-10-10T14:02:09Z

What does this PR do?

Added a data stream.
Added data collection logic to the data stream.
Added the ingest pipeline for the data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added test for pipeline for the data stream.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.16.0).

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/m365_defender directory.
Run the following command to run tests.

elastic-package test

Related issues

Screenshots

elasticmachine · 2022-10-10T14:04:51Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-10-10T14:18:55Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2022-10-10T14:19:07Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-11-08T05:02:40.310+0000
Duration: 16 min 53 sec

Test stats 🧪

Test	Results
Failed	0
Passed	15
Skipped	0
Total	15

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-10-10T14:19:17Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`2/2`)	💚 2.698
Classes	100.0% (`2/2`)	💚 2.698
Methods	96.667% (`29/30`)	👍 6.094
Lines	93.776% (`2275/2426`)	👍 2.197
Conditionals	100.0% (`0/0`)	💚

efd6 · 2022-10-11T22:10:56Z

packages/m365_defender/data_stream/alert/fields/base-fields.yml

@@ -0,0 +1,20 @@
+- name: data_stream.type


Move the fields to ecs.yml and use external definitions where possible.

data_stream.type

data_stream.dataset

data_stream.namespace

@timestamp

Hey @efd6 ,
Also, there is an agent.yml which contains an ECS field description. So, should we remove that file and provide those field descriptions in ecs.yml?

Yeah, that has confused me in the past. @P1llus WDYT?

base-fields.yml is the correct approach, all integrations currently have those, so we shouldn't move them out of that file, unless there is any specific plans to do this on all integrations.

agent.yml needs to stay and usually have all the agent fields populated by the agent, so it should be fairly large. If its very small for some reason then feel free to copy it from any of the other integrations.

efd6 · 2022-10-11T22:19:28Z

packages/m365_defender/data_stream/alert/elasticsearch/ingest_pipeline/default.yml

+        if (!eventType.isEmpty()) {
+          ctx.event.type = eventType;
+        }
+        if (!eventCategory.isEmpty()) {
+          ctx.event.category = eventCategory;
+        }


I've found that this can be a cause of test failures due to changes in iteration order in the HashSet type. See #4296. The fix is to copy the elements to an array list and then sort it (this is done in the linked PR).

efd6 · 2022-10-11T22:29:23Z

packages/m365_defender/data_stream/alert/elasticsearch/ingest_pipeline/default.yml

+        if (!cloudProvider.isEmpty()) {
+          ctx.cloud.provider = cloudProvider;
+        }
+        if (!groupName.isEmpty()) {
+          ctx.group.name = groupName;
+        }
+        if (!hostId.isEmpty()) {
+          ctx.host.id = hostId;
+        }
+        if (!userDomain.isEmpty()) {
+          ctx.user.domain = userDomain;
+        }
+        if (!userId.isEmpty()) {
+          ctx.user.id = userId;
+        }
+        if (!userName.isEmpty()) {
+          ctx.user.name = userName;
+        }
+        if (!userEmail.isEmpty()) {
+          ctx.user.email = userEmail;
+        }
+        if (!processUserId.isEmpty()) {
+          ctx.process.user.id = processUserId;
+        }
+        if (!processUserName.isEmpty()) {
+          ctx.process.user.name = processUserName;
+        }


Same issue here. Given the number of cases a helper function to convert a hash set to a canonically ordered list would probably be best.

efd6 · 2022-10-11T22:29:55Z

packages/m365_defender/data_stream/alert/elasticsearch/ingest_pipeline/default.yml

+        if (!fileSize.isEmpty()) {
+          ctx.file.size = fileSize;
+        }
+        if (!processPid.isEmpty()) {
+          ctx.process.pid = processPid;
+        }
+        if (!processParentPid.isEmpty()) {
+          ctx.process.parent.pid = processParentPid;
+        }


efd6 · 2022-10-11T22:35:09Z

packages/m365_defender/data_stream/incident/elasticsearch/ingest_pipeline/default.yml

+        if (!cloudProvider.isEmpty()) {
+          ctx.cloud.provider = cloudProvider;
+        }
+        if (!groupName.isEmpty()) {
+          ctx.group.name = groupName;
+        }
+        if (!hostId.isEmpty()) {
+          ctx.host.id = hostId;
+        }
+        if (!userDomain.isEmpty()) {
+          ctx.user.domain = userDomain;
+        }
+        if (!userId.isEmpty()) {
+          ctx.user.id = userId;
+        }
+        if (!userName.isEmpty()) {
+          ctx.user.name = userName;
+        }
+        if (!userEmail.isEmpty()) {
+          ctx.user.email = userEmail;
+        }
+        if (!processUserId.isEmpty()) {
+          ctx.process.user.id = processUserId;
+        }
+        if (!processUserName.isEmpty()) {
+          ctx.process.user.name = processUserName;
+        }


efd6 · 2022-10-11T22:35:21Z

packages/m365_defender/data_stream/incident/elasticsearch/ingest_pipeline/default.yml

+        if (!fileSize.isEmpty()) {
+          ctx.file.size = fileSize;
+        }
+        if (!processPid.isEmpty()) {
+          ctx.process.pid = processPid;
+        }
+        if (!processParentPid.isEmpty()) {
+          ctx.process.parent.pid = processParentPid;
+        }


packages/m365_defender/data_stream/incident/fields/base-fields.yml

efd6 · 2022-10-11T22:40:23Z

packages/m365_defender/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.0.0"


Is there a reason this needs to bump the major version? I don't see any breaking change here.

As discussed, the existing log data stream will be deprecated and will be replaced by a new alert and incident data stream that supports the latest Microsoft Security Graph API.
@P1llus Should we include a note in the readme or do you have other suggestions?

Yeah this is a breaking change and would need a major bump, but I believe there is also a few other things that needs to be discussed.

efd6 · 2022-10-16T21:39:47Z

packages/m365_defender/data_stream/alert/elasticsearch/ingest_pipeline/default.yml

+        ctx.cloud.provider = convertToOrderedArray(cloudProvider);
+        ctx.group.name = convertToOrderedArray(groupName);
+        ctx.host.id = convertToOrderedArray(hostId);
+        ctx.user.domain = convertToOrderedArray(userDomain);
+        ctx.user.id = convertToOrderedArray(userId);
+        ctx.user.name = convertToOrderedArray(userName);
+        ctx.user.email = convertToOrderedArray(userEmail);
+        ctx.process.user.id = convertToOrderedArray(processUserId);
+        ctx.process.user.name = convertToOrderedArray(processUserName);


I think these and the similar calls below should still be conditional to avoid fruitless work here and in the empty value cleaning processor.

packages/m365_defender/data_stream/alert/_dev/test/pipeline/test-alert.log

packages/m365_defender/data_stream/alert/elasticsearch/ingest_pipeline/default.yml

P1llus · 2022-11-07T08:55:43Z

Outside of the comments from @efd6 I have nothing else to add here, it seems good to me 👍

efd6

LGTM after conflict resolved.

Add support for Alert and Incident data-stream

54edf73

vinit-chauhan requested a review from a team as a code owner October 10, 2022 14:02

Updated the changelog file

ad0d66c

vinit-chauhan added enhancement New feature or request Team:Security-External Integrations Integration:m365_defender Microsoft M365 Defender labels Oct 10, 2022

vinit-chauhan changed the title ~~[m365_defender] Add support for Alert and Incident data-stream~~ [m365_defender] Add support for Alert and Incident data-stream with new Security Graph API v2 Oct 11, 2022

vinit-chauhan changed the title ~~[m365_defender] Add support for Alert and Incident data-stream with new Security Graph API v2~~ [m365_defender] Add support for Alert and Incident data-stream with new Security Graph API Oct 11, 2022

efd6 reviewed Oct 11, 2022

View reviewed changes

epixa assigned efd6 Oct 13, 2022

Changes as per Review Comments

59c610a

efd6 reviewed Oct 16, 2022

View reviewed changes

Resolved the comments

2ce2880

efd6 reviewed Oct 30, 2022

View reviewed changes

Remove Alert datstream and resolved review comments

ddb8234

vinit-chauhan changed the title ~~[m365_defender] Add support for Alert and Incident data-stream with new Security Graph API~~ [m365_defender] Add support for Incident data-stream with new Security Graph API Nov 4, 2022

P1llus approved these changes Nov 7, 2022

View reviewed changes

Resolved conflict

a45fed2

efd6 approved these changes Nov 7, 2022

View reviewed changes

Resolved Conflicts

23b5baf

P1llus merged commit 020f3e3 into elastic:main Nov 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[m365_defender] Add support for Incident data-stream with new Security Graph API #4435

[m365_defender] Add support for Incident data-stream with new Security Graph API #4435

vinit-chauhan commented Oct 10, 2022 •

edited

Loading

elasticmachine commented Oct 10, 2022

elasticmachine commented Oct 10, 2022

elasticmachine commented Oct 10, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Oct 10, 2022 •

edited

Loading

efd6 Oct 11, 2022

vinit-chauhan Oct 12, 2022

efd6 Oct 12, 2022

P1llus Oct 18, 2022

efd6 Oct 11, 2022

efd6 Oct 11, 2022

efd6 Oct 11, 2022

efd6 Oct 11, 2022

efd6 Oct 11, 2022

efd6 Oct 11, 2022

vinit-chauhan Oct 12, 2022

P1llus Oct 18, 2022

efd6 Oct 16, 2022

P1llus commented Nov 7, 2022

efd6 left a comment

[m365_defender] Add support for Incident data-stream with new Security Graph API #4435

[m365_defender] Add support for Incident data-stream with new Security Graph API #4435

Conversation

vinit-chauhan commented Oct 10, 2022 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Oct 10, 2022

elasticmachine commented Oct 10, 2022

🚀 Benchmarks report

elasticmachine commented Oct 10, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Oct 10, 2022 • edited Loading

🌐 Coverage report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

P1llus commented Nov 7, 2022

efd6 left a comment

Choose a reason for hiding this comment

vinit-chauhan commented Oct 10, 2022 •

edited

Loading

elasticmachine commented Oct 10, 2022 •

edited

Loading

elasticmachine commented Oct 10, 2022 •

edited

Loading