[M365 Defender] Adjust mappings for detection rules #9860

kcreddy · 2024-05-14T09:40:05Z

Proposed commit message

See title

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

host.os.type: Need sample data to validate non-windows OSPlatform values for: DeviceInfo, CloudAppEvents, IdentifyLogonEvents.

How to test this PR locally

$ elastic-package stack down && elastic-package build && elastic-package stack up -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v

--- Test results for package: m365_defender - START ---
╭───────────────┬─────────────┬───────────┬─────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME                           │ RESULT │ TIME ELAPSED │
├───────────────┼─────────────┼───────────┼─────────────────────────────────────┼────────┼──────────────┤
│ m365_defender │ alert       │ pipeline  │ test-alert.log                      │ PASS   │  11.464875ms │
│ m365_defender │ event       │ pipeline  │ test-alert.log                      │ PASS   │    3.17075ms │
│ m365_defender │ event       │ pipeline  │ test-app-and-identity.log           │ PASS   │   4.927334ms │
│ m365_defender │ event       │ pipeline  │ test-device.log                     │ PASS   │  50.800875ms │
│ m365_defender │ event       │ pipeline  │ test-email.log                      │ PASS   │  10.704625ms │
│ m365_defender │ incident    │ pipeline  │ test-incident.log                   │ PASS   │  18.932917ms │
│ m365_defender │ log         │ pipeline  │ test-m365-defender-empty-ndjson.log │ PASS   │   1.284125ms │
│ m365_defender │ log         │ pipeline  │ test-m365-defender-ndjson.log       │ PASS   │   8.650125ms │
╰───────────────┴─────────────┴───────────┴─────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: m365_defender - END   ---
Done

Related issues

elasticmachine · 2024-05-14T10:38:15Z

🚀 Benchmarks report

Package `m365_defender` 👍(2) 💚(0) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`incident`	620.73	518.67	-102.06 (-16.44%)	💔
`log`	4975.12	3246.75	-1728.37 (-34.74%)	💔

Package `ti_rapid7_threat_command` 👍(1) 💚(1) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`alert`	3412.97	2232.14	-1180.83 (-34.6%)	💔

To see the full report comment with /test benchmark fullreport

…vents

elasticmachine · 2024-05-17T10:47:11Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

packages/m365_defender/data_stream/event/_dev/test/pipeline/test-device.log-expected.json

efd6 · 2024-05-19T21:34:14Z

packages/m365_defender/data_stream/event/fields/fields.yml

+- name: dll.Ext.size
+  type: long
+  description: Size of the dll executable.


Is there a reason not to use file.size?

Looks like file.size is already populated with same value. The value is also copied into this field (taken from endpoint-dev's ECS flat file), which is required for detection rules.

The detection rules probably should be made to conform to the ECS.

The detection rules probably should be made to conform to the ECS.

The proposed changes are meant to adjust previous work on the integration while enhancing compatibility with our own EDR. Detection rules use the data. Definitions and background around mapping decisions within Elastic Defend can be discussed with endpoint-dev.

packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/pipeline_device.yml

efd6 · 2024-05-20T07:23:43Z

packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/pipeline_device.yml

  - set:
      field: process.pe.sections.physical_size
      copy_from: m365_defender.event.file.size
      ignore_empty_value: true
      tag: set_process_pe_sections_physical_size
-      if: ctx.event?.category != null && !ctx.m365_defender.event.category.toLowerCase().contains('deviceimageloadevents') && (ctx.event.category.contains('process') || ctx.m365_defender.event.category.toLowerCase().contains('deviceevents'))
+      if: ctx.event?.category != null && !ctx.event.category.contains('api') && !ctx.event.category.contains('driver') && !ctx.m365_defender.event.category.toLowerCase().contains('deviceimageloadevents') && (ctx.event.category.contains('process') || ctx.m365_defender.event.category.toLowerCase().contains('deviceevents'))


This has the same issue. The field is being used incorrectly.

I have filed an issue.

It makes sense to me. @w0rk3r WDYT?

No objections, we don't use process.pe.sections.physical_size anywhere

Removed reference to *pe.sections.physical_size in 69e9734

elastic-sonarqube · 2024-05-24T08:42:42Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
86.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

This reverts commit 9520e62.

efd6

The use of capitalised fields here makes me concerned.

The ECS allows the use of capitialised fields to prevent namespace collision as noted here. While this is allowed it weakens the ECS and opens up the potential for users to experience difficult to understand and harder to resolve problems with ingest and mapping.

The (unfortunately not explicitly stated) background for field capitalisation use is that we (Elastic) do not use capitalised fields, and so it is safe for anyone else to use them so long as they do not make that capitalisation public (i.e. the schema they use does not end up in publicly visible documents). This ensures that there is no possible way for two fields to have the same name with different mapping types except within a single org (that should have avoided it since it was entirely under their control). By providing a mapping with capitalised name in a publicly visible mapping definition, we (Elastic) break that contract.

The use of these fields to allow the ECS-ish to conform to the mappings used by another product is further troubling since it indicates a breakdown of component interface boundaries, where in this case aspects of our EDR contaminate how we use ECS. Ideally, if fields are missing from that ECS to properly capture the semantics of our EDR, fields should be added to the ECS to allow those semantics to be used more widely. The proliferation of fields with essentially identical semantics is also an issue (in particular dll.Ext.size which is perfectly captured by file.size in conjunction with the presence of any dll.* field).

I'm not going to block this, but I wanted to register that I don't agree with the approach that has been required here.

efd6 · 2024-05-26T20:58:03Z

packages/m365_defender/data_stream/event/_dev/test/pipeline/test-device.log-expected.json

                    },
-                    "pid": 5692,
-                    "start": "2022-11-07T16:45:10.395Z"
+                    "time": "2024-05-08T15:39:06.410Z",


What does "time" mean? It is not documented in the README or fields.yml.

I am unable to find references to this field. My guess is its probably the time when event is made available in EventHub. We derive @timestamp from the event's Timestamp field.

@kcreddy
It looks like time here is when the event was received by M365 Defender XDR

{ "records": [ { "time": "<The time Microsoft Defender XDR received the event>" "tenantId": "<The Id of the tenant that the event belongs to>" "category": "<The Advanced Hunting table name with 'AdvancedHunting-' prefix>" "properties": { <Microsoft Defender XDR Advanced Hunting event as Json> } } ... ] }

This may be good for event.created maybe. Or an extended field in m365_defender.event

Added description for time and category fields in: 8638954

the-pixel-hunter · 2024-05-26T22:09:32Z

Can .caseless be added ot process name as this is used in many pre built detections

efd6 · 2024-05-26T22:19:37Z

@the-pixel-hunter See #9850 (review) and elastic/ecs#2341.

kcreddy · 2024-05-27T06:57:43Z

Thanks, I'll collect all the other items and drop in an issue.

Hey @jvalente-salemstate Just checking if you ended up creating the issue? If not maybe we can move forward with this PR and create another for the new issue.

jvalente-salemstate · 2024-05-27T19:11:15Z

@kcreddy Would you prefer the issue before or after this is merged?

There were several items and a few were fixed (like the handling of initiating vs parent process) so I was holding off until this was merged to assess the entire scope of what was fixed and what wasn't. Either is fine with me.

Perhaps after may be better since some items are just about dashboards and a few are also about the field naming @efd6 had mentioned.

efd6 · 2024-05-27T21:14:02Z

Perhaps after may be better since some items are just about dashboards and a few are also about the field naming @efd6 had mentioned.

If there are things that need to be addressed that would result in additions here being removed, they should be dealt with before this is concretised into a released version.

jvalente-salemstate · 2024-05-27T22:14:20Z

If there are things that need to be addressed that would result in additions here being removed, they should be dealt with before this is concretised into a released version.

Sounds good. I'll pull these out separately into an issue and share it once posted.

jvalente-salemstate · 2024-05-28T02:17:24Z

packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/pipeline_device.yml

  - set:
      field: process.parent.start
      copy_from: m365_defender.event.initiating_process.creation_time
      ignore_empty_value: true
      tag: set_process_parent_start
-      if: ctx.event?.category != null && (ctx.event.category.contains('process') || ctx.m365_defender.event.category.toLowerCase().contains('deviceevents'))
+      if: ctx.event?.category != null && !ctx.event.category.contains('api') && !ctx.event.category.contains('driver') && (ctx.event.category.contains('process') || ctx.m365_defender.event.category.toLowerCase().contains('deviceevents'))
  - set:
      field: process.parent.name


@kcreddy This should also exclude deviceevents or map this to process.name. See #9993
This change does seem to fix dll load process events but this would still be an issue on device events. I could see this also introducing a large number of false positives with events where that specific process being a parent process is an IoC.

@w0rk3r WDYT?

jvalente-salemstate · 2024-05-29T23:08:47Z

packages/m365_defender/data_stream/event/fields/fields.yml

+- name: process.Ext.api.parameters.desired_access_numeric
+  type: long
+  description: This parameter indicates the numeric value of the `DesiredAccess` field passed to `OpenProcess` or `OpenThread`.
+- name: Target.process.name


@kcreddy I've read the discussion on the review by @efd6 so I get this may be a much larger discussion between internal Elastic teams than this PR; this is more for suggestions for future revisions. This may also need involvement with the ECS team.

ECS does lack a good way to demonstrate this but I am thinking this bit can be resolved by either/both:

Allowing/Using target.* as a root level field (same approach here, without the capital T)

Expanding where target can be nested, and what can be nested under target. The ECS docs make a very good case for this regarding IAM, but I see a strong case in allowing others such as process and host. That would allow for accurate representation of things like this as well as a wider range of EDR/XDR or administrative activities. Examples would be the target host for an isolation event. Target IP/URL for block/allow list changes. Using target rather than the existing fields (source.*, destination.*, etc) also enables filtering for/against those details when it's specifically a target.

I agree this would need discussing above possible solutions with ECS team. Also, in current ECS definitions, the target does appear to be nested in few places such as user.target.*, service.target.*, cloud.target.*, and this could be extended further.

elasticmachine · 2024-05-30T08:41:59Z

💔 Build Failed

Buildkite Build
Commit: 8638954

Failed CI Steps

:pipeline:⬆️ Upload Pipeline: .buildkite/pipeline.yml

History

💚 Build #11858 succeeded 69e9734
💚 Build #11678 succeeded 4455a7c
💚 Build #11643 succeeded 56e220a
💔 Build #11639 failed d130f11
💚 Build #11537 succeeded 511c329
💔 Build #11521 failed 5b09ee4

cc @kcreddy

kcreddy added 2 commits May 14, 2024 15:06

Add changelog

31bfb5e

Add host.os.type ECS field

4d84c64

kcreddy self-assigned this May 14, 2024

kcreddy added enhancement New feature or request Integration: M365_Defender labels May 14, 2024

kcreddy added 3 commits May 14, 2024 15:10

update pr entry

1fd25bd

fix compile error

d8ed917

Change event.type for processcreated

a59f8f5

kcreddy added 21 commits May 14, 2024 17:09

process.parent.code_signature fields

346adaa

change event.category to library

22fbe0a

updated event.action to load from imageloaded

021d77a

Add user.id

70c1a17

Add user.name

2292f55

Add user.domain

fdab734

Add library condition

1190e24

Replace dll.pe.sections.physical_size with dll.Ext.size for library e…

6a094dc

…vents

Add process.Ext.token.integrity_level_name for process

5d1ba74

Add process.Ext.token.integrity_level_name for library

6308cdd

Test if process.parent.* becomes process.* after changing to library

4c19edc

update event.type and event.action for file

714769c

file.path instead of file.directory for file

d179742

Add file.directory and file.extension for file

c1f556f

Add user.id for file

1331a52

Update event.action for registry

4629538

Add windows registry.path for registry

9da32a7

Add user.id for registry

fe55b3d

Add user.name and user.domain for registry

faeeb9b

Remove HKEY_* from registry.key for registry

07bb9da

Add registry.data.type for registry

cf94d7e

kcreddy added 4 commits May 17, 2024 15:30

Add Target.process.* when remtoe apicalls for api

6343bfb

update fields in README

d130f11

Merge remote-tracking branch 'upstream/main' into m365_defender_rules

5aa74de

check host.os.type with linux device

56e220a

kcreddy marked this pull request as ready for review May 17, 2024 10:46

kcreddy requested a review from a team as a code owner May 17, 2024 10:46

kcreddy added the Team:Security-Service Integrations Security Service Integrations Team label May 17, 2024

efd6 reviewed May 19, 2024

View reviewed changes

Address PR comments

4455a7c

efd6 reviewed May 20, 2024

View reviewed changes

narph added the Category: 3rd EDR label May 21, 2024

kcreddy requested a review from w0rk3r May 21, 2024 09:29

Removes incorrect usage of *.pe.sections.physical_size

69e9734

kcreddy requested a review from efd6 May 24, 2024 09:57

kcreddy added 2 commits May 24, 2024 17:03

Add pagesize and pageindex to request

9520e62

Revert "Add pagesize and pageindex to request"

8fd4e2c

This reverts commit 9520e62.

efd6 approved these changes May 26, 2024

View reviewed changes

jvalente-salemstate reviewed May 28, 2024

View reviewed changes

kcreddy mentioned this pull request May 29, 2024

[Microsoft M365 Defender] Incorrect mapping initiating process to process.parent. #9993

Open

jvalente-salemstate reviewed May 29, 2024

View reviewed changes

Add description for fields: time and category

8638954

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[M365 Defender] Adjust mappings for detection rules #9860

[M365 Defender] Adjust mappings for detection rules #9860

kcreddy commented May 14, 2024 •

edited

elasticmachine commented May 14, 2024 •

edited

elasticmachine commented May 17, 2024

efd6 May 19, 2024

kcreddy May 20, 2024 •

edited

efd6 May 20, 2024

w0rk3r May 21, 2024

efd6 May 20, 2024

kcreddy May 21, 2024

w0rk3r May 21, 2024

kcreddy May 24, 2024 •

edited

elastic-sonarqube bot commented May 24, 2024

efd6 left a comment

efd6 May 26, 2024

kcreddy May 27, 2024

jvalente-salemstate May 27, 2024 •

edited

kcreddy May 30, 2024

the-pixel-hunter commented May 26, 2024

efd6 commented May 26, 2024 •

edited

kcreddy commented May 27, 2024 •

edited

jvalente-salemstate commented May 27, 2024 •

edited

efd6 commented May 27, 2024

jvalente-salemstate commented May 27, 2024

jvalente-salemstate May 28, 2024 •

edited

kcreddy May 29, 2024

jvalente-salemstate May 29, 2024

kcreddy May 30, 2024

elasticmachine commented May 30, 2024

[M365 Defender] Adjust mappings for detection rules #9860

Are you sure you want to change the base?

[M365 Defender] Adjust mappings for detection rules #9860

Conversation

kcreddy commented May 14, 2024 • edited

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

elasticmachine commented May 14, 2024 • edited

🚀 Benchmarks report

Package m365_defender 👍(2) 💚(0) 💔(2)

Package ti_rapid7_threat_command 👍(1) 💚(1) 💔(1)

elasticmachine commented May 17, 2024

Choose a reason for hiding this comment

kcreddy May 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kcreddy May 24, 2024 • edited

Choose a reason for hiding this comment

elastic-sonarqube bot commented May 24, 2024

Quality Gate passed

efd6 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jvalente-salemstate May 27, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

the-pixel-hunter commented May 26, 2024

efd6 commented May 26, 2024 • edited

kcreddy commented May 27, 2024 • edited

jvalente-salemstate commented May 27, 2024 • edited

efd6 commented May 27, 2024

jvalente-salemstate commented May 27, 2024

jvalente-salemstate May 28, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented May 30, 2024

💔 Build Failed

Failed CI Steps

History

kcreddy commented May 14, 2024 •

edited

elasticmachine commented May 14, 2024 •

edited

Package `m365_defender` 👍(2) 💚(0) 💔(2)

Package `ti_rapid7_threat_command` 👍(1) 💚(1) 💔(1)

kcreddy May 20, 2024 •

edited

kcreddy May 24, 2024 •

edited

jvalente-salemstate May 27, 2024 •

edited

efd6 commented May 26, 2024 •

edited

kcreddy commented May 27, 2024 •

edited

jvalente-salemstate commented May 27, 2024 •

edited

jvalente-salemstate May 28, 2024 •

edited