[Microsoft_Defender_Cloud] Initial Release for the Microsoft Defender for Cloud #6593

piyush-elastic · 2023-06-16T08:37:45Z

What does this PR do?

Updated data collection logic for the event data stream.
Updated the ingest pipeline for the event data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate .yml files.
Added dashboards and visualizations.
Updated test for pipeline for the event data stream.
Updated system test cases for the event data stream.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

Change follows the contributing guidelines
Supported versions of the monitoring target is documented
Supported operating systems are documented (if applicable)
Integration or System tests exist
Documentation exists
Fields follow ECS and naming conventions
At least a manual test with ES / Kibana / Agent has been performed.
Required Kibana version set to: ^8.3.0

New Package

Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

Dashboards exists
Screenshots added or updated
Data stream filters added to visualizations

Log dataset changes

Pipeline tests exist (if applicable)
Generated output for at least 1 log file exists
Sample event (sample_event.json) exists

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/microsoft_defender_cloud directory.
Run the following command to run tests.

elastic-package test

Related issues

Closes Microsoft Defender for Cloud #1148

Screenshots

2023/06/15 14:42:25  INFO New version is available - v0.82.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.82.0
Run test suite for the package
Run system tests for the package
--- Test results for package: microsoft_defender_cloud - START ---
No test results
--- Test results for package: microsoft_defender_cloud - END   ---
Done
Run asset tests for the package
--- Test results for package: microsoft_defender_cloud - START ---
╭──────────────────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE                  │ DATA STREAM │ TEST TYPE │ TEST NAME                                                                         │ RESULT │ TIME ELAPSED │
├──────────────────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ microsoft_defender_cloud │             │ asset     │ dashboard microsoft_defender_cloud-97eaf040-0516-11ee-b4db-89b3a5f6df7f is loaded │ PASS   │      2.874µs │
│ microsoft_defender_cloud │             │ asset     │ search microsoft_defender_cloud-52f0f2f0-039f-11ee-bafb-95960de71508 is loaded    │ PASS   │        652ns │
│ microsoft_defender_cloud │ event       │ asset     │ index_template logs-microsoft_defender_cloud.event is loaded                      │ PASS   │        836ns │
│ microsoft_defender_cloud │ event       │ asset     │ ingest_pipeline logs-microsoft_defender_cloud.event-0.1.0 is loaded               │ PASS   │        686ns │
╰──────────────────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: microsoft_defender_cloud - END   ---
Done
Run pipeline tests for the package
--- Test results for package: microsoft_defender_cloud - START ---
╭──────────────────────────┬─────────────┬───────────┬────────────────┬────────┬──────────────╮
│ PACKAGE                  │ DATA STREAM │ TEST TYPE │ TEST NAME      │ RESULT │ TIME ELAPSED │
├──────────────────────────┼─────────────┼───────────┼────────────────┼────────┼──────────────┤
│ microsoft_defender_cloud │ event       │ pipeline  │ test-alert.log │ PASS   │    67.6318ms │
╰──────────────────────────┴─────────────┴───────────┴────────────────┴────────┴──────────────╯
--- Test results for package: microsoft_defender_cloud - END   ---
Done
Run static tests for the package
--- Test results for package: microsoft_defender_cloud - START ---
No test results
--- Test results for package: microsoft_defender_cloud - END   ---
Done

elasticmachine · 2023-06-16T08:43:45Z

💚 Build Succeeded

Buildkite Build
Commit: be4dc36

History

💚 Build #166 succeeded 9e5fbcb

elasticmachine · 2023-06-16T08:51:55Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-11T06:02:43.278+0000
Duration: 13 min 43 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5
Skipped	0
Total	5

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-06-16T08:52:04Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚 2.861
Classes	100.0% (`1/1`)	💚 2.861
Methods	81.818% (`9/11`)	👎 -10.462
Lines	95.166% (`1260/1324`)	👍 4.025
Conditionals	100.0% (`0/0`)	💚

…olved

elasticmachine · 2023-06-26T13:53:42Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6 · 2023-06-28T21:40:31Z

.github/CODEOWNERS

@@ -242,3 +242,4 @@
 /packages/statsd_input @elastic/obs-infraobs-integrations
 /packages/zeronetworks @elastic/security-external-integrations
 /packages/prometheus_input @elastic/obs-infraobs-integrations
+/packages/microsoft_defender_cloud @elastic/security-external-integrations


Please add this in sort order.

efd6 · 2023-06-28T21:47:57Z

packages/microsoft_defender_cloud/_dev/build/docs/README.md

+
+## Data streams
+
+The Microsoft Defender for Cloud integration collects one type of data : event.


Suggested change

The Microsoft Defender for Cloud integration collects one type of data : event.

The Microsoft Defender for Cloud integration collects one type of data: event.

efd6 · 2023-06-28T21:49:14Z

packages/microsoft_defender_cloud/_dev/build/docs/README.md

+
+### To collect data from Microsoft Azure Event Hub, follow the below steps:
+
+- Configure the microsoft defender for cloud on azure subscription. For more detail, refer to the link [here](https://learn.microsoft.com/en-us/azure/defender-for-cloud/get-started).


Suggested change

- Configure the microsoft defender for cloud on azure subscription. For more detail, refer to the link [here](https://learn.microsoft.com/en-us/azure/defender-for-cloud/get-started).

- Configure the Microsoft Defender for Cloud on Azure subscription. For more detail, refer to the link [here](https://learn.microsoft.com/en-us/azure/defender-for-cloud/get-started).

efd6 · 2023-06-28T22:10:18Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+      on_failure:
+        - append:
+            field: error.message
+            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag fail-{{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'


This is not currently a correct message; it includes a reference to "fail-{{{_ingest.on_failure_processor_tag}}}" which is not placed anywhere. There are cases where this is done in other packages, but this is not done consistently where this error message is used. We need to make a decision about whether the machine-friendly tag version is going to be used and make sure the human-friendly message is consistent with that.

@efd6 - not sure if I understood this correctly, but do you mean we should not use this standard format for each and every package and prefer something human friendly? days before @P1llus asked to used tag for each and every processor.

The format that's here includes reference to "fail-{{{_ingest.on_failure_processor_tag}}}" as a tag. There is no tag of that format added in this pipeline. This was done in 74f8fc4 for checkpoint, and I think for some others in other commits. Either we should remove this reference to the tag or the tag should be added as it is in that commit. I don't know which is the better solution. @P1llus do you have a view?

It probably should suffice to remove the "fail-" prefix.

@efd6 - I recall the discussion on this, and we opted against using the -fail processor because if the pipeline reaches the -fail processor, it will stop the execution of any further code. I agree on removing fail- suffix as with that whole message looks mess up with fail keywords.

The alternative would be to add the "fail-{{{_ingest.on_failure_processor_tag}}}" value to error.message as well, but I agree, not doing that and trimming the prefix is probably the way to go.

efd6 · 2023-06-28T22:19:03Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+  - set:
+      field: event.type
+      value: [indicator]
+      tag: set_event_type
+      if: ctx.json?.alerttype != null && ['arm_anomalousserviceoperation.credentialaccess','arm_anomalousserviceoperation.collection','arm_anomalousserviceoperation.defenseevasion','arm_anomalousserviceoperation.execution','arm_anomalousserviceoperation.impact','arm_anomalousserviceoperation.initialaccess','arm_anomalousserviceoperation.lateralmovement','arm_anomalousserviceoperation.persistence','arm_anomalousserviceoperation.privilegeescalation','arm_unusedaccountpersistence','arm_unusedapppowershellpersistence','arm_unusedappibizapersistence','arm_privilegedroledefinitioncreation','arm_anomalousrbacroleassignment','arm_anomalousoperation.credentialaccess','arm_anomalousoperation.collection','arm_anomalousoperation.defenseevasion','arm_anomalousoperation.execution','arm_anomalousoperation.impact','arm_anomalousoperation.initialaccess','arm_anomalousoperation.lateralmovement','arm_anomalousoperation.persistence','arm_anomalousoperation.privilegeescalation','arm_microburst.runcodeonbehalf','arm_netspi.maintainpersistence','arm_powerzure.runcodeonbehalf','arm_powerzure.maintainpersistence','arm_anomalousclassicroleassignment'].contains(ctx.json.alerttype.toLowerCase())


I'm wondering if this would not be improved from a code maintenance perspective by making it like so:

Suggested change

- set:

field: event.type

value: [indicator]

tag: set_event_type

if: ctx.json?.alerttype != null && ['arm_anomalousserviceoperation.credentialaccess','arm_anomalousserviceoperation.collection','arm_anomalousserviceoperation.defenseevasion','arm_anomalousserviceoperation.execution','arm_anomalousserviceoperation.impact','arm_anomalousserviceoperation.initialaccess','arm_anomalousserviceoperation.lateralmovement','arm_anomalousserviceoperation.persistence','arm_anomalousserviceoperation.privilegeescalation','arm_unusedaccountpersistence','arm_unusedapppowershellpersistence','arm_unusedappibizapersistence','arm_privilegedroledefinitioncreation','arm_anomalousrbacroleassignment','arm_anomalousoperation.credentialaccess','arm_anomalousoperation.collection','arm_anomalousoperation.defenseevasion','arm_anomalousoperation.execution','arm_anomalousoperation.impact','arm_anomalousoperation.initialaccess','arm_anomalousoperation.lateralmovement','arm_anomalousoperation.persistence','arm_anomalousoperation.privilegeescalation','arm_microburst.runcodeonbehalf','arm_netspi.maintainpersistence','arm_powerzure.runcodeonbehalf','arm_powerzure.maintainpersistence','arm_anomalousclassicroleassignment'].contains(ctx.json.alerttype.toLowerCase())

- set:

field: event.type

value: [indicator]

tag: set_event_type

if: |

ctx.json?.alerttype != null && [

'arm_anomalousserviceoperation.credentialaccess',

'arm_anomalousserviceoperation.collection',

'arm_anomalousserviceoperation.defenseevasion',

'arm_anomalousserviceoperation.execution',

'arm_anomalousserviceoperation.impact',

'arm_anomalousserviceoperation.initialaccess',

'arm_anomalousserviceoperation.lateralmovement',

'arm_anomalousserviceoperation.persistence',

'arm_anomalousserviceoperation.privilegeescalation',

'arm_unusedaccountpersistence',

'arm_unusedapppowershellpersistence',

'arm_unusedappibizapersistence',

'arm_privilegedroledefinitioncreation',

'arm_anomalousrbacroleassignment',

'arm_anomalousoperation.credentialaccess',

'arm_anomalousoperation.collection',

'arm_anomalousoperation.defenseevasion',

'arm_anomalousoperation.execution',

'arm_anomalousoperation.impact',

'arm_anomalousoperation.initialaccess',

'arm_anomalousoperation.lateralmovement',

'arm_anomalousoperation.persistence',

'arm_anomalousoperation.privilegeescalation',

'arm_microburst.runcodeonbehalf',

'arm_netspi.maintainpersistence',

'arm_powerzure.runcodeonbehalf',

'arm_powerzure.maintainpersistence',

'arm_anomalousclassicroleassignment'

].contains(ctx.json.alerttype.toLowerCase())

(similar below)

efd6 · 2023-06-28T22:20:39Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+          return updatedJson;
+        }
+
+        def keyMap = new HashMap();


Can this not be defined as a parameter map? (similar elsewhere)

efd6 · 2023-06-28T22:26:51Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+      ignore_missing: true
+  - foreach:
+      field: microsoft_defender_cloud.event.entities
+      if: ctx.microsoft_defender_cloud?.event?.entities != null && ctx.microsoft_defender_cloud.event.entities instanceof List


Suggested change

if: ctx.microsoft_defender_cloud?.event?.entities != null && ctx.microsoft_defender_cloud.event.entities instanceof List

if: ctx.microsoft_defender_cloud?.event?.entities instanceof List

and similar below

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

…crosoft_defender_cloud-0.1.0

efd6 · 2023-07-09T21:15:40Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+      if: ctx.microsoft_defender_cloud?.event?.entities != null && ctx.microsoft_defender_cloud.event.entities instanceof List
+      processor:
+        foreach:
+          field: _ingest._value.ip_adresses


s/adresses/addresses/g

efd6 · 2023-07-09T21:21:58Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+        "hostname": "host_name"
+        "imagefile": "image_file"
+        "imageid": "image_id"
+        "ipadresses": "ip_adresses"


Suggested change

"ipadresses": "ip_adresses"

"ipadresses": "ip_addresses"

efd6 · 2023-07-09T21:22:18Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+            convert:
+              field: _ingest._value.address
+              type: ip
+              tag: convert_ip_adresses_address_to_ip


Suggested change

tag: convert_ip_adresses_address_to_ip

tag: convert_ip_addresses_address_to_ip

efd6 · 2023-07-09T21:22:34Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+      ignore_failure: true
+      processor:
+        foreach:
+          field: _ingest._value.ip_adresses


Suggested change

field: _ingest._value.ip_adresses

field: _ingest._value.ip_addresses

efd6 · 2023-07-09T21:22:48Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+      ignore_failure: true
+      processor:
+        foreach:
+          field: _ingest._value.ip_adresses


Suggested change

field: _ingest._value.ip_adresses

field: _ingest._value.ip_addresses

efd6 · 2023-07-09T21:23:00Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+            convert:
+              field: _ingest._value.asset
+              type: boolean
+              tag: convert_ip_adresses_asset_to_boolean


Suggested change

tag: convert_ip_adresses_asset_to_boolean

tag: convert_ip_addresses_asset_to_boolean

efd6 · 2023-07-09T21:23:12Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+            convert:
+              field: _ingest._value.location.asn
+              type: long
+              tag: convert_ip_adresses_location_asn_to_long


Suggested change

tag: convert_ip_adresses_location_asn_to_long

tag: convert_ip_addresses_location_asn_to_long

efd6 · 2023-07-09T21:23:22Z

packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml

+      ignore_failure: true
+      processor:
+        foreach:
+          field: _ingest._value.ip_adresses


Suggested change

field: _ingest._value.ip_adresses

field: _ingest._value.ip_addresses

efd6 · 2023-07-09T21:23:51Z

packages/microsoft_defender_cloud/data_stream/event/fields/fields.yml

+                  type: keyword
+            - name: image_id
+              type: keyword
+            - name: ip_adresses


Suggested change

- name: ip_adresses

- name: ip_addresses

efd6 · 2023-07-09T21:24:43Z

packages/microsoft_defender_cloud/img/microsoft-defender-cloud-dashboard-event.png

Can this be downscaled? It's currently 1.23MB.

…astic/integrations into microsoft_defender_cloud-0.1.0

efd6 · 2023-07-10T22:48:59Z

packages/microsoft_defender_cloud/data_stream/event/_dev/test/pipeline/test-alert.log

Can I confirm that the data comes from the API with the correct spelling?

Yes , we have verified that and it's identical.

elasticmachine · 2023-07-11T07:42:58Z

Package microsoft_defender_cloud - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_defender_cloud

… for Cloud (#6593) * Initial Release * Update the changelog entry * Indentified Readme changes from comments on other integration and resolved * Resolve review comments * Conflicts Resolve * Review comments resolve

piyush-elastic added 2 commits June 16, 2023 13:57

Initial Release

9e5fbcb

Update the changelog entry

be4dc36

Indentified Readme changes from comments on other integration and res…

8ff045f

…olved

jamiehynds requested a review from a team June 26, 2023 13:53

jamiehynds added the Team:Security-External Integrations label Jun 26, 2023

jamiehynds added the New Integration label Jun 26, 2023

efd6 reviewed Jun 28, 2023

View reviewed changes

efd6 mentioned this pull request Jun 29, 2023

[google_scc] Initial release for the Google SCC #6645

Merged

15 tasks

piyush-elastic and others added 4 commits July 3, 2023 16:18

Resolve review comments

64df965

Merge branch 'main' of github.com:piyush-elastic/integrations into mi…

59f5ccd

…crosoft_defender_cloud-0.1.0

Conflicts Resolve

8d35043

Merge branch 'main' into microsoft_defender_cloud-0.1.0

bfe2cea

piyush-elastic requested a review from efd6 July 3, 2023 12:19

efd6 reviewed Jul 9, 2023

View reviewed changes

piyush-elastic added 3 commits July 10, 2023 12:14

Sync with main branch

2ba2231

Merge branch 'microsoft_defender_cloud-0.1.0' of github.com:piyush-el…

0c5cd3a

…astic/integrations into microsoft_defender_cloud-0.1.0

Review comments resolve

812b82f

piyush-elastic requested a review from efd6 July 10, 2023 07:30

efd6 reviewed Jul 10, 2023

View reviewed changes

Merge branch 'elastic:main' into microsoft_defender_cloud-0.1.0

b6d3f8d

efd6 approved these changes Jul 11, 2023

View reviewed changes

P1llus merged commit 2159cb1 into elastic:main Jul 11, 2023
4 checks passed

andrewkroh added the Integration:microsoft_defender_cloud Microsoft Defender for Cloud label Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Microsoft_Defender_Cloud] Initial Release for the Microsoft Defender for Cloud #6593

[Microsoft_Defender_Cloud] Initial Release for the Microsoft Defender for Cloud #6593

piyush-elastic commented Jun 16, 2023

elasticmachine commented Jun 16, 2023

elasticmachine commented Jun 16, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jun 16, 2023 •

edited

Loading

elasticmachine commented Jun 26, 2023

efd6 Jun 28, 2023

efd6 Jun 28, 2023

efd6 Jun 28, 2023

efd6 Jun 28, 2023

piyush-elastic Jun 29, 2023

efd6 Jun 29, 2023 •

edited

Loading

piyush-elastic Jun 29, 2023 •

edited

Loading

efd6 Jun 29, 2023

efd6 Jun 28, 2023

efd6 Jun 28, 2023

efd6 Jun 28, 2023

efd6 Jul 9, 2023

efd6 Jul 9, 2023

efd6 Jul 9, 2023

efd6 Jul 9, 2023

efd6 Jul 9, 2023

efd6 Jul 9, 2023

efd6 Jul 9, 2023

efd6 Jul 9, 2023

efd6 Jul 9, 2023

efd6 Jul 9, 2023

efd6 Jul 10, 2023

piyush-elastic Jul 11, 2023

elasticmachine commented Jul 11, 2023


		## Data streams

		The Microsoft Defender for Cloud integration collects one type of data : event.


		### To collect data from Microsoft Azure Event Hub, follow the below steps:

		- Configure the microsoft defender for cloud on azure subscription. For more detail, refer to the link [here](https://learn.microsoft.com/en-us/azure/defender-for-cloud/get-started).

	if: ctx.microsoft_defender_cloud?.event?.entities != null && ctx.microsoft_defender_cloud.event.entities instanceof List
	if: ctx.microsoft_defender_cloud?.event?.entities instanceof List

	tag: convert_ip_adresses_address_to_ip
	tag: convert_ip_addresses_address_to_ip

	field: _ingest._value.ip_adresses
	field: _ingest._value.ip_addresses

	tag: convert_ip_adresses_asset_to_boolean
	tag: convert_ip_addresses_asset_to_boolean

	tag: convert_ip_adresses_location_asn_to_long
	tag: convert_ip_addresses_location_asn_to_long

[Microsoft_Defender_Cloud] Initial Release for the Microsoft Defender for Cloud #6593

[Microsoft_Defender_Cloud] Initial Release for the Microsoft Defender for Cloud #6593

Conversation

piyush-elastic commented Jun 16, 2023

What does this PR do?

Integration release checklist

All changes

New Package

Dashboards changes

Log dataset changes

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jun 16, 2023

💚 Build Succeeded

History

elasticmachine commented Jun 16, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jun 16, 2023 • edited Loading

🌐 Coverage report

elasticmachine commented Jun 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 Jun 29, 2023 • edited Loading

Choose a reason for hiding this comment

piyush-elastic Jun 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Jul 11, 2023

elasticmachine commented Jun 16, 2023 •

edited

Loading

elasticmachine commented Jun 16, 2023 •

edited

Loading

efd6 Jun 29, 2023 •

edited

Loading

piyush-elastic Jun 29, 2023 •

edited

Loading