[m365_defender] Add Support of Alert Data Stream #8950
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
efd6
reviewed
Jan 24, 2024
| "tenantId": "b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c", | ||
| "title": "Suspicious execution of hidden file", | ||
| "description": "A hidden file has been launched. This activity could indicate a compromised host. Attackers often hide files associated with malicious tools to evade file system inspection and defenses.", | ||
| "recommendedActions": "Collect artifacts and determine scope\n�\tReview the machine timeline for suspicious activities that may have occurred before and after the time of the alert, and record additional related artifacts (files, IPs/URLs) \n�\tLook for the presence of relevant artifacts on other systems. Identify commonalities and differences between potentially compromised systems.\n�\tSubmit relevant files for deep analysis and review resulting detailed behavioral information.\n�\tSubmit undetected files to the MMPC malware portal\n\nInitiate containment & mitigation \n�\tContact the user to verify intent and initiate local remediation actions as needed.\n�\tUpdate AV signatures and run a full scan. The scan might reveal and remove previously-undetected malware components.\n�\tEnsure that the machine has the latest security updates. In particular, ensure that you have installed the latest software, web browser, and Operating System versions.\n�\tIf credential theft is suspected, reset all relevant users passwords.\n�\tBlock communication with relevant URLs or IPs at the organization�s perimeter.", |
There was a problem hiding this comment.
This line contains invalid UTF-8. How was the data collected? (this is also true of the cases already in this file).
00006440 6e 64 20 64 65 74 65 72 6d 69 6e 65 20 73 63 6f |nd determine sco|
00006450 70 65 5c 6e>ef bf bd<5c 74 52 65 76 69 65 77 20 |pe\n...\tReview |
00006460 74 68 65 20 6d 61 63 68 69 6e 65 20 74 69 6d 65 |the machine time|
0xef 0xbf 0xbd is 0xfffd which is the Unicode replacement character.
There was a problem hiding this comment.
This applies to the other additions. Can you explain why this is happening?
packages/m365_defender/data_stream/alert/_dev/test/pipeline/test-alert.log
Outdated
Show resolved
Hide resolved
Comment on lines
44
to
64
| if (['#microsoft.graph.security.analyzedMessageEvidence', '#microsoft.graph.security.mailboxEvidence', '#microsoft.graph.security.mailClusterEvidence'].contains(evidence["@odata.type"])) { | ||
| eventCategory.add('email'); | ||
| eventType.add('info'); | ||
| } else if (evidence["@odata.type"] == '#microsoft.graph.security.deviceEvidence') { | ||
| eventCategory.add('host'); | ||
| eventType.add('info'); | ||
| } else if (evidence["@odata.type"] == '#microsoft.graph.security.fileEvidence') { | ||
| eventCategory.add('file'); | ||
| eventType.add('info'); | ||
| } else if (evidence["@odata.type"] == '#microsoft.graph.security.ipEvidence') { | ||
| eventCategory.add('network'); | ||
| eventType.add('info'); | ||
| } else if (evidence["@odata.type"] == '#microsoft.graph.security.processEvidence') { | ||
| eventCategory.add('process'); | ||
| eventType.add('info'); | ||
| } else if (['#microsoft.graph.security.registryValueEvidence', '#microsoft.graph.security.registryKeyEvidence'].contains(evidence["@odata.type"])) { | ||
| eventCategory.add('registry'); | ||
| eventType.add('access') | ||
| } else if (['#microsoft.graph.security.userEvidence', '#microsoft.graph.security.securityGroupEvidence'].contains(evidence["@odata.type"])) { | ||
| eventCategory.add('iam'); | ||
| eventType.add('info'); |
There was a problem hiding this comment.
Can this be done with a param map for a look-up?
| def processUserId = new HashSet(); | ||
| def processUserName = new HashSet(); | ||
| for (evidence in ctx.json.evidence) { | ||
| if (evidence?.odata_type != null) { |
There was a problem hiding this comment.
Just for this first condition, can we invert the logic and continue instead. This will de-indent the rest of the loop body by one level.
Suggested change
| if (evidence?.odata_type != null) { | |
| if (evidence?.odata_type == null) { | |
| continue | |
| } |
| def processPid = new HashSet(); | ||
| def processParentPid = new HashSet(); | ||
| for (evidence in ctx.json.evidence) { | ||
| if (evidence.odata_type != null) { |
|
/test |
🚀 Benchmarks reportTo see the full report comment with |
|
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
narph
added
Team:Security-Service Integrations
…m365_defender-2.7.0
efd6
reviewed
Jan 31, 2024
| @@ -0,0 +1 @@ | |||
| {"@odata.type":"#microsoft.graph.security.alert","id":"da637551227677560813_-961444813","providerAlertId":"da637551227677560813_-961444813","incidentId":"28282","status":"new","severity":"low","classification":"unknown","determination":"apt","serviceSource":"microsoftDefenderForEndpoint","detectionSource":"antivirus","detectorId":"e0da400f-affd-43ef-b1d5-afc2eb6f2756","tenantId":"b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","title":"Suspicious execution of hidden file","description":"A hidden file has been launched. This activity could indicate a compromised host. Attackers often hide files associated with malicious tools to evade file system inspection and defenses.","recommendedActions":"Collect artifacts and determine scope\n\ufffd\tReview the machine timeline for suspicious activities that may have occurred before and after the time of the alert, and record additional related artifacts (files, IPs\/URLs) \n\ufffd\tLook for the presence of relevant artifacts on other systems. Identify commonalities and differences between potentially compromised systems.\n\ufffd\tSubmit relevant files for deep analysis and review resulting detailed behavioral information.\n\ufffd\tSubmit undetected files to the MMPC malware portal\n\nInitiate containment & mitigation \n\ufffd\tContact the user to verify intent and initiate local remediation actions as needed.\n\ufffd\tUpdate AV signatures and run a full scan. The scan might reveal and remove previously-undetected malware components.\n\ufffd\tEnsure that the machine has the latest security updates. In particular, ensure that you have installed the latest software, web browser, and Operating System versions.\n\ufffd\tIf credential theft is suspected, reset all relevant users passwords.\n\ufffd\tBlock communication with relevant URLs or IPs at the organization\ufffds perimeter.","category":"DefenseEvasion","assignedTo":null,"alertWebUrl":"https:\/\/security.microsoft.com\/alerts\/da637551227677560813_-961444813?tid=b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","incidentWebUrl":"https:\/\/security.microsoft.com\/incidents\/28282?tid=b3c1b5fc-828c-45fa-a1e1-10d74f6d6e9c","actorDisplayName":null,"threatDisplayName":null,"threatFamilyName":null,"mitreTechniques":["T1564.001"],"createdDateTime":"2021-04-27T12:19:27.7211305Z","lastUpdateDateTime":"2021-05-02T14:19:01.3266667Z","resolvedDateTime":null,"firstActivityDateTime":"2021-04-26T07:45:50.116Z","lastActivityDateTime":"2021-05-02T07:56:58.222Z","comments":[],"evidence":[{"@odata.type":"#microsoft.graph.security.analyzedMessageEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"firstSeenDateTime":"2020-09-12T07:28:32.4321753Z","mdeDeviceId":"73e7e2de709dff64ef64b1d0c30e67fab63279db","azureAdDeviceId":null,"deviceDnsName":"tempDns","osPlatform":"Windows10","osBuild":22424,"version":"Other","healthStatus":"active","riskScore":"medium","rbacGroupId":75,"rbacGroupName":"UnassignedGroup","onboardingStatus":"onboarded","defenderAvStatus":"unknown","ipInterfaces":["1.1.1.1"],"loggedOnUsers":[],"roles":["compromised"],"detailedRoles":["Main device"],"tags":["Test Machine"],"vmMetadata":{"vmId":"ca1b0d41-5a3b-4d95-b48b-f220aed11d78","cloudProvider":"azure","resourceId":"\/subscriptions\/8700d3a3-3bb7-4fbe-a090-488a1ad04161\/resourceGroups\/WdatpApi-EUS-STG\/providers\/Microsoft.Compute\/virtualMachines\/NirLaviTests","subscriptionId":"8700d3a3-3bb7-4fbe-a090-488a1ad04161"}},{"@odata.type":"#microsoft.graph.security.analyzedMessageEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"detectionStatus":"detected","mdeDeviceId":"73e7e2de709dff64ef64b1d0c30e67fab63279db","roles":[],"detailedRoles":["Referred in command line"],"tags":[],"fileDetails":{"sha1":"5f1e8acedc065031aad553b710838eb366cfee9a","sha256":"8963a19fb992ad9a76576c5638fd68292cffb9aaac29eb8285f9abf6196a7dec","fileName":"MsSense.exe","filePath":"C:\\Program Files\\temp","fileSize":6136392,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null}},{"@odata.type":"#microsoft.graph.security.fileEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"processId":4780,"parentProcessId":668,"processCommandLine":"\"MsSense.exe\"","processCreationDateTime":"2021-08-12T12:43:19.0772577Z","parentProcessCreationDateTime":"2021-08-12T07:39:09.0909239Z","detectionStatus":"detected","mdeDeviceId":"73e7e2de709dff64ef64b1d0c30e67fab63279db","roles":[],"detailedRoles":[],"tags":[],"imageFile":{"sha1":"5f1e8acedc065031aad553b710838eb366cfee9a","sha256":"8963a19fb992ad9a76576c5638fd68292cffb9aaac29eb8285f9abf6196a7dec","fileName":"MsSense.exe","filePath":"C:\\Program Files\\temp","fileSize":6136392,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null},"parentProcessImageFile":{"sha1":null,"sha256":null,"fileName":"services.exe","filePath":"C:\\Windows\\System32","fileSize":731744,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null},"userAccount":{"accountName":"SYSTEM","domainName":"NT AUTHORITY","userSid":"S-1-5-18","azureAdUserId":null,"userPrincipalName":null,"displayName":"System"}},{"@odata.type":"#microsoft.graph.security.registryKeyEvidence","createdDateTime":"2021-04-27T12:19:27.7211305Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"registryKey":"SYSTEM\\CONTROLSET001\\CONTROL\\WMI\\AUTOLOGGER\\SENSEAUDITLOGGER","registryHive":"HKEY_LOCAL_MACHINE","roles":[],"detailedRoles":[],"tags":[]}]} | |||
|
/test |
|
95.8% Coverage on New Code
0.0% Duplication on New Code
💚 Build Succeeded
History
|
efd6
approved these changes
Feb 5, 2024
|
Package m365_defender - 2.7.0 containing this change is available at https://epr.elastic.co/search?package=m365_defender |
nicpenning
reviewed
Feb 6, 2024
| The Microsoft 365 Defender integration collects logs for three types of events: Event, Incident and Log. | ||
| The Microsoft 365 Defender integration collects logs for four types of events: Alert, Event, Incident and Log. | ||
|
|
||
| **Alert:** This data streams leverages the [M365 Defender Streaming API](https://learn.microsoft.com/en-us/graph/api/resources/security-alert?view=graph-rest-1.0) to collect alerts including suspicious activities in a customer's tenant that Microsoft or partner security providers have identified and flagged for action. |
There was a problem hiding this comment.
Drop the "s" on data stream so it reads:
This data stream leverages the...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Type of change
What does this PR do?
1. Add Support of Alert Data Stream
2. Fixed the dashboard issue in Incident and Event Data Stream
Combined the aggregated alerts coming through the incident endpoint in the existing dashboard of Incident and created
new dashboard for the alert.
Checklist
changelog.ymlfile.All changes
How to test this PR locally
Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/m365_defender directory.
Run the following command to run tests.
elastic-package test -vRelated issues
Automated Test