[Filebeat] [Microsoft module] - Lack of ECS utilization

[defendable-forfot]

We are ingesting Microsoft ATP and M365 Defender data into our Elasticsearch for search, detection in Elastic Security, and visualization through Kibana. However, we have noticed a few specific fields where the Microsoft module does not optimally utilize ECS.

Note: we are running filebeat version 8.1.3, but have noticed that none of the newer releases solves all our issues. Issue elastic/beats#29859 has solved issues related to one specific field, but we still need more improvements.

Microsoft ATP

microsoft.defender_atp.evidence.userPrincipalName
    ECS fields: user.email | user.domain | user.name | related.user
    Suggestion: The userPrincipalName contains the email address of the user. With this it should be possible to populate the user.email field, and also do parsing to extract data into other relevant ECS fields.
microsoft.defender_atp.loggedOnUsers
    ECS fields: related.user
    Suggestion: Populate the related.user field with usernames that are added within the source field based on microsoft.defender_atp.loggedOnUsers.X.accountName
microsoft.defender_atp.evidence.ipAddress
    ECS fields: host.ip | related.ip
    Suggestion: related.ip is populated, but not host.ip. We would like to see this implemented.
microsoft.defender_atp.evidence.parentProcessFileName
    ECS fields: file.name | process.name
    Suggestion: file.name is populated, but not process.name. We would like to see this implemented.
microsoft.defender_atp.evidence.parentProcessFilePath
    ECS fields: file.path | process.executable
    Suggestion: file.path is populated, but not process.executable. We would like to see this implemented.
file.hash.*, related.hash
    ECS fields: file.hash.*, process.hash.*, related hash
    Suggestion: The file.hash.* fields and related hash fields are populated, but not the corresponding process.hash.* fields. We would like to see this implemented.
message
    ECS fields: message | rule.name
    Suggestion: The message field contains the expected data. We would like to see that this data is also populated into the rule.name field.

M365 Defender

microsoft.m365_defender.incidentUri
    ECS fields: cloud.account.id
    Suggestion: Tenant ID is missing in m365_defender documents, but it can be extracted from the incidentUri field based on the value in the tid paramater.
microsoft.m365_defender.incidentName
    ECS fields: message | rule.name
    Suggestion: (1) if the message field is used but incidentName exists, the data in incidentName should replace the data in the message field. (2) if the message is used but incidentName doesn’t exist, the data in the message field can remain the same.
microsoft.m365_defender.entities.accountName | microsoft.m365_defender.alerts.entities.accountName
    ECS fields: user.name | user.email | user.domain | related.user
    Suggestion: user.name and related.user fields contains an email address and not the value specified in the accountName field. The email address should be placed in the user.email field instead and the user.name field populated with the value in from accountName.
user.id
    ECS fields: user.id | related.user
    Suggestion: user.id is populated, but not related user. We would like to see this implemented.
microsoft.m365_defender.devices.*.loggedOnUsers | microsoft.m365_defender.alerts.devices.*.loggedOnUsers
    ECS fields: related.user
    Suggestion: We would like the related.user field to be populated based upon the accountName.
microsoft.m365_defender.alerts.entities.ipAddress |
microsoft.m365_defender.entities.ipAddress
    ECS fields: host.ip | related.ip
    Suggestion: The IP address is populated in related.ip, but not host.ip. We would like to see this implemented.
rule.description
    ECS fields: related.ip
    Suggestion: The related.ip field can in a lot of cases be populated by other IP adressess that are found within the rule.description field.
microsoft.m365_defender.entities.parentProcessFilePath
    ECS fields: file.path | process.executable
    Suggestion: file.path is populated but not the process.executable field. We would like to see this implemented.
microsoft.m365_defender.entities.parentProcessFileName
    ECS fields: file.name | process.name
    Suggestion: file.name is populated but not the process.name field. We would like to see this implemented.
file.hash.*, related.hash
    ECS fields: file.hash.*, process.hash.*, related hash
    Suggestion: The file.hash.* fields and related hash fields are populated, but not the corresponding process.hash.* fields. We would like to see this implemented.
microsoft.m365_defender.alerts.entities.detectionStatus
    ECS fields: event.action
    Suggestion: If the detectionStatus field is included in the document the event.action field should be populated with the corresponding value.
microsoft.m365_defender.incidentUri
    ECS fields: event.url
    Suggestion: The event.url field should be populated by the data in the .incidentUri field.

This is a copy of https://discuss.elastic.co/t/microsoft-filebeat-module-lack-of-ecs-utilization/315125, as I was recommended to post this as a GitHub issue instead.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[jamiehynds]

@vinit-elastic only assigning this to you to ensure these mappings are taking into account as part of the new M365D integration.

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[vinit-chauhan]

AFAIK, These fields are covered as part of our new M365 integration. However, @piyush-elastic would you mind confirming and closing this issue?

[jamiehynds]

Closing as our M365 Defender integration address these ECS mapping improvements.