Merge branch 'elastic:main' into main

.github/CODEOWNERS

@@ -222,6 +222,7 @@
 /packages/lmd @elastic/ml-ui @elastic/sec-applied-ml
 /packages/log @elastic/elastic-agent-data-plane
 /packages/logstash @elastic/stack-monitoring
+/packages/lumos @elastic/security-service-integrations
 /packages/lyve_cloud @elastic/security-service-integrations
 /packages/m365_defender @elastic/security-service-integrations
 /packages/mattermost @elastic/security-service-integrations

packages/azure/_dev/build/docs/README.md

@@ -1,6 +1,6 @@
 # Azure Logs Integration
 
-The Azure Logs integration collects logs for specific Azure services like Azure Active Directory (Sign-in, Audit, Identity Protection, and Provisioning logs), Azure Spring Apps, Azure Firewall, and several others using the Activity and Platform logs.
+The Azure Logs integration collects logs for specific Azure services like Azure Active Directory (Sign-in, Audit, Identity Protection, and Provisioning logs), Azure Spring Apps, Azure Firewall, Microsoft Graph Activity, and several others using the Activity and Platform logs.
 
 You can then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
 
@@ -16,7 +16,7 @@ fail to start due to an exceed quota limit.
 The Azure Logs integration collects logs.
 
 **Logs** help you keep a record of events that happen on your Azure account.
-Log data streams collected by the Azure Logs integration include Activity, Platform, Active Directory (Sign-in, Audit, Identity Protection, Provisioning), and Spring Apps logs.
+Log data streams collected by the Azure Logs integration include Activity, Platform, Active Directory (Sign-in, Audit, Identity Protection, Provisioning), Microsoft Graph Activity, and Spring Apps logs.
 
 ## Requirements
 

packages/azure/_dev/build/docs/graphactivitylogs.md

@@ -0,0 +1,84 @@
+# Microsoft Graph Activity Logs
+
+Microsoft Graph Activity Logs provide an audit trail of all HTTP requests that the Microsoft Graph service has received and processed for a tenant. Microsoft Graph Activity Logs gives full visibility into all transactions made by applications and other API clients that you have consented to in the tenant. Refer to [Microsoft Graph Activity Common Usecases](https://learn.microsoft.com/en-us/graph/microsoft-graph-activity-logs-overview#common-use-cases-for-microsoft-graph-activity-logs) for more use cases.
+
+Tenant administrators can configure the collection and storage destinations of Microsoft Graph Activity Logs through Diagnostic Setting in the Entra Portal. This integration uses Azure Event Hubs destination to stream Microsoft Graph Activity Logs to Elastic.
+
+## Requirements and Setup
+
+### Prerequisites
+
+Following privileges are required to collect Microsoft Graph Activity Logs:
+- A Microsoft Entra ID P1 or P2 tenant license in your tenant.
+- A `Security Administrator` or `Global Administrator` Microsoft Entra ID role to configure the diagnostic settings.
+Refer to [Microsoft Graph Prerequisites](https://learn.microsoft.com/en-us/graph/microsoft-graph-activity-logs-overview#prerequisites) for more information on required privileges.
+
+### Setup
+
+Refer to the [Azure Logs](https://docs.elastic.co/integrations/azure) page for more information about setting up and using this integration.
+
+### Limitations
+
+- Activities of multi-tenant applications belonging to another tenant are not available.
+- In few rare cases, events might take up to 2 hours to be delivered to Event Hubs.
+Refer to [Microsoft Graph Activity Limitations](https://learn.microsoft.com/en-us/graph/microsoft-graph-activity-logs-overview#limitations) for more information.
+
+## Settings
+
+`eventhub` :
+  _string_
+It is a fully managed, real-time data ingestion service. Elastic recommends using only letters, numbers, and the hyphen (-) character for Event Hub names to maximize compatibility. You _can_ use existing Event Hubs having underscores (_) in the Event Hub name; in this case, the integration will replace underscores with hyphens (-) when it uses the Event Hub name to create dependent Azure resources behind the scenes (e.g., the storage account container to store Event Hub consumer offsets). Elastic also recommends using a separate event hub for each log type as the field mappings of each log type differ.
+Default value `insights-operational-logs`.
+
+`consumer_group` :
+_string_
+ The publish/subscribe mechanism of Event Hubs is enabled through consumer groups. A consumer group is a view (state, position, or offset) of an entire event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets.
+Default value: `$Default`
+
+`connection_string` :
+_string_
+The connection string required to communicate with Event Hubs, steps [here](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string).
+
+A Blob Storage account is required in order to store/retrieve/update the offset or state of the eventhub messages. This means that after stopping the filebeat azure module it can start back up at the spot that it stopped processing messages.
+
+`storage_account` :
+_string_
+The name of the storage account the state/offsets will be stored and updated.
+
+`storage_account_key` :
+_string_
+The storage account key, this key will be used to authorize access to data in your storage account.
+
+`storage_account_container` :
+_string_
+The storage account container where the integration stores the checkpoint data for the consumer group. It is an advanced option to use with extreme care. You MUST use a dedicated storage account container for each Azure log type (activity, sign-in, audit logs, and others). DO NOT REUSE the same container name for more than one Azure log type. See [Container Names](https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata#container-names) for details on naming rules from Microsoft. The integration generates a default container name if not specified. 
+
+`resource_manager_endpoint` :
+_string_
+Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different Azure environment.
+
+Resource manager endpoints:
+
+```text
+# Azure ChinaCloud
+https://management.chinacloudapi.cn/
+
+# Azure GermanCloud
+https://management.microsoftazure.de/
+
+# Azure PublicCloud 
+https://management.azure.com/
+
+# Azure USGovernmentCloud
+https://management.usgovcloudapi.net/
+```
+
+## Logs
+
+### graphactivitylogs
+
+The `graphactivitylogs` data stream of the Azure Logs package will collect Microsoft Graph activity events that have been streamed through an azure event hub.
+
+{{event "graphactivitylogs"}}
+
+{{fields "graphactivitylogs"}}

packages/azure/changelog.yml

@@ -1,3 +1,8 @@
+- version: "1.10.0"
+  changes:
+    - description: Add Microsoft Graph Activity Logs
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/9314
 - version: "1.9.2"
   changes:
     - description: Add docs for running the integration behind a firewall.

packages/azure/data_stream/graphactivitylogs/_dev/test/pipeline/test-activitylogs-raw.log

@@ -0,0 +1,5 @@
+{"Level":4,"callerIpAddress":"2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6","category":"MicrosoftGraphActivityLogs","correlationId":"f7739da0-e6d1-4e3f-985a-64937fbge347","durationMs":462680,"location":"West Europe","operationName":"Microsoft Graph Activity","operationVersion":"v1.0","properties":{"apiVersion":"v1.0","appId":"a5a68e12-268a-3c91-a5e2-b9254e67hb29","atContent":"","clientAuthMethod":"2","clientRequestId":"2fe56789-a848-4c93-9d2c-5675972aejk9","durationMs":462680,"identityProvider":"https://sts.windows.net/a140785b-418d-4344-a4ec-8e9648919gdb/","ipAddress":"2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6","location":"West Europe","operationId":"f7739da0-e6d1-4e3f-985a-64937fbge347","requestId":"f7739da0-e6d1-4e3f-985a-64937fbge347","requestMethod":"GET","requestUri":"https://graph.microsoft.com/v1.0/organization/a140785b-418d-4344-a4ec-8e9648919gdb?$select=Id,DisplayName,CreatedDateTime,ReplicationScope,Country,CountryLetterCode,DataBoundary","responseSizeBytes":364,"responseStatusCode":200,"roles":"Organization.Read.All User.Read.All","scopes":null,"servicePrincipalId":"f2aq4c71-31e3-5066-92g3-4b3dfbav50f0","signInActivityId":"sign-in_ActivityId","tenantId":"a140785b-418d-4344-a4ec-8e9648919gdb","timeGenerated":"2024-03-07T11:47:00.6442361Z","tokenIssuedAt":"2024-03-07T11:42:00Z","userAgent":"python-requests/2.27.1","userId":null,"wids":"1997b4d3-0g8d-90cb-bhj8-d89n3121e90"},"resourceId":"/TENANTS/A140785B-418D-4344-A4EC-8E9648919GDB/PROVIDERS/MICROSOFT.AADIAM","resultSignature":"200","tenantId":"a140785b-418d-4344-a4ec-8e9648919gdb","time":"2024-03-07T11:47:00.6442361Z"}
+{"Level":4,"callerIpAddress":"81.2.69.143","category":"MicrosoftGraphActivityLogs","correlationId":"f7739jk0-e6d1-4e3f-985a-64937fbge367","durationMs":453011,"location":"Germany West Central","operationName":"Microsoft Graph Activity","operationVersion":"v1.0","properties":{"apiVersion":"v1.0","appId":"a5a68e12-268a-3c91-a5e2-b9254e67hb29","atContent":"","clientAuthMethod":"2","clientRequestId":"2fe56790-a848-4c83-9d2c-5675972aejk9","durationMs":453011,"identityProvider":"https://sts.windows.net/aa30985b-427d-4434-b4dc-8f9040719adb/","ipAddress":"81.2.69.143","location":"Germany West Central","operationId":"f7739jk0-e6d1-4e3f-985a-64937fbge367","requestId":"f7739jk0-e6d1-4e3f-985a-64937fbge367","requestMethod":"GET","requestUri":"https://graph.microsoft.com/v1.0/me/photos/96x96/$value","responseSizeBytes":294,"responseStatusCode":404,"roles":null,"scopes":"email openid Organization.Read.All Policy.ReadWrite.ApplicationConfiguration profile User.Read","servicePrincipalId":null,"signInActivityId":"sign-in_ActivityId","tenantId":"aa30985b-427d-4434-b4dc-8f9040719adb","timeGenerated":"2024-03-07T10:35:31.9597832Z","tokenIssuedAt":"2024-03-07T10:30:30Z","userAgent":"","userId":"b37ec517-0a34-4266-b627-f7bb0d679d70","wids":"1997b4d3-0g8d-90cb-bhj5-d80n3122e98 1997b4d3-0f8d-76cb-bhj5-d80n3122e98"},"resourceId":"/TENANTS/AA30985B-427D-4434-B4DC-8F9040719ADB/PROVIDERS/MICROSOFT.AADIAM","resultSignature":"404","tenantId":"aa30985b-427d-4434-b4dc-8f9040719adb","time":"2024-03-07T10:35:31.9597832Z"}
+{"Level":4,"callerIpAddress":"81.2.69.144","category":"MicrosoftGraphActivityLogs","correlationId":"f7739da0-e6d1-4e3f-875a-64934fbge347","durationMs":16688471,"location":"UK South","operationName":"Microsoft Graph Activity","operationVersion":"beta","properties":{"apiVersion":"beta","appId":"a5a68e12-268a-3c91-a5f2-b9254e67hb28","atContent":"","clientAuthMethod":"0","clientRequestId":"2fe56789-a848-4c93-9s2c-5675972aghk9","durationMs":16688471,"identityProvider":null,"ipAddress":"81.2.69.144","location":"UK South","operationId":"f7739da0-e6d1-4e3f-875a-64934fbge347","requestId":"f7739da0-e6d1-4e3f-875a-64934fbge347","requestMethod":"GET","requestUri":"https://graph.microsoft.com/beta//users/7ef3c2ad-d52l-4a89-8cf9-c30178181027/photos/48x48/$value","responseSizeBytes":0,"responseStatusCode":404,"roles":null,"scopes":"AdministrativeUnit.ReadWrite.All AuditLog.Read.All Directory.AccessAsUser.All Directory.Write.Restricted email openid Organization.Read.All Policy.ReadWrite.Authorization profile User.EnableDisableAccount.All User.ReadWrite.All","servicePrincipalId":null,"signInActivityId":"signin_ActivityId","tenantId":"ab47545b-420e-46fg-c4dc-8f7697k1aadb","timeGenerated":"2024-03-07T16:42:22.84914Z","tokenIssuedAt":"2024-03-07T16:37:20Z","userAgent":"","userId":"285e0849-a706-4a9a-9eb1-f4e21cc78793","wids":"1997b4d3-0g8d-90cb-bhj5-d80n3122e98 1997b4d3-0g8d-90cb-bhj5-d80n3122e99 1997b4d3-0g8d-90cb-bhj5-d80n3122e80 1997b4d3-0g8d-90cb-bhj5-d80n3122e83"},"resourceId":"/TENANTS/AB47545B-420E-46FG-C4DC-8F7697K1AADB/PROVIDERS/MICROSOFT.AADIAM","resultSignature":"404","tenantId":"ab47545b-420e-46fg-c4dc-8f7697k1aadb","time":"2024-03-07T16:42:22.8491400Z"}
+{"Level":4,"callerIpAddress":"2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6","category":"MicrosoftGraphActivityLogs","correlationId":"f7749da0-e6g1-4f3f-975a-64937fbge347","durationMs":846544,"location":"UK South","operationName":"Microsoft Graph Activity","operationVersion":"beta","properties":{"apiVersion":"beta","appId":"a5a68e32-269a-3c91-a5e2-b9254e67hb29","atContent":"","clientAuthMethod":"0","clientRequestId":"2fe56789-a848-4c93-9d2d-5675972ardk9","durationMs":846544,"identityProvider":null,"ipAddress":"2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6","location":"UK South","operationId":"f7749da0-e6g1-4f3f-975a-64937fbge347","requestId":"f7749da0-e6g1-4f3f-975a-64937fbge347","requestMethod":"GET","requestUri":"https://graph.microsoft.com/beta/devices?$select=displayName,id,deviceId\\u0026$search=%22displayName:a%22\\u0026$top=30","responseSizeBytes":0,"responseStatusCode":200,"roles":null,"scopes":"AccessReview.ReadWrite.All AuditLog.Read.All ChangeManagement.Read.All ConsentRequest.Create ConsentRequest.Read ConsentRequest.ReadApprove.All ConsentRequest.ReadWrite.All CustomSecAttributeAuditLogs.Read.All Directory.AccessAsUser.All Directory.Read.All Directory.ReadWrite.All Directory.Write.Restricted DirectoryRecommendations.Read.All DirectoryRecommendations.ReadWrite.All email EntitlementManagement.Read.All Group.ReadWrite.All IdentityProvider.ReadWrite.All IdentityRiskEvent.ReadWrite.All IdentityRiskyServicePrincipal.ReadWrite.All IdentityRiskyUser.ReadWrite.All IdentityUserFlow.Read.All LifecycleWorkflows.ReadWrite.All openid Policy.Read.All Policy.Read.IdentityProtection Policy.ReadWrite.AuthenticationFlows Policy.ReadWrite.AuthenticationMethod Policy.ReadWrite.ConditionalAccess Policy.ReadWrite.ExternalIdentities Policy.ReadWrite.IdentityProtection Policy.ReadWrite.MobilityManagement profile Reports.Read.All RoleManagement.ReadWrite.Directory RoleManagement.ReadWrite.Exchange SecurityEvents.ReadWrite.All TrustFrameworkKeySet.Read.All User.Export.All User.ReadWrite.All UserAuthenticationMethod.ReadWrite.All","servicePrincipalId":null,"signInActivityId":"signin_Activity-Id","tenantId":"ab47545b-420e-46fg-c4dc-8f7697k1aadb","timeGenerated":"2024-03-07T16:42:12.0485843Z","tokenIssuedAt":"2024-03-07T16:36:51Z","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0","userId":"285e0849-a706-4a9a-9eb1-f4e21cc78793","wids":"a207b4d3-0g8d-90cb-bhj5-d80n3122e67 a207b4d3-0g8d-90cb-bhj5-d80n3122e69 a207b4d3-0g8d-90cb-bhj5-d80n3122e89"},"resourceId":"/TENANTS/AB47545B-420E-46FG-C4DC-8F7697K1AADB/PROVIDERS/MICROSOFT.AADIAM","resultSignature":"200","tenantId":"ab47545b-420e-46fg-c4dc-8f7697k1aadb","time":"2024-03-07T16:42:12.0485843Z"}
+{"Level":4,"callerIpAddress":"81.2.69.143","category":"MicrosoftGraphActivityLogs","correlationId":"f7839da0-e7d1-4e4f-985a-64937fbge347","durationMs":1100725,"location":"France Central","operationName":"Microsoft Graph Activity","operationVersion":"v1.0","properties":{"apiVersion":"v1.0","appId":"a5a68e32-269a-3c91-a5e2-b9254e67hb29","atContent":"","clientAuthMethod":"2","clientRequestId":"2fe58790-a848-4a93-9d2c-5645972aejk9","durationMs":1100725,"identityProvider":"https://sts.windows.net/ab30785b-417f-42a4-b5dc-8f9051718acb/","ipAddress":"81.2.69.143","location":"France Central","operationId":"f7839da0-e7d1-4e4f-985a-64937fbge347","requestId":"f7839da0-e7d1-4e4f-985a-64937fbge347","requestMethod":"GET","requestUri":"https://graph.microsoft.com/v1.0/directoryRoles","responseSizeBytes":4300,"responseStatusCode":200,"roles":"Application.Read.All Domain.Read.All GroupMember.Read.All LicenseAssignment.ReadWrite.All Organization.Read.All Policy.Read.ConditionalAccess RoleManagement.Read.Directory Team.ReadBasic.All TeamsTab.Create TeamsTab.Read.All TeamsTab.ReadWrite.All User.Read.All","scopes":null,"servicePrincipalId":"f2aq4c71-31e3-5065-91g3-4b2dfbsv50fg","signInActivityId":"sign-in_ActivityId","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2024-03-07T10:24:44.7939418Z","tokenIssuedAt":"2024-03-07T10:19:44Z","userAgent":"","userId":null,"wids":"a207b4d3-0g8d-90cb-bhj5-d80n3121e69"},"resourceId":"/TENANTS/AB30785B-417F-42A4-B5DC-8F9051718ACB/PROVIDERS/MICROSOFT.AADIAM","resultSignature":"200","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","time":"2024-03-07T10:24:44.7939418Z"}