title | description | author | ms.author | ms.topic | ms.date |
---|---|---|---|---|---|
Connect Microsoft Sentinel to Amazon Web Services to ingest AWS service log data |
Use the AWS connector to delegate Microsoft Sentinel access to AWS resource logs, creating a trust relationship between Amazon Web Services and Microsoft Sentinel. |
yelevin |
yelevin |
how-to |
01/31/2024 |
Use the Amazon Web Services (AWS) connectors to pull AWS service logs into Microsoft Sentinel. These connectors work by granting Microsoft Sentinel access to your AWS resource logs. Setting up the connector establishes a trust relationship between Amazon Web Services and Microsoft Sentinel. This is accomplished on AWS by creating a role that gives permission to Microsoft Sentinel to access your AWS logs.
This connector is available in two versions: the legacy connector for CloudTrail management and data logs, and the new version that can ingest logs from the following AWS services by pulling them from an S3 bucket (links are to AWS documentation):
- Amazon Virtual Private Cloud (VPC) - VPC Flow Logs
- Amazon GuardDuty - Findings
- AWS CloudTrail - Management and data events
- AWS CloudWatch - CloudWatch logs
This tab explains how to configure the AWS S3 connector. The process of setting it up has two parts: the AWS side and the Microsoft Sentinel side. Each side's process produces information used by the other side. This two-way authentication creates secure communication.
-
Make sure that the logs from your selected AWS service use the format accepted by Microsoft Sentinel:
- Amazon VPC: .csv file in GZIP format with headers; delimiter: space.
- Amazon GuardDuty: json-line and GZIP formats.
- AWS CloudTrail: .json file in a GZIP format.
- CloudWatch: .csv file in a GZIP format without a header. If you need to convert your logs to this format, you can use this CloudWatch lambda function.
-
You must have write permission on the Microsoft Sentinel workspace.
-
Install the Amazon Web Services solution from the Content Hub in Microsoft Sentinel. For more information, see Discover and manage Microsoft Sentinel out-of-the-box content.
This graphic and the following text show how the parts of this connector solution interact.
:::image type="content" source="media/connect-aws/s3-connector-architecture.png" alt-text="Screenshot of A W S S 3 connector architecture.":::
-
AWS services are configured to send their logs to S3 (Simple Storage Service) storage buckets.
-
The S3 bucket sends notification messages to the SQS (Simple Queue Service) message queue whenever it receives new logs.
-
The Microsoft Sentinel AWS S3 connector polls the SQS queue at regular, frequent intervals. If there is a message in the queue, it will contain the path to the log files.
-
The connector reads the message with the path, then fetches the files from the S3 bucket.
-
To connect to the SQS queue and the S3 bucket, Microsoft Sentinel uses a federated web identity provider (Microsoft Entra ID) for authenticating with AWS through OpenID Connect (OIDC), and assuming an AWS IAM role. The role is configured with a permissions policy giving it access to those resources.
-
In your AWS environment:
-
Configure your AWS service(s) to send logs to an S3 bucket.
-
Create a Simple Queue Service (SQS) queue to provide notification.
-
Create a web identity provider to authenticate users to AWS through OpenID Connect (OIDC).
-
Create an assumed role to grant permissions to users authenticated by the OIDC web identity provider to access your AWS resources.
-
Attach the appropriate IAM permissions policies to grant the assumed role access to the appropriate resources (S3 bucket, SQS).
We have made available, in our GitHub repository, a script that automates the AWS side of this process. See the instructions for automatic setup later in this document.
-
-
In Microsoft Sentinel:
- Enable and configure the AWS S3 Connector in the Microsoft Sentinel portal. See the instructions below.
To simplify the onboarding process, Microsoft Sentinel has provided a PowerShell script to automate the setup of the AWS side of the connector - the required AWS resources, credentials, and permissions.
The script takes the following actions:
-
Creates an OIDC web identity provider, to authenticate Microsoft Entra ID users to AWS.
-
Creates an IAM assumed role with the minimal necessary permissions, to grant OIDC-authenticated users access to your logs in a given S3 bucket and SQS queue.
-
Enables specified AWS services to send logs to that S3 bucket, and notification messages to that SQS queue.
-
If necessary, creates that S3 bucket and that SQS queue for this purpose.
-
Configures any necessary IAM permissions policies and applies them to the IAM role created above.
For Azure Government clouds, a specialized script creates a different OIDC web identity provider, to which it assigns the IAM assumed role.
- You must have PowerShell and the AWS CLI on your machine.
- Installation instructions for PowerShell
- Installation instructions for the AWS CLI (from AWS documentation)
To run the script to set up the connector, use the following steps:
-
From the Microsoft Sentinel navigation menu, select Data connectors.
-
Select Amazon Web Services S3 from the data connectors gallery.
If you don't see the connector, install the Amazon Web Services solution from the Content Hub in Microsoft Sentinel.
-
In the details pane for the connector, select Open connector page.
-
In the Configuration section, under 1. Set up your AWS environment, expand Setup with PowerShell script (recommended).
-
Follow the on-screen instructions to download and extract the AWS S3 Setup Script (link downloads a zip file containing the main setup script and helper scripts) from the connector page.
[!NOTE] For ingesting AWS logs into an Azure Government cloud, download and extract this specialized AWS S3 Gov Setup Script instead.
-
Before running the script, run the
aws configure
command from your PowerShell command line, and enter the relevant information as prompted. See AWS Command Line Interface | Configuration basics (from AWS documentation) for details. -
Now run the script. Copy the command from the connector page (under "Run script to set up the environment") and paste it in your command line.
-
The script will prompt you to enter your Workspace ID. This ID appears on the connector page. Copy it and paste it at the prompt of the script.
:::image type="content" source="media/connect-aws/aws-run-script.png" alt-text="Screenshot of command to run setup script and workspace ID." lightbox="media/connect-aws/aws-run-script.png":::
-
When the script finishes running, copy the Role ARN and the SQS URL from the script's output (see example in first screenshot below) and paste them in their respective fields in the connector page under 2. Add connection (see second screenshot below).
:::image type="content" source="media/connect-aws/aws-script-output.png" alt-text="Screenshot of output of A W S connector setup script." lightbox="media/connect-aws/aws-script-output.png":::
:::image type="content" source="media/connect-aws/aws-add-connection.png" alt-text="Screenshot of pasting the A W S role information from the script, to the S3 connector." lightbox="media/connect-aws/aws-add-connection.png":::
-
Select a data type from the Destination table drop-down list. This tells the connector which AWS service's logs this connection is being established to collect, and into which Log Analytics table it will store the ingested data. Then select Add connection.
Note
The script may take up to 30 minutes to finish running.
Microsoft recommends using the automatic setup script to deploy this connector. If for whatever reason you do not want to take advantage of this convenience, follow the steps below to set up the connector manually.
- Prepare your AWS resources
- Create an AWS assumed role and grant access to the AWS Sentinel account
- Add the AWS role and queue information to the S3 data connector
- Configure an AWS service to export logs to an S3 bucket
-
Create an S3 bucket to which you will ship the logs from your AWS services - VPC, GuardDuty, CloudTrail, or CloudWatch.
- See the instructions to create an S3 storage bucket in the AWS documentation.
-
Create a standard Simple Queue Service (SQS) message queue to which the S3 bucket will publish notifications.
- See the instructions to create a standard Simple Queue Service (SQS) queue in the AWS documentation.
-
Configure your S3 bucket to send notification messages to your SQS queue.
- See the instructions to publish notifications to your SQS queue in the AWS documentation.
-
In Microsoft Sentinel, select Data connectors from the navigation menu.
-
Select Amazon Web Services S3 from the data connectors gallery.
If you don't see the connector, install the Amazon Web Services solution from the Content Hub in Microsoft Sentinel. For more information, see Discover and manage Microsoft Sentinel out-of-the-box content.
-
In the details pane for the connector, select Open connector page.
-
Under Configuration, expand Setup with PowerShell script (recommended), then copy the External ID (Workspace ID) to your clipboard.
-
In a different browser window or tab, open the AWS console.
-
Create a web identity provider. Follow these instructions in the AWS documentation:
Creating OpenID Connect (OIDC) identity providers.Parameter Selection/Value Comments Client ID - Ignore this, you already have it. See Audience line below. Provider type OpenID Connect Instead of default SAML. Provider URL Commercial: sts.windows.net/33e01921-4d64-4f8c-a055-5bdaffd5e33d/
Government:sts.windows.net/cab8a31a-1906-4287-a0d8-4eef66b95f6e/
Thumbprint 626d44e704d1ceabe3bf0d53397464ac8080142c
If created in the IAM console, selecting Get thumbprint should give you this result. Audience Commercial: api://1462b192-27f7-4cb9-8523-0f4ecb54b47e
Government:api://d4230588-5f84-4281-a9c7-2c15194b28f7
-
Create an IAM assumed role. Follow these instructions in the AWS documentation:
Creating a role for web identity or OpenID Connect Federation.Parameter Selection/Value Comments Trusted entity type Web identity Instead of default AWS service. Identity provider Commercial: sts.windows.net/33e01921-4d64-4f8c-a055-5bdaffd5e33d/
Government:sts.windows.net/cab8a31a-1906-4287-a0d8-4eef66b95f6e/
The provider you created in the previous step. Audience Commercial: api://1462b192-27f7-4cb9-8523-0f4ecb54b47e
Government:api://d4230588-5f84-4281-a9c7-2c15194b28f7
The audience you defined for the identity provider in the previous step. Permissions to assign AmazonSQSReadOnlyAccess
AWSLambdaSQSQueueExecutionRole
AmazonS3ReadOnlyAccess
ROSAKMSProviderPolicy
- Additional policies for ingesting the different types of AWS service logs
For information on these policies, see the relevant AWS S3 connector permissions policies page, in the Microsoft Sentinel GitHub repository. Name "OIDC_MicrosoftSentinelRole" Choose a meaningful name that includes a reference to Microsoft Sentinel.
The name must include the exact prefixOIDC_
, otherwise the connector will not function properly. -
Edit the new role's trust policy and add another condition:
"sts:RoleSessionName": "MicrosoftSentinel_{WORKSPACE_ID)"
[!IMPORTANT] The value of the
sts:RoleSessionName
parameter must have the exact prefixMicrosoftSentinel_
, otherwise the connector will not function properly.The finished trust policy should look like this:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/sts.windows.net/cab8a31a-1906-4287-a0d8-4eef66b95f6e/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "sts.windows.net/cab8a31a-1906-4287-a0d8-4eef66b95f6e/:aud": "api://d4230588-5f84-4281-a9c7-2c15194b28f7", "sts:RoleSessionName": "MicrosoftSentinel_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" } } } ] }
XXXXXXXXXXXX
is your AWS Account ID.XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
is your Microsoft Sentinel workspace ID.
Update (save) the policy when you're done editing.
-
In the browser tab open to the AWS console, enter the Identity and Access Management (IAM) service and navigate to the list of Roles. Select the role you created above.
-
Copy the ARN to your clipboard.
-
Enter the Simple Queue Service, select the SQS queue you created, and copy the URL of the queue to your clipboard.
-
Return to your Microsoft Sentinel browser tab, which should be open to the Amazon Web Services S3 (Preview) data connector page. Under 2. Add connection:
- Paste the IAM role ARN you copied two steps ago into the Role to add field.
- Paste the URL of the SQS queue you copied in the last step into the SQS URL field.
- Select a data type from the Destination table drop-down list. This tells the connector which AWS service's logs this connection is being established to collect, and into which Log Analytics table it will store the ingested data.
- Select Add connection.
:::image type="content" source="media/connect-aws/aws-add-connection.png" alt-text="Screenshot of adding an A W S role connection to the S3 connector." lightbox="media/connect-aws/aws-add-connection.png":::
See Amazon Web Services documentation (linked below) for the instructions for sending each type of log to your S3 bucket:
-
Publish a VPC flow log to an S3 bucket.
[!NOTE] If you choose to customize the log's format, you must include the start attribute, as it maps to the TimeGenerated field in the Log Analytics workspace. Otherwise, the TimeGenerated field will be populated with the event's ingested time, which doesn't accurately describe the log event.
-
Export your GuardDuty findings to an S3 bucket.
[!NOTE]
-
In AWS, findings are exported by default every 6 hours. Adjust the export frequency for updated Active findings based on your environment requirements. To expedite the process, you can modify the default setting to export findings every 15 minutes. See Setting the frequency for exporting updated active findings.
-
The TimeGenerated field is populated with the finding's Update at value.
-
-
AWS CloudTrail trails are stored in S3 buckets by default.
-
Different types of logs can be stored in the same S3 bucket, but should not be stored in the same path.
-
Each SQS queue should point to one type of message, so if you want to ingest GuardDuty findings and VPC flow logs, you should set up separate queues for each type.
-
Similarly, a single SQS queue can serve only one path in an S3 bucket, so if for any reason you are storing logs in multiple paths, each path requires its own dedicated SQS queue.
Learn how to troubleshoot Amazon Web Services S3 connector issues.
This tab explains how to configure the AWS CloudTrail connector. The process of setting it up has two parts: the AWS side and the Microsoft Sentinel side. Each side's process produces information used by the other side. This two-way authentication creates secure communication.
Note
AWS CloudTrail has built-in limitations in its LookupEvents API. It allows no more than two transactions per second (TPS) per account, and each query can return a maximum of 50 records. Consequently, if a single tenant constantly generates more than 100 records per second in one region, backlogs and delays in data ingestion will result.
Currently, you can only connect your AWS Commercial CloudTrail to Microsoft Sentinel and not AWS GovCloud CloudTrail.
- You must have write permission on the Microsoft Sentinel workspace.
- Install the Amazon Web Services solution from the Content Hub in Microsoft Sentinel. For more information, see Discover and manage Microsoft Sentinel out-of-the-box content.
Note
Microsoft Sentinel collects CloudTrail management events from all regions. It is recommended that you do not stream events from one region to another.
Setting up this connector has two steps:
- Create an AWS assumed role and grant access to the AWS Sentinel account
- Add the AWS role information to the AWS CloudTrail data connector
-
In Microsoft Sentinel, select Data connectors from the navigation menu.
-
Select Amazon Web Services from the data connectors gallery.
If you don't see the connector, install the Amazon Web Services solution from the Content Hub in Microsoft Sentinel. For more information, see Discover and manage Microsoft Sentinel out-of-the-box content.
-
In the details pane for the connector, select Open connector page.
-
Under Configuration, copy the Microsoft account ID and the External ID (Workspace ID) to your clipboard.
-
In a different browser window or tab, open the AWS console. Follow the instructions in the AWS documentation for creating a role for an AWS account.
-
For the account type, instead of This account, choose Another AWS account.
-
In the Account ID field, enter the number 197857026523 (or paste it—the Microsoft account ID you copied in the previous step—from your clipboard). This number is Microsoft Sentinel's service account ID for AWS. It tells AWS that the account using this role is a Microsoft Sentinel user.
-
In the options, select Require external ID (do not select Require MFA). In the External ID field, paste your Microsoft Sentinel Workspace ID that you copied in the previous step. This identifies your specific Microsoft Sentinel account to AWS.
-
Assign the
AWSCloudTrailReadOnlyAccess
permissions policy. Add a tag if you want. -
Name the role with a meaningful name that includes a reference to Microsoft Sentinel. Example: "MicrosoftSentinelRole".
-
-
In the browser tab open to the AWS console, enter the Identity and Access Management (IAM) service and navigate to the list of Roles. Select the role you created above.
-
Copy the ARN to your clipboard.
-
Return to your Microsoft Sentinel browser tab, which should be open to the Amazon Web Services data connector page. In the Configuration section, paste the Role ARN into the Role to add field and select Add.
:::image type="content" source="media/connect-aws/aws-add-connection-ct.png" alt-text="Screenshot of adding an A W S role connection to the AWS connector." lightbox="media/connect-aws/aws-add-connection-ct.png":::
-
To use the relevant schema in Log Analytics for AWS events, search for AWSCloudTrail.
[!IMPORTANT] As of December 1, 2020, the AwsRequestId field has been replaced by the AwsRequestId_ field (note the added underscore). The data in the old AwsRequestId field will be preserved through the end of the customer's specified data retention period.
In this document, you learned how to connect to AWS resources to ingest their logs into Microsoft Sentinel. To learn more about Microsoft Sentinel, see the following articles:
- Learn how to get visibility into your data, and potential threats.
- Get started detecting threats with Microsoft Sentinel.
- Use workbooks to monitor your data.