AWS Security Analytics Bootstrap

AWS Security Analytics Bootstrap enables customers to perform security investigations on AWS service logs by providing an Amazon Athena analysis environment that's quick to deploy, ready to use, and easy to maintain.

If you're ready to deploy Athena Analytics Bootstrap, jump directly to Deploying AWS Security Analytics Bootstrap

AWS Security Analytics Bootstrap Use Cases

AWS Security Analytics Bootstrap is for Amazon Web Services (AWS) customers who need a quick method to setup Athena and perform investigations on AWS service logs archived in Amazon S3 buckets.

AWS Security Analytics Bootstrap is designed to be ready to use for common security investigation use-cases, quick to deploy, and easy to maintain. AWS Security Analytics Bootstrap provides AWS CloudFormation templates to quickly create a fully configured Athena analysis environment including an Amazon Athena Workgroup, AWS Glue Databases, AWS Glue Tables, and Demo Athena Queries. AWS Security Analytics Bootstrap supports common security investigation requirements including partitioning and searches across multiple accounts, regions, and dates. AWS Security Analytics Bootstrap uses Partition Projection with Amazon Athena to provide dynamic partitioning across accounts, regions, and dates without any additional infrastructure, code or frequent maintenance. Partitioning AWS service log data by account, region, and/or date will enable AWS customers to create targeted queries reducing their cost and query times.

Common Use Cases for AWS Security Analytics Bootstrap are When AWS Customers:

• Want to search AWS service logs natively in AWS
• Currently have no SIEM available/accessible
• Need to search logs beyond SIEM retention period
• Don't have the required AWS service logs indexed in the SIEM
• Need to investigate an AWS account which hasn't been centralizing its logs

AWS Service Logs Currently Supported by AWS Security Analytics Bootstrap

AWS Service Log	Event Type
AWS CloudTrail	AWS API Events (Management and Data Events)
Amazon Virtual Private Cloud (VPC) Flow Logs	Network Events
Amazon Route 53 DNS resolver query logs	DNS Resolution Events

NOTE: We will be adding support for additional AWS Service Logs commonly used in security investigations, please feel free to submit or upvote your requests in Issues

AWS Security Analytics Bootstrap Architecture Overview

Single Account Deployment	Cross-Account Deployment

Deploying AWS Security Analytics Bootstrap

Prerequisites

• AWS service logs (e.g. AWS CloudTrail, Amazon VPC Flow Logs, Amazon Route 53 Resolver Query Logs) must be delivered to Amazon S3 buckets unmodified in their native format
• For cross-account deployments bucket policies must be in place and objects need to be owned by the bucket account owner to enable cross-account access
• For logs encrypted via AWS KMS the AWS IAM principal(s) that will be used to submit Athena queries will need to have permissions for kms:Decrypt and kms:DescribeKey in their IAM policy and the KMS key policy will need to grant them the same access

Getting Started

The Athena Infrastructure CloudFormation Template will deploy a fully functional security analytics environment including:

Resource	Notes
Athena Workgroup	- Configured to provide encrypted output to a specified S3 location - Includes pre-configured demo queries as Named Queries
Glue Database	- Contains associated Glue Tables
Glue Tables	Standardized table schemas with dynamic partitions for account, region, and date for: - CloudTrail Logs - VPC Flow Logs - Route53 DNS Resolver Logs

Deployment time: ~10 minutes

Comments are provided in the CloudFormation Parameters section to assist with the parameters required for deployment, and a detailed walk-through of the deployment process is provided in the AWS Security Analytics Bootstrap Deployment Guide.

---

AWS Security Analytics Bootstrap Resources

Note: The Athena Infrastructure CloudFormation Template can be deployed by itself or in combination with any of the additional resources depending on customers' use case(s) and requirements.

Resource Type	Resource	Resource Provides	Cleanup/Removal Notes
AWS CloudFormation Template	Athena Infrastructure CloudFormation Template	Creates the ready-to-use Athena security analytics environment including: Athena Workgroup, Glue Database, Glue Tables, and demo Named Queries. Comments are provided in the CloudFormation Parameters section to walk customers through deployment or customers can review the AWS Security Analytics Bootstrap Deployment Guide for more detail.	All resources created by this template will be deleted when the CloudFormation Stack is deleted. This will not affect the source log data.
AWS CloudFormation Template	IAM Roles and Policies for Athena Admin and Athena Analyst	Creates IAM Roles and Policies for a Athena Admin and Athena Analyst Roles designed according to least privilege principals	All resources created by this template will be deleted when the CloudFormation Stack is deleted.
AWS CloudFormation Template	Enable flow logs	Enables VPC Flow Logs for the specified VPC, Subnet, or ENI with all fields through v5 in the order expected by Athena Bootstrap	The VPC Flow log configuration will be deleted when the CloudFormation Stack is deleted. Any logs created will need to be deleted separately from the target S3 bucket if desired.
CREATE TABLE SQL Statement	AWS CloudTrail Table Schema	Creates a Glue Table for CloudTrail Logs partitioned by account, region and date via Athena SQL query statement. This table is also created by the Athena Infrastructure CloudFormation Template; this SQL statement can be used to create a table in an existing Athena environment for adhoc deployment use cases. "TODO" comments are included above sections which need to be updated with customers' environment details.	The table can be deleted with the Athena query statement DROP TABLE <table name> (e.g. DROP TABLE cloudtrail)
CREATE TABLE SQL Statement	Amazon VPC Flow Logs Table Schema	Creates a Glue Table for VPC Flow Logs partitioned by account, region and date via Athena SQL query statement. This table is also created by the Athena Infrastructure CloudFormation Template; this SQL statement can be used to create a table in an existing Athena environment for adhoc deployment use cases. "TODO" comments are included above sections which need to be updated with customers' environment details.	The table can be deleted with the Athena query statement DROP TABLE <table name> (e.g. DROP TABLE vpcflowlogs)
CREATE TABLE SQL Statement	Amazon Route 53 Resolver Query Logs Table Schema	Creates a Glue Table for Route 53 DNS Resolver Logs partitioned by account, VPC ID and date via Athena SQL query statement. This table is also created by the Athena Infrastructure CloudFormation Template; this SQL statement can be used to create a table in an existing Athena environment for adhoc deployment use cases. "TODO" comments are included above sections which need to be updated with customers' environment details.	The table can be deleted with the Athena query statement DROP TABLE <table name> (e.g. DROP TABLE r53dns)
Demo Athena Queries	AWS CloudTrail Demo Queries	Demo Athena queries for CloudTrail Logs. These queries area also created in the Athena Workgroup as Named Queries by the Athena Infrastructure CloudFormation Template	N/A - No resources created
Demo Athena Queries	Amazon VPC Flow Log Demo Queries	Demo Athena queries for VPC Flow Logs. These queries area also created in the Athena Workgroup as Named Queries by the Athena Infrastructure CloudFormation Template	N/A - No resources created
Demo Athena Queries	Amazon Route 53 Resolver Query Log Demo Queries	Demo Athena queries for Route 53 DNS Resolver Logs. These queries area also created in the Athena Workgroup as Named Queries by the Athena Infrastructure CloudFormation Template	N/A - No resources created

Currently Out of Scope

• How data is provided to Amazon S3 buckets (e.g. configuration of logs)
• Optimization of the underlying data sources (e.g. merging small files, converting to Parquet/ORC columnar formats)
• Bucket policy updates for cross-account deployments
• AWS KMS policy updates are currently out of scope

Cost

By installing AWS Security Analytics Bootstrap, AWS customers may incur charges from the following services:

• Amazon Athena: https://aws.amazon.com/athena/pricing/
• Amazon S3: https://aws.amazon.com/s3/pricing/
• AWS KMS: https://aws.amazon.com/kms/pricing/

Related Resources

Resources from AWS

• [Tool] Assisted Log Enabler for AWS
• [Tool] AthenaGlueService Logs
• [Service] Amazon Detective
• [Doc] Running SQL Queries Using Amazon Athena
• [Doc] Querying AWS Service Logs
• [Blog] Analyze Security, Compliance, and Operational Activity Using AWS CloudTrail and Amazon Athena
• [Blog] Athena Performance Tips
• [Q&A] CTAS Bucketing Guidance
• [Blog] Amazon Route 53 Resolver Query Logs
• [Doc] Configure Amazon Route 53 Resolver Query Logs (choose Amazon S3 destination)
• [Doc] Security Reference Architecture: Log Archive Account

Third Party Resources

• [Guide] The Athena Guide
• [Doc] Presto Documentation (current)
• [Book] Presto The Definitive Guide (e-book)

Contributors

AWS Security Analytics Bootstrap stands on the shoulders of countless giants and has benefited from the assistance of MANY collaborators and contributors. Thanks to everyone who has helped or inspired the project so far and thanks in advance to any future contributions.

Many thanks for your contributions:

• Aaron Lennon
• Anna McAbee
• Bohan Li
• Brian Andrzejewski
• Brian Poole
• Casey Reniker
• Ross Warren
• Clayton Darnell
• Cydney Stude
• Freddy Kasprzykowski
• Jason Hurst
• Jonathon Poling
• Joshua McKiddy
• Justin Fry
• Kyle Dickinson
• Luis Maldonado
• Marc Luescher
• Matt Gurr
• Matt Helgen
• Matthew Harvey
• Pathik Shah
• Ravi Sankar Prasad Kadiri
• Richard Billington
• Ross Warren
• Srinivas Ananda Babu
• Theo Tolv

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.