-
Notifications
You must be signed in to change notification settings - Fork 333
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
migrating Athena function to use tf_lambda module (#1217)
* rename of athena function * updating terraform generation code to use tf_lambda module * updating tf_athena module to remove lambda code * updates for packaging, rollback, and deploy * misc updates related to config path renaming, etc * removing no-longer-used method (athena is default) * addressing PR feedback * adding more granular time prefix to athena client
- Loading branch information
1 parent
0e23761
commit 6df1ed2
Showing
39 changed files
with
372 additions
and
578 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
96 changes: 8 additions & 88 deletions
96
streamalert_cli/_infrastructure/modules/tf_athena/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,88 +1,8 @@ | ||
# StreamAlert Athena Terraform Module | ||
This Terraform module creates a Lambda function for refreshing Athena Partitions once new data is written to S3 | ||
|
||
## Components | ||
* A Python3.7 Lambda Function to perform a table refresh | ||
* IAM Role and Policy to allow for Athena execution | ||
* S3 bucket notifications | ||
* Lambda permissions | ||
|
||
## Example | ||
``` | ||
module "streamalert_athena" { | ||
source = "../modules/tf_athena" | ||
lambda_s3_bucket = "my-source-bucket" | ||
lambda_s3_key = "source/athena_partition_refresh_code.zip" | ||
athena_data_buckets = ["my-org-streamalerts"] | ||
} | ||
``` | ||
|
||
## Inputs | ||
<table> | ||
<tr> | ||
<th>Property</th> | ||
<th>Description</th> | ||
<th>Default</th> | ||
<th>Required</th> | ||
</tr> | ||
<tr> | ||
<td>lambda_handler</td> | ||
<td>The Python function entry point</td> | ||
<td>"main.handler"</td> | ||
<td>False</td> | ||
</tr> | ||
<tr> | ||
<td>lambda_timeout</td> | ||
<td>The max runtime in seconds for the Lambda function</td> | ||
<td>60 seconds</td> | ||
<td>False</td> | ||
</tr> | ||
<tr> | ||
<td>lambda_memory</td> | ||
<td>The memory allocation in MB for the Lambda function</td> | ||
<td>128MB</td> | ||
<td>False</td> | ||
</tr> | ||
<tr> | ||
<td>lambda_s3_bucket</td> | ||
<td>The name of the S3 bucket to store Lambda deployment packages</td> | ||
<td>None</td> | ||
<td>True</td> | ||
</tr> | ||
<tr> | ||
<td>lambda_s3_key</td> | ||
<td>The object in S3 containing the Lambda source</td> | ||
<td>None</td> | ||
<td>True</td> | ||
</tr> | ||
<tr> | ||
<td>lambda_log_level</td> | ||
<td>The current log level of the Lambda function</td> | ||
<td>info</td> | ||
<td>False</td> | ||
</tr> | ||
<tr> | ||
<td>current_version</td> | ||
<td>The currently published version of the Lambda production alias</td> | ||
<td>None</td> | ||
<td>True</td> | ||
</tr> | ||
<tr> | ||
<td>athean_data_buckets</td> | ||
<td>A list of buckets to monitor changes to for Hive partitioning</td> | ||
<td>None</td> | ||
<td>True</td> | ||
</tr> | ||
<tr> | ||
<td>prefix</td> | ||
<td>The resource prefix, normally an organizational name or descriptor</td> | ||
<td>None</td> | ||
<td>True</td> | ||
</tr> | ||
<tr> | ||
<td>schedule_expression</td> | ||
<td>The Cloudwatch-Lambda invocation schedule expression</td> | ||
<td>rate(10 minutes)</td> | ||
<td>False</td> | ||
</tr> | ||
</table> | ||
# Athena Partitioner Permissions | ||
This module adds IAM permissions and other specific resources needed in the Athena partitioner function: | ||
* Athena Database for querying alerts and historical data | ||
* S3 Bucket for storing the results of Athena queries | ||
* SQS Queue for receiving event notifications from S3 buckets | ||
* S3 Event Notifications for sending messages to SQS Queue when objects are created | ||
* KMS Key and Alias for encrypting/decrypting messages on SQS Queue | ||
* Permissions for sending data to SQS Queue and reading/writing data in S3 |
Oops, something went wrong.