Merge pull request #2 from vmalinics0/vm

added sns, kms, multi-rds, multi-events

.gitignore

@@ -0,0 +1,2 @@
+
+*.swp

README.md

@@ -6,7 +6,7 @@ Terraform module that deploys Lambda functions that take care of triggering and
 ## Design
 A Lambda function takes care of triggering the RDS Start Export Task for the given database name. The snapshots will be exported to the given S3 bucket.
 
-Another Lambga function is only interested in RDS Export Task events that match a given database name. Whenever a match is detected, a message will be published in the given SNS topic which you can use to trigger other components. E.g. a Lambda function that sends notifications to Slack.
+Another Lambda function is only interested in RDS Export Task events that match a given database name. Whenever a match is detected, a message will be published in the given SNS topic which you can use to trigger other components. E.g. a Lambda function that sends notifications to Slack.
 
 A single CloudWatch Event Rule takes care of listening for RDS Snapshots Events in order to call the aforementioned Lambda functions.
 
@@ -15,9 +15,11 @@ A single CloudWatch Event Rule takes care of listening for RDS Snapshots Events
 </div>
 
 ## Important considerations
+* Please note, that only customer managed keys (CMK) are allowed.
+* Either `customer_kms_key_arn` provided key is used for exported snapshots encryption or new CMK created with `create_customer_kms_key` enabled
+* Since the module (optionally) creates its own KMS CMK, keep that in mind regarding KMS pricing; not only regarding the pricing of a single key, but also things like key rotations/versions and KMS API requests.
 * The module requires you to provide the S3 bucket that will be used for storing the exported snapshots. The good thing about this is that you are able to configure the bucket in any way you need. E.g. replication, lifecycle, locking, and so on.
-* The module creates a KMS Key (CMK) which is used for encrypting the exported snapshots on S3. The reason for the module not yet supporting passing your own CMK is that the key needs to grant a number of permissions to a role that is also created by this module. If providing your own key was supported, an specific execution order would be required: create the module by passing the key, get the Lambda role from the module's output and update the key permissions to grant it specific actions. So the orchestration becomes complicated.
-* Since the module creates its own KMS CMK, keep that in mind regarding KMS pricing; not only regarding the pricing of a single key but also things like key rotations/versions and KMS API requests.
+* The module can create an export monitor SNS notification topic, also existing SNS topics are supported via `notifications_topic_arn` variable.
 
 ## Requirements
 
@@ -50,22 +52,27 @@ No requirements.
 | [aws_lambda_permission.snsCanTriggerMonitorExportTask](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_permission) | resource |
 | [aws_lambda_permission.snsCanTriggerStartExportTask](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_permission) | resource |
 | [aws_sns_topic.rdsSnapshotsEvents](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sns_topic) | resource |
+| [aws_sns_topic.exportMonitorNotifications](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sns_topic) | resource |
 | [aws_sns_topic_policy.default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sns_topic_policy) | resource |
 | [aws_sns_topic_subscription.lambdaRdsSnapshotToS3Exporter](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sns_topic_subscription) | resource |
 | [aws_sns_topic_subscription.lambdaRdsSnapshotToS3Monitor](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sns_topic_subscription) | resource |
 | [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
+| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
-| <a name="input_database_name"></a> [database\_name](#input\_database\_name) | The name of the database whose snapshots we want to export to S3. | `string` | `null` | no |
+| <a name="input_create_customer_kms_key"></a> [create\_customer\_kms\_key](#input\_create\_customer\_kms\_key) | Create customer managed KMS key which is used for encrypting the exported snapshots on S3. If set to `false`, then `customer_kms_key_arn` is used. | `bool` | `false` | no |
+| <a name="input_create_notifications_topic"></a> [create\_notifications\_topic](#input\_create\_notifications\_topic) | Create new SNS notifications topic which will be used for publishing notifications messages. | `bool` | `true` | no |
+| <a name="input_customer_kms_key_arn"></a> [customer\_kms\_key\_arn](#input\_customer\_kms\_key\_arn) | The ARN of customer managed key used for RDS export encryption. Mandatory if `create_customer_kms_key` is set to `false`. Ex: `"arn:aws:kms:<region>:<accountID>:key/<key-id>"` | `string` | `null` | no |
+| <a name="input_database_names"></a> [database\_names](#input\_database\_names) | The names of the databases whose snapshots we want to export to S3. Comma-separated values), ex: `"db-cluster1, db-cluster2"` | `string` | `null` | yes |
 | <a name="input_log_level"></a> [log\_level](#input\_log\_level) | The log level of the Lambda function. | `string` | `"INFO"` | no |
-| <a name="input_notifications_topic_arn"></a> [notifications\_topic\_arn](#input\_notifications\_topic\_arn) | The ARN of an SNS Topic which will be used for publishing notifications messages. | `string` | `null` | no |
+| <a name="input_notifications_topic_arn"></a> [notifications\_topic\_arn](#input\_notifications\_topic\_arn) | The ARN of an SNS Topic which will be used for publishing notifications messages. Required if `create_notifications_topic` is set to `false`. | `string` | `null` | no |
 | <a name="input_prefix"></a> [prefix](#input\_prefix) | Prefix that will be used for naming resources. | `string` | `null` | no |
-| <a name="input_rds_event_id"></a> [rds\_event\_id](#input\_rds\_event\_id) | RDS (CloudWatch) Event ID that will trigger the calling of RDS Start Export Task API:<br>- Automated snapshots of Aurora RDS: RDS-EVENT-0169<br>- Automated snapshots of non-Aurora RDS: RDS-EVENT-0091<br>Only automated backups of either RDS Aurora and RDS non-Aurora are supported.<br>Ref: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.Messages.html#USER_Events.Messages.snapshot | `string` | n/a | yes |
-| <a name="input_snapshots_bucket_arn"></a> [snapshots\_bucket\_arn](#input\_snapshots\_bucket\_arn) | The ARN of the bucket where the RDS snapshots will be exported to. | `string` | `null` | no |
-| <a name="input_snapshots_bucket_name"></a> [snapshots\_bucket\_name](#input\_snapshots\_bucket\_name) | The name of the bucket where the RDS snapshots will be exported to. | `string` | `null` | no |
+| <a name="input_rds_event_ids"></a> [rds\_event\_id](#input\_rds\_event\_ids) | RDS (CloudWatch) Event IDs that will trigger the calling of RDS Start Export Task API:<br>- Automated snapshots of Aurora RDS: RDS-EVENT-0169<br>- Automated snapshots of non-Aurora RDS: RDS-EVENT-0091<br>Only automated backups of either RDS Aurora and RDS non-Aurora are supported.<br>Ref: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.Messages.html#USER_Events.Messages.snapshot<br>Ref: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_Events.Messages.html#USER_Events.Messages.cluster-snapshot. | `string` | `"RDS-EVENT-0091, RDS-EVENT-0169"` | no |
+| <a name="input_snapshots_bucket_name"></a> [snapshots\_bucket\_name](#input\_snapshots\_bucket\_name) | The name of the bucket where the RDS snapshots will be exported to. | `string` | `null` | yes |
+| <a name="input_snapshots_bucket_prefix"></a> [snapshots\_bucket\_prefix](#input\_snapshots\_bucket\_prefix) | The Amazon S3 bucket prefix to use as the file name and path of the exported snapshot. For example, use the prefix `"exports/2019/"`. | `string` | `null` | yes |
 | <a name="input_tags"></a> [tags](#input\_tags) | (Optional) A mapping of tags to assign to the bucket. | `map(string)` | `{}` | no |
 
 ## Outputs
@@ -74,7 +81,8 @@ No requirements.
 |------|-------------|
 | <a name="output_monitor_export_task_lambda_function_arn"></a> [monitor\_export\_task\_lambda\_function\_arn](#output\_monitor\_export\_task\_lambda\_function\_arn) | Start Export Task Monitor Lambda Function ARN |
 | <a name="output_monitor_export_task_lambda_role_arn"></a> [monitor\_export\_task\_lambda\_role\_arn](#output\_monitor\_export\_task\_lambda\_role\_arn) | Start Export Task Monitor Lambda Role ARN |
+| <a name="output_snapshots_events_export_monitor_sns_topics_arn"></a> [snapshots\_events\_export\_monitor\_sns\_topics\_arn](#output\_snapshots\_events_export\_monitor\_sns\_topics\_arn) | RDS Snapshots Export Monitor Events SNS Topics ARN |
 | <a name="output_snapshots_events_sns_topics_arn"></a> [snapshots\_events\_sns\_topics\_arn](#output\_snapshots\_events\_sns\_topics\_arn) | RDS Snapshots Events SNS Topics ARN |
 | <a name="output_snapshots_export_encryption_key_arn"></a> [snapshots\_export\_encryption\_key\_arn](#output\_snapshots\_export\_encryption\_key\_arn) | Snapshots Export Encryption Key ARN |
 | <a name="output_start_export_task_lambda_function_arn"></a> [start\_export\_task\_lambda\_function\_arn](#output\_start\_export\_task\_lambda\_function\_arn) | Start Export Task Lambda Function ARN |
-| <a name="output_start_export_task_lambda_role_arn"></a> [start\_export\_task\_lambda\_role\_arn](#output\_start\_export\_task\_lambda\_role\_arn) | Start Export Task Lambda Role ARN |
+| <a name="output_start_export_task_lambda_role_arn"></a> [start\_export\_task\_lambda\_role\_arn](#output\_start\_export\_task\_lambda\_role\_arn) | Start Export Task Lambda Role ARN |

cloudwatch.tf

@@ -2,7 +2,7 @@
 # Create an event rule to listen for RDS DB Cluster Snapshot Events
 #
 resource "aws_cloudwatch_event_rule" "rdsSnapshotCreation" {
-  name        = "${var.prefix}-rds-snapshot-creation"
+  name        = "${local.prefix}rds-snapshot-creation"
   description = "RDS Snapshot Creation"
 
   event_pattern = <<PATTERN
@@ -15,6 +15,8 @@ resource "aws_cloudwatch_event_rule" "rdsSnapshotCreation" {
   ]
 }
 PATTERN
+
+  tags = merge({ Name = "${local.prefix}rds-snapshot-creation" }, var.tags)
 }
 
 #

data.tf

@@ -2,3 +2,5 @@
 # Current account data
 #
 data "aws_caller_identity" "current" {}
+
+data "aws_region" "current" {}

examples/complete/main.tf

@@ -1,8 +1,8 @@
 locals {
   bucket_name = "example-rds-exported-snapshots"
   tags = {
-    Name        = "example"
-    Terraform   = "true"
+    Name      = "example"
+    Terraform = "true"
   }
 }
 
@@ -13,27 +13,41 @@ module "rds_export_to_s3" {
   source = "../../"
 
   # Set a prefix for naming resources
-  prefix = "aurora-mysql"
+  #prefix = "binbashar"
 
   # Which RDS snapshots should be exported?
-  database_name = "example-aurora-mysql-database"
-
-  # Which RDS snapshots events should be included (RDS Aurora or RDS non-Aurora)?
-  rds_event_id = "RDS-EVENT-0169"
+  database_names = "test1-aurora-mysql-cluster, test2-aurora-mysql-cluster"
 
   # Which bucket will store the exported snapshots?
   snapshots_bucket_name = module.bucket.s3_bucket_id
-  snapshots_bucket_arn = module.bucket.s3_bucket_arn
+  #snapshots_bucket_name = "export-bucket-name"
+
+  # To group objects in a bucket, S3 uses a prefix before object names. The forward slash (/) in the prefix represents a folder.
+  snapshots_bucket_prefix = "rds_snapshots/"
+
+  # Which RDS snapshots events should be included (RDS Aurora or/and RDS non-Aurora)?
+  #rds_event_ids = "RDS-EVENT-0091, RDS-EVENT-0169"
+
+  # Create customer managed key or use default AWS S3 managed key. If set to 'false', then 'customer_kms_key_arn' is used.
+  create_customer_kms_key = false
 
-  # Which topic should receive notifications about exported snapshots events?
-  notifications_topic_arn = "arn:aws:sns:us-east-1:000000000000:sns-topic-slack-notifications"
+  # Provide CMK if 'create_customer_kms_key = false'
+  customer_kms_key_arn = "arn:aws:kms:eu-west-2:123456789:alias/kms-rds"
+
+  # SNS topic for export monitor notifications
+  create_notifications_topic = true
+
+  # Which topic should receive notifications about exported snapshots events? Only required if 'create_notifications_topic = false'
+  #notifications_topic_arn = "arn:aws:sns:us-east-1:000000000000:sns-topic-slack-notifications"
 
   # Set the logging level
   # log_level = "DEBUG"
 
   tags = local.tags
+  #tags = { Deployment = "binbachar-export" }
 }
 
+
 # -----------------------------------------------------------------------------
 # This bucket will be used for storing the exported RDS snapshots.
 # -----------------------------------------------------------------------------

functions/export-to-s3/index.py

@@ -29,25 +29,31 @@ def handler(event, context):
     eventId = message["detail"]["EventID"]
     sourceId = message["detail"]["SourceIdentifier"]
     sourceArn = message["detail"]["SourceArn"]
-    matchSnapshotRegEx = "^rds:" + os.environ["DB_NAME"] + "-\d{4}-\d{2}-\d{2}-\d{2}-\d{2}$"
-
-    if eventId.endswith(os.environ["RDS_EVENT_ID"]) and re.match(matchSnapshotRegEx, sourceId):
-        exportTaskId = ((sourceId[4:] + '-').replace("--", "-") + messageId)[:60]
-        response = boto3.client("rds").start_export_task(
-            ExportTaskIdentifier=exportTaskId,
-            SourceArn=sourceArn,
-            S3BucketName=os.environ["SNAPSHOT_BUCKET_NAME"],
-            IamRoleArn=os.environ["SNAPSHOT_TASK_ROLE"],
-            KmsKeyId=os.environ["SNAPSHOT_TASK_KEY"],
-        )
-        response["SnapshotTime"] = str(response["SnapshotTime"])
-
-        logger.info("Snapshot export task started")
-        logger.info(json.dumps(response))
+    rdsEventID = os.environ["RDS_EVENT_ID"].replace(' ', '').split(',')
+    dbName = os.environ['DB_NAME'].replace(' ', '').split(',')
+
+    if eventId in rdsEventID:
+        for db in dbName:
+            matchSnapshotRegEx = "^rds:" + db + "-\d{4}-\d{2}-\d{2}-\d{2}-\d{2}$"
+            if re.match(matchSnapshotRegEx, sourceId):
+                exportTaskId = ((sourceId[4:] + '-').replace("--", "-") + messageId)[:60]
+                response = boto3.client("rds").start_export_task(
+                    ExportTaskIdentifier=exportTaskId,
+                    SourceArn=sourceArn,
+                    S3BucketName=os.environ["SNAPSHOT_BUCKET_NAME"],
+                    S3Prefix=os.environ["SNAPSHOT_BUCKET_PREFIX"],
+                    IamRoleArn=os.environ["SNAPSHOT_TASK_ROLE"],
+                    KmsKeyId=os.environ["SNAPSHOT_TASK_KEY"],
+                )
+                response["SnapshotTime"] = str(response["SnapshotTime"])
+
+                logger.info(f"Snapshot export task started on {db}")
+                logger.info(json.dumps(response))
+            else:
+                logger.info(f"Ignoring event notification for {sourceId} - {eventId}")
+                logger.info(f"notifications for {dbName} only")
 
     else:
-        logger.info(f"Ignoring event notification for {sourceId}")
-        logger.info(
-            f"Function is configured to accept {os.environ['RDS_EVENT_ID']} "
-            f"notifications for {os.environ['DB_NAME']} only"
-        )
+        logger.info(f"Ignoring event notification for {sourceId} - {eventId}")
+        logger.info(f"Function is configured to accept {rdsEventID} only")
+

functions/monitor-export-to-s3/index.py

@@ -29,6 +29,7 @@ def handler(event, context):
     sourceType = message["detail"]["SourceType"]
     sourceId = message["detail"]["SourceIdentifier"]
     sourceArn = message["detail"]["SourceArn"]
+    dbName = os.environ['DB_NAME'].replace(' ', '').split(',')
 
     # Ref: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.Messages.html#USER_Events.Messages.snapshot
     # Ref: https://docs.amazonaws.cn/en_us/AmazonRDS/latest/AuroraUserGuide/USER_Events.Messages.html#USER_Events.Messages.cluster-snapshot
@@ -42,25 +43,29 @@ def handler(event, context):
         "RDS-EVENT-0163": "DB cluster snapshot export task canceled",
         "RDS-EVENT-0164": "DB cluster snapshot export task completed"
     }
-    matchSnapshotRegEx = "^rds:" + os.environ["DB_NAME"] + "-\d{4}-\d{2}-\d{2}-\d{2}-\d{2}$"
 
-    if eventId in supportedEvents.keys() and re.match(matchSnapshotRegEx, sourceId):
-        messageTitle = supportedEvents[eventId]
-        messageBody = {
-            "SourceType": sourceType,
-            "SourceIdentifier": sourceId,
-            "SourceArn": sourceArn
-        }
-        response = boto3.client('sns').publish(
-            TargetArn=os.environ["SNS_NOTIFICATIONS_TOPIC_ARN"],
-            Subject=messageTitle,
-            Message=json.dumps({'default': json.dumps(messageBody)}),
-            MessageStructure='json'
-        )
+    if eventId in supportedEvents.keys():
+        for db in dbName:
+            matchSnapshotRegEx = "^rds:" + db + "-\d{4}-\d{2}-\d{2}-\d{2}-\d{2}$"
+            if re.match(matchSnapshotRegEx, sourceId):
+                messageTitle = supportedEvents[eventId]
+                messageBody = {
+                    "SourceType": sourceType,
+                    "SourceIdentifier": sourceId,
+                    "SourceArn": sourceArn
+                }
+                response = boto3.client('sns').publish(
+                    TargetArn=os.environ["SNS_NOTIFICATIONS_TOPIC_ARN"],
+                    Subject=messageTitle,
+                    Message=json.dumps({'default': json.dumps(messageBody)}),
+                    MessageStructure='json'
+                )
+            else:
+                logger.info(f"Ignoring event notification for {sourceId} - {eventId}")
+                logger.info(f"notifications for {dbName} only")
 
     else:
-        logger.info(f"Ignoring event notification for {sourceId}")
-        logger.info(
-            f"Function is configured to accept {supportedEvents} "
-            f"notifications for {os.environ['DB_NAME']} only"
-        )
+        logger.info(f"Ignoring event notification for {sourceId} - {eventId}")
+        logger.info(f"Function is configured to accept {supportedEvents} only")
+
+