New pattern submission - sqs lambda dynamodb cdk

sqs-lambda-dynamodb-cdk/README.md

@@ -0,0 +1,180 @@
+
+# Amazon SQS to Amazon DynamoDB
+
+This pattern deploys a SQS Queue, a Lambda Function and a DynamoDB allowing batch writes from SQS messages to a DynamoDb Table. The CDK application contains the minimum IAM resources required to run the application. 
+
+Learn more about this pattern at: https://serverlessland.com/patterns/sqs-lambda-dynamodb-cdk
+
+Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the AWS Pricing page for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
+
+## Requirements
+
+* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
+* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
+* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
+* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) (AWS CDK >= 1.124.0) Installed
+
+## Language
+Python
+
+## Framework
+CDK
+
+## Services From/To
+Amazon SQS to AWS Lambda to Amazon DynamoDb
+
+## Deployment Instructions
+
+1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
+    ```bash 
+    git clone https://github.com/aws-samples/serverless-patterns
+    ```
+1. Change directory to the pattern directory:
+    ```bash
+    cd sqs-lambda-dynamodb-cdk
+    ```
+1. Create a virtual environment for python:
+    ```bash
+    python3 -m venv .venv
+    ```
+1. Activate the virtual environment:
+    ```bash
+    source .venv/bin/activate
+    ```
+
+    If you are in Windows platform, you would activate the virtualenv like this:
+
+    ```
+    % .venv\Scripts\activate.bat
+    ```
+
+1. Install python modules:
+    ```bash
+    python3 -m pip install -r requirements.txt
+    ```
+1. From the command line, use CDK to synthesize the CloudFormation template and check for errors:
+    ```bash
+    cdk synth
+    ```
+1. From the command line, use CDK to deploy the stack:
+    ```bash
+    cdk deploy
+    ```
+1. Note the outputs from the CDK deployment process. These contain the resource names and/or ARNs which are used for testing.
+
+1. Run unit tests:
+
+    ````bash
+    python3 -m pytest
+    ````
+
+
+## How it works
+
+The CDK stack deploys the resources and the IAM permissions required to run the application.
+
+The SQS Queue specified in the stack vsam_to_dynamo_stack.py has a Lambda Function responding to the events. The function extracts the body from the message payload and performs a batch write in DynamoDB. The body content must comply with json format expected by DynamoDb (see example below). Messages containing more than 25 itens are not processed by dynamodb.batch_write() as it is limited by DynamoDB.
+
+```
+{"CLIENT": [
+{
+    "PutRequest": {
+        "Item": {
+            "CLIENT-KEY": {
+                "S": "1|0"
+            },
+            "CLIENT-NAME": {
+                "S": "ALBERT EINSTEIN"
+            },
+            "CLIENT-RECORD-COUNT": {
+                "N": "220"
+            },
+            "FILLER_3": {
+                "S": ""
+            }
+        }
+    }
+}
+,{
+    "PutRequest": {
+        "Item": {
+            "CLIENT-KEY": {
+                "S": "1|1"
+            },
+            "CLIENT-NAME": {
+                "S": "HERBERT MOHAMED"
+            },
+            "CLIENT-BDATE": {
+                "S": "1958-08-31"
+            },
+            "CLIENT-ED-LVL": {
+                "S": "BACHELOR"
+            },
+            "CLIENT-INCOME": {
+                "N": "0010000.00"
+            },
+            "FILLER_1": {
+                "S": ""
+            }
+        }
+    }
+}
+]}
+```
+
+
+## Testing
+
+Tests can be done using aws-cli or directly on the console. Follow the steps below to test from the command line. A file containing a payload with sample records has been provided in the project's root folder, example-message.json.
+
+1. After sucessfully deploying the stack, get the SQS Queue url:
+
+````
+aws sqs get-queue-url --queue-name VsamToDynamoQueue
+````
+
+Response
+````
+{
+    "QueueUrl": "https://sqs.<your region>.amazonaws.com/<your account>/VsamToDynamoQueue"
+}
+````
+
+2. Send a message to the queue passing the example file as the message-body parameter value:
+
+````
+aws sqs send-message --queue-url <Replace by QueueUrl value> --message-body file://example-message.json
+````
+
+3. Scan your table to confirm that items have been recorded:
+
+````
+aws dynamodb scan --table-name CLIENT
+````
+
+## Cleanup
+
+1. Delete the stack
+    ```bash
+    aws cloudformation delete-stack --stack-name STACK_NAME
+    ```
+1. Confirm the stack has been deleted
+    ```bash
+    aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'STACK_NAME')].StackStatus"
+    ```
+
+
+
+## Tutorial  
+See [this useful workshop](https://cdkworkshop.com/30-python.html) on working with the AWS CDK for Python projects.
+
+## Useful commands
+
+ * `cdk ls`          list all stacks in the app
+ * `cdk synth`       emits the synthesized CloudFormation template
+ * `cdk deploy`      deploy this stack to your default AWS account/region
+ * `cdk diff`        compare deployed stack with current state
+ * `cdk docs`        open CDK documentation
+
+
+Enjoy!

sqs-lambda-dynamodb-cdk/README.old

@@ -0,0 +1,91 @@
+# Amazon SNS to Amazon SQS
+
+This CDK template deploys a SNS topic and an SQS queue. The SQS queue is subscribed to the SNS topic. SNS invokes the SQS queue when new messages are available. When messages are sent to the SNS topic, they are delivered as a JSON event payload to the SQS queue.
+
+Learn more about this pattern at Serverless Land Patterns: [serverlessland.com/patterns/sns-sqs-cdk](https://serverlessland.com/patterns/sns-sqs-cdk)
+
+Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
+
+## Requirements
+
+* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
+* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
+* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
+* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) (AWS CDK >= 1.124.0) Installed
+
+## Deployment Instructions
+
+1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
+    ``` 
+    git clone https://github.com/aws-samples/serverless-patterns
+    ```
+2. Change directory to the pattern directory:
+    ```
+    cd sns-sqs-cdk
+    ```
+3. From the command line, use AWS CDK to deploy the AWS resources for the pattern as specified in the template.yml file:
+    ```
+    cdk deploy
+    ```
+4. Create a virtual environment for python:
+    ```
+    python3 -m venv .venv
+    ```
+5. Activate the virtual environment:
+    ```
+    source .venv/bin/activate
+    ```
+6. Install python modules:
+    ```
+    python3 -m pip install -r requirements.txt
+    ```
+7. From the command line, use CDK to synthesize the CloudFormation template and check for errors:
+    ```
+    cdk synth
+    ```
+8. From the command line, use CDK to deploy the stack:
+    ```
+    cdk deploy
+    ```
+9. Note the outputs from the CDK deployment process. This contains the SNS and SQS attributes used to test this deployment.
+
+## Example event payload received by SQS
+
+```
+{
+    "Messages": [
+        {
+            "MessageId": "12345678-fef2-4639-b269-123456789012",
+            "ReceiptHandle": "123456/g66F9oFe9GjqvCV123456789012qXNQWhEPjaet123456789012P1Najh0B8L4v123456789012tMhDCW8+4HemB123456789012PzU2ZccaD+TRQA6eo123456789012FEz123456789012AeJt4q123456789012xVHh7nEtwEW6/123456789012a9uXzmVl123456789012YRr/slwbLOz3H41234567890129Okiu2rM12345678901231H/5wS123456789012SJsc6juhL5RLLtlJg7GyZcfekyHR7MpVOR123456789012pqh7pJNTa1nFZwfZS2Z123456789012Y0K5d+0xyglCvxfpmg+RzH0ZKIhxN123456789012Nn9PRiTl",
+            "MD5OfBody": "12345678901280dcda123456789012",
+            "Body": "{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"12345678-be92-513a-a017-1234567890\",\n  \"TopicArn\" : \"arn:aws:sns:us-east-1:123456789012:patterns-sns-to-sqs-MySnsTopic-123456789012\",\n  
+\"Subject\" : \"testSubject\",\n  \"Message\" : \"testMessage\",\n  \"Timestamp\" : \"2021-02-10T15:51:48.748Z\",\n  \"SignatureVersion\" : \"1\",\n  \"Signature\" : \"FMCq/123456789012+cHuU123456789012uge/123456789012/74VExy8A1234567890120LfxjMZvR123456789012pxk6YasI123456789012S7N/CM+qhHOs94lVfdu8zjauMMvRBfBL22qsU14iPB8DTHTuK766DT2IAh+eNTY123456789012u2c8D4gdzl123456789012rpqyf3j123456789012L+BIYpANf123456789012TjxeXNS+Mxh123456789012sq4cAjIqB7CA123456789012j+YpeK123456789012CMulNP282ME123456789012GjIUG6K65MKpA==\",\n  \"SigningCertURL\" : \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-12345678901236cd94b123456789012.pem\",\n  \"UnsubscribeURL\" 
+: \"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:123456789012:patterns-sns-to-sqs-MySnsTopic-123456789012:123456789-1745-440a-94a2-1234567890\"\n}"
+        }
+    ]
+}
+```
+### Testing
+
+Use the [AWS CLI](https://aws.amazon.com/cli/) to send a message to the SNS topic and observe the event delivered to the Lambda function:
+
+1. Send the SNS message:
+
+```bash
+aws sns publish --topic-arn ENTER_SNS_TOPIC_ARN  --subject testSubject --message testMessage
+```
+2. Retrieve the message from the SQS queue, using the queue URL from the AWS CDK deployment outputs:
+```bash
+aws sqs receive-message --queue-url ENTER_YOUR_QUEUE_URL
+```
+
+## Cleanup
+
+1. Delete the stack
+    ```bash
+    cdk destroy
+    ```
+----
+Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+SPDX-License-Identifier: MIT-0

sqs-lambda-dynamodb-cdk/app.py

@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+from aws_cdk import core
+
+from vsam_to_dynamo.vsam_to_dynamo_stack import VsamToDynamoStack
+
+
+app = core.App()
+VsamToDynamoStack(app, "vsam-to-dynamo")
+
+app.synth()

sqs-lambda-dynamodb-cdk/cdk.json

@@ -0,0 +1,32 @@
+{
+  "app": "python3 app.py",
+  "watch": {
+    "include": [
+      "**"
+    ],
+    "exclude": [
+      "README.md",
+      "cdk*.json",
+      "requirements*.txt",
+      "source.bat",
+      "**/__init__.py",
+      "python/__pycache__",
+      "tests"
+    ]
+  },
+  "context": {
+    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
+    "@aws-cdk/core:enableStackNameDuplicates": true,
+    "aws-cdk:enableDiffNoFail": true,
+    "@aws-cdk/core:stackRelativeExports": true,
+    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
+    "@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,
+    "@aws-cdk/aws-kms:defaultKeyPolicies": true,
+    "@aws-cdk/aws-s3:grantWriteWithoutAcl": true,
+    "@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount": true,
+    "@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
+    "@aws-cdk/aws-efs:defaultEncryptionAtRest": true,
+    "@aws-cdk/aws-lambda:recognizeVersionProps": true,
+    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true
+  }
+}