From d0a241930a0d154c986cbd6522de5126af025c12 Mon Sep 17 00:00:00 2001 From: Paulo Heneine Date: Thu, 2 Dec 2021 10:13:40 -0300 Subject: [PATCH 1/3] Creating the CDK pattern SQS -> Lambda -> DynamoDB --- sqs-lambda-dynamodb-cdk/README.md | 68 ++++++ sqs-lambda-dynamodb-cdk/README.old | 91 ++++++++ sqs-lambda-dynamodb-cdk/app.py | 11 + sqs-lambda-dynamodb-cdk/cdk.json | 32 +++ sqs-lambda-dynamodb-cdk/example-message.json | 212 ++++++++++++++++++ .../lambda_fns/insertRecord.py | 22 ++ sqs-lambda-dynamodb-cdk/requirements-dev.txt | 1 + sqs-lambda-dynamodb-cdk/requirements.txt | 10 + sqs-lambda-dynamodb-cdk/tests/__init__.py | 0 .../tests/__pycache__/__init__.cpython-39.pyc | Bin 0 -> 181 bytes .../tests/unit/__init__.py | 0 .../unit/__pycache__/__init__.cpython-39.pyc | Bin 0 -> 186 bytes ...o_dynamo_stack.cpython-39-pytest-6.2.5.pyc | Bin 0 -> 1211 bytes .../tests/unit/test_vsam_to_dynamo_stack.py | 32 +++ .../vsam_to_dynamo/__init__.py | 0 .../__pycache__/__init__.cpython-39.pyc | Bin 0 -> 190 bytes .../vsam_to_dynamo_stack.cpython-39.pyc | Bin 0 -> 1454 bytes .../vsam_to_dynamo/vsam_to_dynamo_stack.py | 43 ++++ 18 files changed, 522 insertions(+) create mode 100644 sqs-lambda-dynamodb-cdk/README.md create mode 100644 sqs-lambda-dynamodb-cdk/README.old create mode 100644 sqs-lambda-dynamodb-cdk/app.py create mode 100644 sqs-lambda-dynamodb-cdk/cdk.json create mode 100644 sqs-lambda-dynamodb-cdk/example-message.json create mode 100644 sqs-lambda-dynamodb-cdk/lambda_fns/insertRecord.py create mode 100644 sqs-lambda-dynamodb-cdk/requirements-dev.txt create mode 100644 sqs-lambda-dynamodb-cdk/requirements.txt create mode 100644 sqs-lambda-dynamodb-cdk/tests/__init__.py create mode 100644 sqs-lambda-dynamodb-cdk/tests/__pycache__/__init__.cpython-39.pyc create mode 100644 sqs-lambda-dynamodb-cdk/tests/unit/__init__.py create mode 100644 sqs-lambda-dynamodb-cdk/tests/unit/__pycache__/__init__.cpython-39.pyc create mode 100644 sqs-lambda-dynamodb-cdk/tests/unit/__pycache__/test_vsam_to_dynamo_stack.cpython-39-pytest-6.2.5.pyc create mode 100644 sqs-lambda-dynamodb-cdk/tests/unit/test_vsam_to_dynamo_stack.py create mode 100644 sqs-lambda-dynamodb-cdk/vsam_to_dynamo/__init__.py create mode 100644 sqs-lambda-dynamodb-cdk/vsam_to_dynamo/__pycache__/__init__.cpython-39.pyc create mode 100644 sqs-lambda-dynamodb-cdk/vsam_to_dynamo/__pycache__/vsam_to_dynamo_stack.cpython-39.pyc create mode 100644 sqs-lambda-dynamodb-cdk/vsam_to_dynamo/vsam_to_dynamo_stack.py diff --git a/sqs-lambda-dynamodb-cdk/README.md b/sqs-lambda-dynamodb-cdk/README.md new file mode 100644 index 000000000..ec91a363a --- /dev/null +++ b/sqs-lambda-dynamodb-cdk/README.md @@ -0,0 +1,68 @@ + +# Welcome to your CDK Python project! + +You should explore the contents of this project. It demonstrates a CDK app with an instance of a stack (`vsam_to_dynamo_stack`) +which contains an Amazon SQS queue that is subscribed to an Amazon SNS topic. + +The `cdk.json` file tells the CDK Toolkit how to execute your app. + +This project is set up like a standard Python project. The initialization process also creates +a virtualenv within this project, stored under the .venv directory. To create the virtualenv +it assumes that there is a `python3` executable in your path with access to the `venv` package. +If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv +manually once the init process completes. + +To manually create a virtualenv on MacOS and Linux: + +``` +$ python3 -m venv .venv +``` + +After the init process completes and the virtualenv is created, you can use the following +step to activate your virtualenv. + +``` +$ source .venv/bin/activate +``` + +If you are a Windows platform, you would activate the virtualenv like this: + +``` +% .venv\Scripts\activate.bat +``` + +Once the virtualenv is activated, you can install the required dependencies. + +``` +$ pip install -r requirements.txt +``` + +At this point you can now synthesize the CloudFormation template for this code. + +``` +$ cdk synth +``` + +You can now begin exploring the source code, contained in the hello directory. +There is also a very trivial test included that can be run like this: + +``` +$ pytest +``` + +To add additional dependencies, for example other CDK libraries, just add to +your requirements.txt file and rerun the `pip install -r requirements.txt` +command. + +## Tutorial +See [this useful workshop](https://cdkworkshop.com/30-python.html) on working with the AWS CDK for Python projects. + +## Useful commands + + * `cdk ls` list all stacks in the app + * `cdk synth` emits the synthesized CloudFormation template + * `cdk deploy` deploy this stack to your default AWS account/region + * `cdk diff` compare deployed stack with current state + * `cdk docs` open CDK documentation + +Enjoy! diff --git a/sqs-lambda-dynamodb-cdk/README.old b/sqs-lambda-dynamodb-cdk/README.old new file mode 100644 index 000000000..8fd7a5202 --- /dev/null +++ b/sqs-lambda-dynamodb-cdk/README.old @@ -0,0 +1,91 @@ +# Amazon SNS to Amazon SQS + +This CDK template deploys a SNS topic and an SQS queue. The SQS queue is subscribed to the SNS topic. SNS invokes the SQS queue when new messages are available. When messages are sent to the SNS topic, they are delivered as a JSON event payload to the SQS queue. + +Learn more about this pattern at Serverless Land Patterns: [serverlessland.com/patterns/sns-sqs-cdk](https://serverlessland.com/patterns/sns-sqs-cdk) + +Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. + +## Requirements + +* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured +* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) (AWS CDK >= 1.124.0) Installed + +## Deployment Instructions + +1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository: + ``` + git clone https://github.com/aws-samples/serverless-patterns + ``` +2. Change directory to the pattern directory: + ``` + cd sns-sqs-cdk + ``` +3. From the command line, use AWS CDK to deploy the AWS resources for the pattern as specified in the template.yml file: + ``` + cdk deploy + ``` +4. Create a virtual environment for python: + ``` + python3 -m venv .venv + ``` +5. Activate the virtual environment: + ``` + source .venv/bin/activate + ``` +6. Install python modules: + ``` + python3 -m pip install -r requirements.txt + ``` +7. From the command line, use CDK to synthesize the CloudFormation template and check for errors: + ``` + cdk synth + ``` +8. From the command line, use CDK to deploy the stack: + ``` + cdk deploy + ``` +9. Note the outputs from the CDK deployment process. This contains the SNS and SQS attributes used to test this deployment. + +## Example event payload received by SQS + +``` +{ + "Messages": [ + { + "MessageId": "12345678-fef2-4639-b269-123456789012", + "ReceiptHandle": "123456/g66F9oFe9GjqvCV123456789012qXNQWhEPjaet123456789012P1Najh0B8L4v123456789012tMhDCW8+4HemB123456789012PzU2ZccaD+TRQA6eo123456789012FEz123456789012AeJt4q123456789012xVHh7nEtwEW6/123456789012a9uXzmVl123456789012YRr/slwbLOz3H41234567890129Okiu2rM12345678901231H/5wS123456789012SJsc6juhL5RLLtlJg7GyZcfekyHR7MpVOR123456789012pqh7pJNTa1nFZwfZS2Z123456789012Y0K5d+0xyglCvxfpmg+RzH0ZKIhxN123456789012Nn9PRiTl", + "MD5OfBody": "12345678901280dcda123456789012", + "Body": "{\n \"Type\" : \"Notification\",\n \"MessageId\" : \"12345678-be92-513a-a017-1234567890\",\n \"TopicArn\" : \"arn:aws:sns:us-east-1:123456789012:patterns-sns-to-sqs-MySnsTopic-123456789012\",\n +\"Subject\" : \"testSubject\",\n \"Message\" : \"testMessage\",\n \"Timestamp\" : \"2021-02-10T15:51:48.748Z\",\n \"SignatureVersion\" : \"1\",\n \"Signature\" : \"FMCq/123456789012+cHuU123456789012uge/123456789012/74VExy8A1234567890120LfxjMZvR123456789012pxk6YasI123456789012S7N/CM+qhHOs94lVfdu8zjauMMvRBfBL22qsU14iPB8DTHTuK766DT2IAh+eNTY123456789012u2c8D4gdzl123456789012rpqyf3j123456789012L+BIYpANf123456789012TjxeXNS+Mxh123456789012sq4cAjIqB7CA123456789012j+YpeK123456789012CMulNP282ME123456789012GjIUG6K65MKpA==\",\n \"SigningCertURL\" : \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-12345678901236cd94b123456789012.pem\",\n \"UnsubscribeURL\" +: \"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:123456789012:patterns-sns-to-sqs-MySnsTopic-123456789012:123456789-1745-440a-94a2-1234567890\"\n}" + } + ] +} +``` +### Testing + +Use the [AWS CLI](https://aws.amazon.com/cli/) to send a message to the SNS topic and observe the event delivered to the Lambda function: + +1. Send the SNS message: + +```bash +aws sns publish --topic-arn ENTER_SNS_TOPIC_ARN --subject testSubject --message testMessage +``` +2. Retrieve the message from the SQS queue, using the queue URL from the AWS CDK deployment outputs: +```bash +aws sqs receive-message --queue-url ENTER_YOUR_QUEUE_URL +``` + +## Cleanup + +1. Delete the stack + ```bash + cdk destroy + ``` +---- +Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +SPDX-License-Identifier: MIT-0 diff --git a/sqs-lambda-dynamodb-cdk/app.py b/sqs-lambda-dynamodb-cdk/app.py new file mode 100644 index 000000000..b139fac0c --- /dev/null +++ b/sqs-lambda-dynamodb-cdk/app.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python3 + +from aws_cdk import core + +from vsam_to_dynamo.vsam_to_dynamo_stack import VsamToDynamoStack + + +app = core.App() +VsamToDynamoStack(app, "vsam-to-dynamo") + +app.synth() diff --git a/sqs-lambda-dynamodb-cdk/cdk.json b/sqs-lambda-dynamodb-cdk/cdk.json new file mode 100644 index 000000000..b1104c3df --- /dev/null +++ b/sqs-lambda-dynamodb-cdk/cdk.json @@ -0,0 +1,32 @@ +{ + "app": "python3 app.py", + "watch": { + "include": [ + "**" + ], + "exclude": [ + "README.md", + "cdk*.json", + "requirements*.txt", + "source.bat", + "**/__init__.py", + "python/__pycache__", + "tests" + ] + }, + "context": { + "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true, + "@aws-cdk/core:enableStackNameDuplicates": true, + "aws-cdk:enableDiffNoFail": true, + "@aws-cdk/core:stackRelativeExports": true, + "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true, + "@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true, + "@aws-cdk/aws-kms:defaultKeyPolicies": true, + "@aws-cdk/aws-s3:grantWriteWithoutAcl": true, + "@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount": true, + "@aws-cdk/aws-rds:lowercaseDbIdentifier": true, + "@aws-cdk/aws-efs:defaultEncryptionAtRest": true, + "@aws-cdk/aws-lambda:recognizeVersionProps": true, + "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true + } +} diff --git a/sqs-lambda-dynamodb-cdk/example-message.json b/sqs-lambda-dynamodb-cdk/example-message.json new file mode 100644 index 000000000..7b35b4278 --- /dev/null +++ b/sqs-lambda-dynamodb-cdk/example-message.json @@ -0,0 +1,212 @@ +SQS Message example +The name of the table ("CLIENT") is been used as defined in app.py. Max number of itens is 25 as limited by DynamoDB. + +{"CLIENT": [ +{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "0|0" + }, + "CLIENT-RECORD-COUNT": { + "N": "220" + }, + "FILLER_3": { + "S": "" + } + } + } +} +,{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "1|1" + }, + "CLIENT-NAME": { + "S": "HERBERT MOHAMED" + }, + "CLIENT-BDATE": { + "S": "1958-08-31" + }, + "CLIENT-ED-LVL": { + "S": "BACHELOR" + }, + "CLIENT-INCOME": { + "N": "0010000.00" + }, + "FILLER_1": { + "S": "" + } + } + } +} +,{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "1|2" + }, + "CLIENT-ADDR-NUMBER": { + "N": "36" + }, + "CLIENT-ADDR-STREET": { + "S": "THE ROE AVENUE" + }, + "FILLER_2": { + "S": "" + } + } + } +} +,{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "2|1" + }, + "CLIENT-NAME": { + "S": "JAYLEN GEORGE" + }, + "CLIENT-BDATE": { + "S": "1969-05-29" + }, + "CLIENT-ED-LVL": { + "S": "ELEMENTARY" + }, + "CLIENT-INCOME": { + "N": "0020000.00" + }, + "FILLER_1": { + "S": "" + } + } + } +} +,{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "2|2" + }, + "CLIENT-ADDR-NUMBER": { + "N": "365" + }, + "CLIENT-ADDR-STREET": { + "S": "HEATHFIELD ESPLANADE" + }, + "FILLER_2": { + "S": "" + } + } + } +} +,{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "3|1" + }, + "CLIENT-NAME": { + "S": "MIKAEEL WEBER" + }, + "CLIENT-BDATE": { + "S": "1982-02-17" + }, + "CLIENT-ED-LVL": { + "S": "MASTER" + }, + "CLIENT-INCOME": { + "N": "0030000.00" + }, + "FILLER_1": { + "S": "" + } + } + } +} +,{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "3|2" + }, + "CLIENT-ADDR-NUMBER": { + "N": "4555" + }, + "CLIENT-ADDR-STREET": { + "S": "MORRISON STRAND" + }, + "FILLER_2": { + "S": "" + } + } + } +} +,{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "4|1" + }, + "CLIENT-NAME": { + "S": "APRIL BARRERA" + }, + "CLIENT-BDATE": { + "S": "1967-01-12" + }, + "CLIENT-ED-LVL": { + "S": "DOCTOR" + }, + "CLIENT-INCOME": { + "N": "0030000.00" + }, + "FILLER_1": { + "S": "" + } + } + } +} +,{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "4|2" + }, + "CLIENT-ADDR-NUMBER": { + "N": "1311" + }, + "CLIENT-ADDR-STREET": { + "S": "MARMION PARK" + }, + "FILLER_2": { + "S": "" + } + } + } +} +,{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "5|1" + }, + "CLIENT-NAME": { + "S": "ALEEZA PLANT" + }, + "CLIENT-BDATE": { + "S": "1985-03-01" + }, + "CLIENT-ED-LVL": { + "S": "BACHELOR" + }, + "CLIENT-INCOME": { + "N": "0008000.00" + }, + "FILLER_1": { + "S": "" + } + } + } +} +]} \ No newline at end of file diff --git a/sqs-lambda-dynamodb-cdk/lambda_fns/insertRecord.py b/sqs-lambda-dynamodb-cdk/lambda_fns/insertRecord.py new file mode 100644 index 000000000..5eba367ad --- /dev/null +++ b/sqs-lambda-dynamodb-cdk/lambda_fns/insertRecord.py @@ -0,0 +1,22 @@ +import logging +import json +import boto3 +import ast + +dynamo_client = boto3.client('dynamodb') + +def handler(event, context): + logger = logging.getLogger(__name__) + logger.setLevel(logging.INFO) + logger.info("request: " + json.dumps(event)) + + for record in event['Records']: + payload = record["body"] + logger.info("received message " + payload) + + try: + dynamo_client.batch_write_item(RequestItems=ast.literal_eval(payload)) + except Exception as e: + logger.error(e) + + \ No newline at end of file diff --git a/sqs-lambda-dynamodb-cdk/requirements-dev.txt b/sqs-lambda-dynamodb-cdk/requirements-dev.txt new file mode 100644 index 000000000..927094516 --- /dev/null +++ b/sqs-lambda-dynamodb-cdk/requirements-dev.txt @@ -0,0 +1 @@ +pytest==6.2.5 diff --git a/sqs-lambda-dynamodb-cdk/requirements.txt b/sqs-lambda-dynamodb-cdk/requirements.txt new file mode 100644 index 000000000..ebe45289c --- /dev/null +++ b/sqs-lambda-dynamodb-cdk/requirements.txt @@ -0,0 +1,10 @@ +aws-cdk.core==1.134.0 +aws-cdk.aws_iam==1.134.0 +aws-cdk.aws_sqs==1.134.0 +aws-cdk.aws_sns==1.134.0 +aws-cdk.aws_sns_subscriptions==1.134.0 +aws-cdk.aws_s3==1.134.0 +aws-cdk.assertions==1.134.0 +aws-cdk.aws-dynamodb==1.134.0 +aws-cdk.aws-lambda==1.134.0 +aws-cdk.aws_lambda_event_sources==1.134.0 \ No newline at end of file diff --git a/sqs-lambda-dynamodb-cdk/tests/__init__.py b/sqs-lambda-dynamodb-cdk/tests/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/sqs-lambda-dynamodb-cdk/tests/__pycache__/__init__.cpython-39.pyc b/sqs-lambda-dynamodb-cdk/tests/__pycache__/__init__.cpython-39.pyc new file mode 100644 index 0000000000000000000000000000000000000000..c252bd019322958850ecb21b4c6ff5454f243c9d GIT binary patch literal 181 zcmYe~<>g`kg7!@-5<&E15P=LBfgA@QE@lA|DGb33nv8xc8Hzx{2;x_ierR!OQL%nO zVrfoBYF?^-N@`hoeo=ODL1J>MK2WMGwJ0aGxLCI!v7{umD6d$*xUg6^CowlEB~dq} wGA}VVKP5>wIVD@aB(=DtSU)~KGcU6wK3=b&@)m~;P_i^9)edCmXCP((01>n>o&W#< literal 0 HcmV?d00001 diff --git a/sqs-lambda-dynamodb-cdk/tests/unit/__init__.py b/sqs-lambda-dynamodb-cdk/tests/unit/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/sqs-lambda-dynamodb-cdk/tests/unit/__pycache__/__init__.cpython-39.pyc b/sqs-lambda-dynamodb-cdk/tests/unit/__pycache__/__init__.cpython-39.pyc new file mode 100644 index 0000000000000000000000000000000000000000..7b8e1874a58bf9942ce182fdefd3b0b057575bf3 GIT binary patch literal 186 zcmYe~<>g`kg7!@-5<&E15P=LBfgA@QE@lA|DGb33nv8xc8Hzx{2;x_~erR!OQL%nO zVrfoBYF?^-N@`hoeo=ODL1J>MK2WMGwJ0aGxLCI!v7{umD6d$*xUg6^CowlEB~dq} zGA}VVKP5>wIVD@aB(=DtSidwcvqV2WJ`+gA$LkeT-r}$UswmA#wF6oF8HgDG8Qe0G literal 0 HcmV?d00001 diff --git a/sqs-lambda-dynamodb-cdk/tests/unit/__pycache__/test_vsam_to_dynamo_stack.cpython-39-pytest-6.2.5.pyc b/sqs-lambda-dynamodb-cdk/tests/unit/__pycache__/test_vsam_to_dynamo_stack.cpython-39-pytest-6.2.5.pyc new file mode 100644 index 0000000000000000000000000000000000000000..d57b6622920863d0ec0dbf3e9b72222fdfb5d9e2 GIT binary patch literal 1211 zcmcIjOK%e~5VrS`-7HBQzt9Vey-jEZ2QFChcxG*nzi+--+U*7i#@FxP=%$U(4;|bb76j`s<$Ev; zF`S|Zo7Ku}3t^Nwk&E??m9{c3@-jd2O>C#_EQkW|9p-*P(H+)e9;_Yavo@?xDRSAzb#GVFY*K8^bD9-nMdMklk1qxQMlQ@ZFy%`y3Z3C| zbc&R90#H9&r&!saoHO?XpW{>O#5!v+i`mRc@VTWy0swdWYUL1K8K`2w%sa=c8z0A` z(Rc^UPQ@!8^>!1POp`QG^GTBNqEg8Vj8NYbx;(YLQI^61PiR23KjB%KQUzr8M3E7x zaYv8$sU(8Sq7pGDr6@{GlgkUcwrN?`t~plw>iY5p5?#o4RfZp+ATlgzmG1MLhm0Q{ z6=Eh!8uJjg4!KCVl!KBg#YHYdc_0TV&8CcA6Jpar%x0nDQpvE&6J=bYSrb)|hIIpT zSkCK5`XGTUOnfI-;D0q$5W?EDm~5dWcxu7%NL2+qH0ved)b7FyEd zhem{VhR101w#sAO6qitTbkWpZw7~yecj4*K2NU&{b&iWFS0s@_qc;U@%nF3{Q$w^V z71_IfVL|skoRB)(g08p+{(qtRz))?Pd~0(wn$T&=#a(z#^tAaK_BDgOk&aD?(pMJH z*Wl7^=$cG_rS_*)lBy(^b!WYtlV$XTlyjYLxcvEsf**+lTDZOZRC|kW0bDlEQxSRe zND}BS^Cy`8kDlFbtl{mj>9a;Q-8`nOLIb=ZkM4Y7H1c0%1*=m2S_^MTJ$E~{ANc++ Ds+~6E literal 0 HcmV?d00001 diff --git a/sqs-lambda-dynamodb-cdk/tests/unit/test_vsam_to_dynamo_stack.py b/sqs-lambda-dynamodb-cdk/tests/unit/test_vsam_to_dynamo_stack.py new file mode 100644 index 000000000..f81ca8ea7 --- /dev/null +++ b/sqs-lambda-dynamodb-cdk/tests/unit/test_vsam_to_dynamo_stack.py @@ -0,0 +1,32 @@ +from aws_cdk import ( + core, + assertions + ) + +from vsam_to_dynamo.vsam_to_dynamo_stack import VsamToDynamoStack + + +def test_sqs_queue_created(): + app = core.App() + stack = VsamToDynamoStack(app, "vsam-to-dynamo") + template = assertions.Template.from_stack(stack) + + template.has_resource_properties("AWS::SQS::Queue", { + "VisibilityTimeout": 300 + }) + + +def test_lambda_function_created(): + app = core.App() + stack = VsamToDynamoStack(app, "vsam-to-dynamo") + template = assertions.Template.from_stack(stack) + + template.resource_count_is("AWS::Lambda::Function", 1) + + +def test_dynamodb_table_created(): + app = core.App() + stack = VsamToDynamoStack(app, "vsam-to-dynamo") + template = assertions.Template.from_stack(stack) + + template.resource_count_is("AWS::DynamoDB::Table", 1) diff --git a/sqs-lambda-dynamodb-cdk/vsam_to_dynamo/__init__.py b/sqs-lambda-dynamodb-cdk/vsam_to_dynamo/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/sqs-lambda-dynamodb-cdk/vsam_to_dynamo/__pycache__/__init__.cpython-39.pyc b/sqs-lambda-dynamodb-cdk/vsam_to_dynamo/__pycache__/__init__.cpython-39.pyc new file mode 100644 index 0000000000000000000000000000000000000000..82ce9cbf42ab8b0cc114aa9b0d2a57cb33907725 GIT binary patch literal 190 zcmYe~<>g`kg7!@-5<&E15P=LBfgA@QE@lA|DGb33nv8xc8Hzx{2;x_=erR!OQL%nO zVrfoBYF?^-N@`hoeo=ODL1J>MK2WMGwJ0aGxLCI!v7{umD6d$*xUg6^CowlEB~dq} zGA}VVKP5>wIVD@atT-_@z9c^$BB&o9pP83g5+AQuPkub<45>1-1!D>x<=#-R>mwk=mYC zsj{bCsi$)0z#)}5uvh*ZUpetFtN`&mlP+DbtGa zn}dMxIZS;QKp=rLlw<1UJ|%fT!#tv`yiMc0LpvDVN5T{SGbDW3nRcK0v?s_2^@Bg) zU(^q6#LqNKdERXFiMCyLA{GVDOEIyrlQN!91h;SgyJzxL7KZ7vQi;@-B&CvxyYl96 zMBUII0Z2r#K-81|l*m9};r;Aipd&1NL7sJKc!3C_k)vjB{h)b1F7MSGMusP|Vc)lb zPRhBo-J~qEQB`7CDr~5vsZ^2pwV?(gSXwYUF!fIW20cYnyu?fI$V<`E{{{buF*4rN zUxN1xnFgy+cvl!m`IqDgG10UI8n?DZ02og@BE0kj5fSJE)1CG}--2~x`kH9Ny1BX~ zVz{Am89WJ>!D?$6UV)7zS#7U(;p(<|eYLYBSJ-I;f~R(e^p&?htYjtAk1<5Xqg$t` zPA6%WngugyF3ZZ$_{4p|AU2YQ``>{5X5Ibz;8EYR#OW+Ci@Ah5dULK!YSOY` zGr6F#;gf9r`}XMJ=-S6$Rz>1II=`J3S}JoWA)aFIF)u_W6^)ztv!g;o93oW}j*pGj zdKY~m`)@#ms<~9Q#aLRThA~SRBoqzmXDJsJ+qzdN?o8Q8gONf&?zm*5!Prr=?JtZ` z>7+7}_Fits@E9A8#)l7v_ieP!M%ub&O{2qx5skllH2&(_A-lsqp=2LO+4xA6IpbPO zLpP5VhnziAsgX=@!|6_Q;xekrCCjDO{8(zb$%S|&!)@4zbYpa)4_xYx>dfy)mS~wB zsWv=Lxw5t4?2M~pZG$>gw9`~l{W{uy>B#5Uwi!z@V8+;U^r9Sm502M^Ij^$EvXBEI zPtQs<({rB40jN%;%7EVPoEsxmp$FjE-R8R8>tfwa#B6X1wZlxguHgTOOuK5@n=jNB zaP7j>djJUUcmxw1WACppbUT7Q9R5v+_af{%DFiwGGmK&{#^-PUuRiy#L*8$>@Vgv@ z@W+fn52+yInC&ulQt_U|*Bz`Jhufdg0D&ehrj zXnD0f>~Az33|yexIVG;EDc2X&wgcCU)Q$t_sYt!$AawAKgE~2by4TeA-7s8JaE5K` P)8;v!L4h{b-)sK^5&@=M literal 0 HcmV?d00001 diff --git a/sqs-lambda-dynamodb-cdk/vsam_to_dynamo/vsam_to_dynamo_stack.py b/sqs-lambda-dynamodb-cdk/vsam_to_dynamo/vsam_to_dynamo_stack.py new file mode 100644 index 000000000..35addf7d0 --- /dev/null +++ b/sqs-lambda-dynamodb-cdk/vsam_to_dynamo/vsam_to_dynamo_stack.py @@ -0,0 +1,43 @@ +from aws_cdk import ( + aws_iam as _iam, + aws_sqs as _sqs, + aws_dynamodb as _dyn, + aws_lambda as _lambda, + aws_lambda_event_sources as _event, + core +) + +class VsamToDynamoStack(core.Stack): + + def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None: + super().__init__(scope, construct_id, **kwargs) + + queue = _sqs.Queue( + self, "VsamToDynamoQueue", + visibility_timeout=core.Duration.seconds(300), + queue_name='VsamToDynamoQueue') + + + dynamoTable = _dyn.Table( + self, "CLIENT", + partition_key=_dyn.Attribute( + name="CLIENT-KEY", + type=_dyn.AttributeType.STRING + ), + table_name = "CLIENT", + + ) + + # Create the Lambda function to subscribe to SQS and store the record in DynamoDB + # The source code is in './src' directory + lambda_fn = _lambda.Function( + self, "SQSToDynamoFunction", + runtime=_lambda.Runtime.PYTHON_3_9, + handler="insertRecord.handler", + code=_lambda.Code.from_asset("lambda_fns"), + ) + + dynamoTable.grant_write_data(lambda_fn) + + queue.grant_consume_messages(lambda_fn) + lambda_fn.add_event_source(_event.SqsEventSource(queue)) \ No newline at end of file From e386f38fa14949861dd8feb16921a50776682cc3 Mon Sep 17 00:00:00 2001 From: Paulo Beleza Heneine Date: Thu, 2 Dec 2021 14:30:49 -0300 Subject: [PATCH 2/3] Update README.md Adding related details and instructions. --- sqs-lambda-dynamodb-cdk/README.md | 167 ++++++++++++++++++++++-------- 1 file changed, 125 insertions(+), 42 deletions(-) diff --git a/sqs-lambda-dynamodb-cdk/README.md b/sqs-lambda-dynamodb-cdk/README.md index ec91a363a..eb78b26c8 100644 --- a/sqs-lambda-dynamodb-cdk/README.md +++ b/sqs-lambda-dynamodb-cdk/README.md @@ -1,58 +1,140 @@ -# Welcome to your CDK Python project! - -You should explore the contents of this project. It demonstrates a CDK app with an instance of a stack (`vsam_to_dynamo_stack`) -which contains an Amazon SQS queue that is subscribed to an Amazon SNS topic. - -The `cdk.json` file tells the CDK Toolkit how to execute your app. - -This project is set up like a standard Python project. The initialization process also creates -a virtualenv within this project, stored under the .venv directory. To create the virtualenv -it assumes that there is a `python3` executable in your path with access to the `venv` package. -If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv -manually once the init process completes. - -To manually create a virtualenv on MacOS and Linux: +# Amazon SQS to Amazon DynamoDB + +This CDK application deploys a SQS Queue, a Lambda Function and a DynamoDB Table. The SQS Queue has Lambda Function attached and listening to events. When a new message arrives, the Lambda Function gets the body from the payload and does a batch write in DynamoDb, putting all received items to the table. The CDK application contains the minimum IAM resources required to run the application. + +Learn more about this pattern at: https://serverlessland.com/patterns/sqs-lambda-dynamodb-cdk. + + +## Requirements + +* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured +* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) (AWS CDK >= 1.124.0) Installed + + +## Deployment Instructions + +1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository: + ```bash + git clone https://github.com/aws-samples/serverless-patterns + ``` +1. Change directory to the pattern directory: + ```bash + cd sqs-lambda-dynamodb-cdk + ``` +1. Create a virtual environment for python: + ```bash + python3 -m venv .venv + ``` +1. Activate the virtual environment: + ```bash + source .venv/bin/activate + ``` + + If you are in Windows platform, you would activate the virtualenv like this: + + ``` + % .venv\Scripts\activate.bat + ``` + +1. Install python modules: + ```bash + python3 -m pip install -r requirements.txt + ``` +1. From the command line, use CDK to synthesize the CloudFormation template and check for errors: + ```bash + cdk synth + ``` +1. From the command line, use CDK to deploy the stack: + ```bash + cdk deploy + ``` +1. Note the outputs from the CDK deployment process. These contain the resource names and/or ARNs which are used for testing. + +1. Run code tests to confirm your infrastructure is created: + + ````bash + python3 -m pytest + ```` + + +## How it works + +The CDK stack deploys the resources and the IAM permissions required to run the application. + +The SQS Queue specified in the stack vsam_to_dynamo_stack.py has a Lambda Function responding to the events. The function extracts the body from the message payload and performs a batch write in DynamoDB. The body content must comply with json format expected by DynamoDb (see example below). Messages containing more than 25 itens are not processed by dynamodb.batch_write as limited by DynamoDb. ``` -$ python3 -m venv .venv +{"CLIENT": [ +{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "1|0" + }, + "CLIENT-NAME": { + "S": "ALBERT EINSTEIN" + }, + "CLIENT-RECORD-COUNT": { + "N": "220" + }, + "FILLER_3": { + "S": "" + } + } + } +} +,{ + "PutRequest": { + "Item": { + "CLIENT-KEY": { + "S": "1|1" + }, + "CLIENT-NAME": { + "S": "HERBERT MOHAMED" + }, + "CLIENT-BDATE": { + "S": "1958-08-31" + }, + "CLIENT-ED-LVL": { + "S": "BACHELOR" + }, + "CLIENT-INCOME": { + "N": "0010000.00" + }, + "FILLER_1": { + "S": "" + } + } + } +} +]} ``` -After the init process completes and the virtualenv is created, you can use the following -step to activate your virtualenv. -``` -$ source .venv/bin/activate -``` +## How to test -If you are a Windows platform, you would activate the virtualenv like this: +Tests can be done using aws-cli or directly on the console. Follow the steps below to test from the command line. A file containing a payload with sample records has been provided in the project's root folder, example-message.json. -``` -% .venv\Scripts\activate.bat -``` +1. After sucessfully deploying the stack, get the SQS Queue url: -Once the virtualenv is activated, you can install the required dependencies. +```` +aws sqs get-queue-url --queue-name VsamToDynamoQueue +```` -``` -$ pip install -r requirements.txt -``` - -At this point you can now synthesize the CloudFormation template for this code. - -``` -$ cdk synth -``` +2. Send a message to the queue passing the example file as the message-body parameter value: -You can now begin exploring the source code, contained in the hello directory. -There is also a very trivial test included that can be run like this: +```` +aws sqs send-message --queue-url --message-body file://example-message.json +```` -``` -$ pytest -``` +3. Scan your table to confirm that items have been recorded: -To add additional dependencies, for example other CDK libraries, just add to -your requirements.txt file and rerun the `pip install -r requirements.txt` -command. +```` +aws dynamodb scan --table-name CLIENT +```` ## Tutorial See [this useful workshop](https://cdkworkshop.com/30-python.html) on working with the AWS CDK for Python projects. @@ -65,4 +147,5 @@ See [this useful workshop](https://cdkworkshop.com/30-python.html) on working wi * `cdk diff` compare deployed stack with current state * `cdk docs` open CDK documentation + Enjoy! From e66198c27595696d51fc25a32e5e91f379c52e60 Mon Sep 17 00:00:00 2001 From: Paulo Beleza Heneine Date: Thu, 2 Dec 2021 17:10:06 -0300 Subject: [PATCH 3/3] Update README.md Adding language and framework --- sqs-lambda-dynamodb-cdk/README.md | 39 +++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/sqs-lambda-dynamodb-cdk/README.md b/sqs-lambda-dynamodb-cdk/README.md index eb78b26c8..0966e283d 100644 --- a/sqs-lambda-dynamodb-cdk/README.md +++ b/sqs-lambda-dynamodb-cdk/README.md @@ -1,10 +1,11 @@ # Amazon SQS to Amazon DynamoDB -This CDK application deploys a SQS Queue, a Lambda Function and a DynamoDB Table. The SQS Queue has Lambda Function attached and listening to events. When a new message arrives, the Lambda Function gets the body from the payload and does a batch write in DynamoDb, putting all received items to the table. The CDK application contains the minimum IAM resources required to run the application. +This pattern deploys a SQS Queue, a Lambda Function and a DynamoDB allowing batch writes from SQS messages to a DynamoDb Table. The CDK application contains the minimum IAM resources required to run the application. -Learn more about this pattern at: https://serverlessland.com/patterns/sqs-lambda-dynamodb-cdk. +Learn more about this pattern at: https://serverlessland.com/patterns/sqs-lambda-dynamodb-cdk +Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the AWS Pricing page for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. ## Requirements @@ -13,6 +14,14 @@ Learn more about this pattern at: https://serverlessland.com/patterns/sqs-lambda * [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) * [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) (AWS CDK >= 1.124.0) Installed +## Language +Python + +## Framework +CDK + +## Services From/To +Amazon SQS to AWS Lambda to Amazon DynamoDb ## Deployment Instructions @@ -53,7 +62,7 @@ Learn more about this pattern at: https://serverlessland.com/patterns/sqs-lambda ``` 1. Note the outputs from the CDK deployment process. These contain the resource names and/or ARNs which are used for testing. -1. Run code tests to confirm your infrastructure is created: +1. Run unit tests: ````bash python3 -m pytest @@ -64,7 +73,7 @@ Learn more about this pattern at: https://serverlessland.com/patterns/sqs-lambda The CDK stack deploys the resources and the IAM permissions required to run the application. -The SQS Queue specified in the stack vsam_to_dynamo_stack.py has a Lambda Function responding to the events. The function extracts the body from the message payload and performs a batch write in DynamoDB. The body content must comply with json format expected by DynamoDb (see example below). Messages containing more than 25 itens are not processed by dynamodb.batch_write as limited by DynamoDb. +The SQS Queue specified in the stack vsam_to_dynamo_stack.py has a Lambda Function responding to the events. The function extracts the body from the message payload and performs a batch write in DynamoDB. The body content must comply with json format expected by DynamoDb (see example below). Messages containing more than 25 itens are not processed by dynamodb.batch_write() as it is limited by DynamoDB. ``` {"CLIENT": [ @@ -114,7 +123,7 @@ The SQS Queue specified in the stack vsam_to_dynamo_stack.py has a Lambda Functi ``` -## How to test +## Testing Tests can be done using aws-cli or directly on the console. Follow the steps below to test from the command line. A file containing a payload with sample records has been provided in the project's root folder, example-message.json. @@ -124,6 +133,13 @@ Tests can be done using aws-cli or directly on the console. Follow the steps bel aws sqs get-queue-url --queue-name VsamToDynamoQueue ```` +Response +```` +{ + "QueueUrl": "https://sqs..amazonaws.com//VsamToDynamoQueue" +} +```` + 2. Send a message to the queue passing the example file as the message-body parameter value: ```` @@ -136,6 +152,19 @@ aws sqs send-message --queue-url --message-body file aws dynamodb scan --table-name CLIENT ```` +## Cleanup + +1. Delete the stack + ```bash + aws cloudformation delete-stack --stack-name STACK_NAME + ``` +1. Confirm the stack has been deleted + ```bash + aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'STACK_NAME')].StackStatus" + ``` + + + ## Tutorial See [this useful workshop](https://cdkworkshop.com/30-python.html) on working with the AWS CDK for Python projects.