New pattern submission : sfn-glue-sync-cdk

[shsnket]

To submit a template to the Serverless Patterns Collection, submit an issue with the following information.

To learn more about submitting a pattern, read the publishing guidelines page.

Use the model template located at https://github.com/aws-samples/serverless-patterns/tree/main/_pattern-model to set up a README, template and any associated code.

Description (mid-length e.g. "Create a Lambda function that sends events to EventBridge.")

The CDK stack deploys Step Functions workflow, that takes in a payload and trigger a AWS Glue job.

Language: (optional e.g. "Python", if you have a Lambda function in your example)

Python

YouTube videoId (optional e.g. "VI79XQW4dIM")

Framework (currently we support SAM or CDK)

CDK

Services from/to (e.g. "Lambda to EventBridge)

Step Functions to Glue Job

Description (this must include a throughout explanation of the pattern together with details of IAM permissioning)

This CDK application deploys a Step Functions workflow, that takes in a payload and trigger a AWS Glue job synchronously. In this pattern, the state machine does wait for Glue job to finish. The application contains the minimum IAM resources required to run the workflow and Glue job.

Deployment commands

1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:

   git clone https://github.com/aws-samples/serverless-patterns

2. Change directory to the pattern directory:

   cd sfn-glue-sync-cdk

3. Create a virtual environment for python:

   python3 -m venv .venv

4. Activate the virtual environment:

   source .venv/bin/activate

5. Install python modules:

   python3 -m pip install -r requirements.txt

6. From the command line, use CDK to synthesize the CloudFormation template and check for errors:

   cdk synth

7. From the command line, use CDK to deploy the stack:

   cdk deploy

8. Note the outputs from the CDK deployment process. These contain the resource names and/or ARNs which are used for testing.

GitHub PR for template:

#214

Payload example (e.g. Lambda event payload from source service).

aws stepfunctions start-execution --state-machine-arn "{StateMachineArn}" --input "{\"message\": \"Hello from sfn glue step\"}"

Additional resources (optional: link and anchor text, up to 5 resources)

• What Is AWS Glue?
• Manage AWS Glue Jobs with Step Functions
• CDK documentation for Glue
• CDK documentation for Step Functions

Author bio

Name: Sanket Shinde
Photo URL:https://avatars.githubusercontent.com/u/81987730?s=400&u=6ee77a41f2a59a88c05a04522908a7497c0e9f02&v=4
Twitter handle:
Description (up to 255 chars): Sanket is a Data Engineer at AWS Professional Service.

[jbesw]

Thanks for the contribution! This is now live at https://serverlessland.com/patterns/sfn-glue-sync-cdk.