Skip to content
Use AWS State Machine to process DynamoDB Objects.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
.faas
.gitignore
CODE_OF_CONDUCT.md
Gopkg.lock
Gopkg.toml
LICENSE.md
Makefile
README.md
infrastructure.yml
machine.png

README.md

DynamoDB Step Worker

Example project for AWS Step Functions and State Machines to process data objects stored in a DynamoDB table with parallel Scans using Segments.

The CloudFormation Stack creates two State Machines: Puppeteer and Worker. You can configure the Puppeteer to fan out the desired number of Worker executions to process the DynamoDB objects with the following configuration:

{
  "Comment": "Run State Machine Workers",
  "Configuration": {
    "TableName": "step-worker-url-stable",
    "TableHash": "url",
    "Workers": 4,
    "BatchSize": 250
  }
}

State Machine

Usage

Dependencies

# Install Go dependencies

$ > make install

Deployment

# Compile Lambda w/ Go and deploy CloudFormation Stack

$ > make build package deploy

Execute State Machine

# Get ARN of State Machine 

$ > make outputs

[
  {
    "OutputKey": "Puppeteer",
    "OutputValue": "arn:aws:states:eu-west-1:1234567890:stateMachine:step-worker-puppeteer-stable",
    "Description": "ARN for State Machine"
  }
]

# Execute State Machine and configure BatchSize

$ > aws stepfunctions start-execution \
    --state-machine-arn arn:aws:states:eu-west-1:1234567890:stateMachine:step-worker-puppeteer-stable \
    --input '
    {
      "Comment": "Run State Machine with BatchSize 25",
      "Configuration": {
        "TableName": "step-worker-url-stable",
        "TableHash": "url",
        "Workers": 4,
        "BatchSize": 250
      }
    }'

License

Feel free to use the code, it's released using the MIT license.

Contribution

You are welcome to contribute to this project! 😘

To make sure you have a pleasant experience, please read the code of conduct. It outlines core values and beliefs and will make working together a happier experience.

You can’t perform that action at this time.