Demo app to illustrate how to write a Lambda function that self-adjusts its batch size to control throughput of Kinesis events
To deploy this project:
npm installto restore all dependencies
npm run deploy, which would deploy the project using the Serverless framework.
What's included in this project
workerfunction which lets you control how often it will receive a slow or error response via the environment variables
worker: handler: functions/worker.handler timeout: 6 environment: SLOW_PERCENTAGE: 10 ERROR_PERCENTAGE: 10
ventilatorfunction which process data from Kinesis in batches and fans them out to the
workerfunction. In addition, it does the following:
- dedup records against DynamoDB to avoid processing the same record twice.
- performs retry (up to 3 attempts) by excepting when it encounters partial failures.
- decrements batch size by 1 if slow + error accounts for over 50% of all tasks.
- disables the event trigger altogether when reaching batch size of 0.
- if all tasks were processed quickly and no errors then increment the batch size by 1 at a time, up to the configured max (via the
- cron job (every 10 mins) that re-enables the stream if it was disabled.
feed-streamscript that will continuously write data to the stream. To run the script, run