S3 to StepFunctions integrations

This repository contains examples of integrating Amazon S3 events with AWS StepFunctions.

There are three examples in total. Two of them leverage AWS CloudTrail and Amazon EventBridge, while the last one leverages the AWS Lambda function triggered by Amazon S3 notification.

Deployment

1. npm run bootstrap
2. npm run deploy

Learnings

• To my best knowledge, there are two ways one might create such integration.

  • The first is to use S3 bucket notification feature that points to an AWS Lambda. The AWS Lambda would then either invoke AWS StepFunctions directly or forward the event to an event bus.
  • The second does not use any AWS Lambda functions at all. You create a CloudTrail and use EventSelectors to filter those events that you want to react to. The event will be pushed onto the default event bus automatically by CloudTrail.

• When it comes to pricing, remember that CloudTrail Data events are not free!. If you wish to listen to S3 object-created related events, you will incur cost for event event delivered.

• Then using CloudTrail EventSelectors be mindful of different predicates that are available to you.

  • The equals predicate is pretty weird. I've uploaded an object to a path of /uploads/foo.txt but the equals matched on ${bucket.arn}/ path
  • I find the startsWith work best where you would want to filter s3 events for a given path.

• When configuring Amazon S3 notifications, the service will send a test message to validate that the subscription is setup correctly. I suspect that this message is send at deploy time to validate IAM permissions.

  • When configuring AWS Lambda as the receiver of the event, Amazon CloudWatch Logs report a strange invocation that seemingly does not invoke my handler at all. Despite having multiple console.log scattered through the code, all I see in the logs is Status code: OK. Very interesting.
  • Which component of the AWS Lambda service handles that event?
    • I could not confirm or deny this. The Nodejs RIC repository does not contain any references to S3. Maybe it is the AWS Lambda service job?