[FEATURE] Add Module/Example for Step Function to AWS Batch Workloads

[a13zen]

Is your feature request related to a problem? Please describe.
Provide Amazon Step Functions as sample orchestrator in addition to Airflow/MWAA.

Describe the solution you'd like
Add example module for EventBridge -> Amazon Step Functions -> AWS Batch execution.

This is a common pattern in many AV/ADAS workloads such as resimulation with SILs for example.

[srinivasreddych]

Hello @a13zen Can you provide more context around the use-case for EventBridge here in the ask? Are you using EventPattern or CronBased schedule? If CronBased schedule, can you provide some additional understanding on how it fits the AV/ADAS workload?

[a13zen]

Sure. So, workflows run at difference cadence. You might ingest data, extract the signals/images and run object detection at one cadence lets call it Pipeline1. You might want to run a job that gathers all outputs from Pipeline1 and generate new outputs for instance synchronizing the sensor timings, augmenting the signal data with external data (imagine you have GPS points from the car, and you look up the road type from a third party provider) etc. but once a day only, let's call it Pipeline2.

You might want Pipeline1 to run as a reaction to events since the workloads would be spiky and react to new files being ingested.

You might want Pipeline2 to run as time based job to make use of Autoscaling schedules or Spot instances effectively.