Sfn does not scale up to MaxConcurrency proactively
#44
Comments
31 tasks
rivernews
added a commit
that referenced
this issue
Oct 1, 2022
rivernews
added a commit
that referenced
this issue
Oct 2, 2022
* temp store all * remove go_poc * upgrade so project runs on M1 * Try S3 notification * Fix prefix to include newssite alias * Fix aws lambda PathError issue * Save to metadata.json complete * add untitled stories in metadata.json * rename stories function to landing_metadata * rename batch stories fetch tf to metadata * Improved metadata access s3 event * Metadata.json trigger computing env * read parse metadata.json * fetch a story POC #24 * Sfn map parallism POC #24 * randomize requests * Refactor to allow individual tf modules address #25 (comment) * scaffold table * draft table design * create table * Draining mechanism draft - identify all TODOs #25 (comment) * Draft for put landing page; identified TODOs Issue: #25 * Complete tf surgery; Identify all TODOs in golang For #25 * fix compile error; progress in metadata cronjob add query * Ready to test * Fix db field first char not lowercase Tracked by #25 (comment) * Fix permission of db index, S3 pull Tracked by #25 (comment) * All tests complete Tracked by #25 (comment) * Move landing PutItem out to s3 trigger lambda; ready for S3 batch move * create reusable lambda module; optimize package size #25 (comment) * Fix golang build path * Refactor to use our custom lambda module * add landing s3 trigger * rm golang module stories that are renamed * Fix env var * Fix permission for PutItem move from landing to s3 trigger * Fix metadata s3 trigger not fired * Fix s3 trigger not working - S3 notification can only have one resource * Make it easier to test * prod grade setting enabled * In Sfn pin lambda version, so rolling deploy works better for lambda * Display sfn map result / target stories count info in finalizer * stop landing s3 trigger from sending slack logs Fixes #40 * Let Sfn pin lambda version Fixes #39 * improve log for metadata trigger * improve cronjob log * log cronjob event for better understanding of how it get triggered * Disable cronjob to better debug Fixes #43 * workaround to scale up our Sfn pipeline Fix #44 * improve log for landing S3 trigger * re-enable prod config plus cronjob
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This will affect the time Sfn needs and affects the proper cronjob rate (shall not run concurrent cronjob, only one cronjob should be running at a time). Currently we always see around 45 concurrency that's the max. If so, given the stories usually around 87~130, this mean we need up to 3 pass. Each pass is up to 13 minutes. Total of 39 minutes each Sfn could take, which is a bit slow. It means to process all the landing pages, it's going to take
40min * 818 landings / 60 / 24~~ 22 days.This SO someone says this is expected and usually AWS StepFunction only gives you around 50, and AWS Sfn decides the actual concurrency. Is this true, can we get more source confirming this?
Workaround
If there's a (semi) hard limit on concurrency per Sfn, another way is to run multiple Sfn at the same time! Say, cronjob instead of starting one Sfn, start 3 Sfn executions instead:
The text was updated successfully, but these errors were encountered: