AWS Step Functions Integration #204
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced May 20, 2020
romain-intel
reviewed
Jun 5, 2020
| "that currently doesn't support AWS Step Functions. ") | ||
| obj.echo("For more information on how to upgrade your " | ||
| "service to a compatible version (>= 2.0), visit:") | ||
| obj.echo(" [url]", fg='green') |
There was a problem hiding this comment.
Just putting a TODO here :). So easy to forget.
| ) | ||
|
|
||
| def list_executions(self, state_machine_arn, states): | ||
| if len(states) > 0: |
There was a problem hiding this comment.
Nit, check for uniqueness.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please see #211 instead.
Integrates Metaflow with AWS Step Functions.
Introduces a new command
step-functions:python my_flow.py step-functions createwill compile and export the user-defined Metaflow flow into an AWS Step Functions state machine. This will allow users of Metaflow to move their flows into production seamlessly with AWS.@schedule, allows users to optionally schedule the execution of their flows by integrating with AWS Event Bridge.@batch, dependency management via@conda, retrying mechanisms through@retry,@catchand@timeout, parameters, branches, and for-eaches are now available within AWS Step Functions through this integration.production tokento ensure flow deployments to AWS Step Functions have proper safeguards against unintended production deploymentspython my_flow.py step-functions triggerwill trigger a deployed workflow on AWS Step FunctionsRequirements:
METAFLOW_SFN_IAM_ROLE: IAM role that allows AWS Step Functions to interact with AWS Batch and AWS Dynamo DBMETAFLOW_EVENTS_SFN_ACCESS_IAM_ROLE: IAM role that allows AWS Event Bridge to send notifications to AWS Step FunctionsMETAFLOW_SFN_DYNAMO_DB_TABLE: AWS Dynamo DB Table which allows AWS Step Functions to keep track of the necessary state for foreachesNotes:
DEFAULT_METADATAis set toservice. A subsequent update to the service will fix that issue. For testing, please use -DEFAULT_METADATA=local.Instructions and detailed docs are on their way.
Links: