Skip to content

Add MwaaServerlessStartWorkflowRunOperator#66046

Merged
vincbeck merged 1 commit intoapache:mainfrom
john-jac:add-mwaa-serverless-start-workflow
May 1, 2026
Merged

Add MwaaServerlessStartWorkflowRunOperator#66046
vincbeck merged 1 commit intoapache:mainfrom
john-jac:add-mwaa-serverless-start-workflow

Conversation

@john-jac
Copy link
Copy Markdown
Contributor

@john-jac john-jac commented Apr 28, 2026

What

Add an operator for Amazon MWAA Serverless to start workflow runs.

Motivation

TriggerDagRunOperator triggers a DAG run within the same Airflow environment. MwaaServerlessStartWorkflowRunOperator serves the analogous purpose for MWAA Serverless — it triggers a workflow execution on a remote MWAA Serverless environment from any Airflow DAG. This enables patterns like:

  • Cross-environment orchestration: A central Airflow instance triggers serverless workflows that run on MWAA Serverless with automatic scaling and task isolation
  • Hybrid pipelines: Combine provisioned MWAA for long-running orchestration with MWAA Serverless for burst workloads
  • CI/CD for workflows: A deployment DAG validates and triggers workflow runs on MWAA Serverless as part of a release pipeline

MWAA Serverless is a separate service from MWAA (similar to how EMR Serverless is separate from EMR), with its own API (mwaa-serverless) and resource model (workflows instead of environments).

How

  • New MwaaServerlessStartWorkflowRunOperator using AwsBaseOperator[AwsBaseHook] with client_type="mwaa-serverless"
  • Supports override_parameters for runtime customization and workflow_version for version pinning
  • Returns the workflow run ID
  • Follows EMR/EMR Serverless pattern: separate integration entry, separate operator file, shared logo

Files

  • operators/mwaa_serverless.py — operator
  • test_mwaa_serverless.py — unit tests (3 tests)
  • example_mwaa_serverless.py — system test
  • mwaa_serverless.rst — docs
  • provider.yaml + get_provider_info.py — new MWAA Serverless integration + operator entries

Testing

  • Unit tests: 3 passed
  • mypy: no issues
  • Static checks: 43 passed, 0 PR-related failures

@john-jac john-jac requested a review from o-nikolas as a code owner April 28, 2026 19:04
@john-jac john-jac force-pushed the add-mwaa-serverless-start-workflow branch from 6e3cc32 to b3ffd39 Compare April 28, 2026 19:53
Comment thread providers/amazon/tests/system/amazon/aws/example_mwaa_serverless.py
@john-jac john-jac force-pushed the add-mwaa-serverless-start-workflow branch 8 times, most recently from 2774af1 to ab5f2b3 Compare April 29, 2026 00:36
@john-jac
Copy link
Copy Markdown
Contributor Author

Closing to resubmit with clean branch.

@john-jac john-jac closed this Apr 29, 2026
@john-jac john-jac reopened this Apr 29, 2026
Comment thread providers/amazon/tests/system/amazon/aws/example_mwaa_serverless.py Outdated
Comment thread providers/amazon/tests/system/amazon/aws/example_mwaa_serverless.py Outdated
@john-jac john-jac force-pushed the add-mwaa-serverless-start-workflow branch 2 times, most recently from 5cc916e to b66c361 Compare April 29, 2026 14:34
@john-jac
Copy link
Copy Markdown
Contributor Author

john-jac commented Apr 29, 2026

Thanks @vincbeck for the review! Addressed both points:

  1. IAM role as external parameter — now uses SystemTestContextBuilder().add_variable("ROLE_ARN") instead of creating roles in the test (following the example_batch.py pattern).

  2. Split into individual tasks — replaced the monolithic setup/teardown with separate @task functions: create_bucket (using S3 operators), upload_workflow_yaml, create_workflow, stop_workflow_run, delete_workflow, delete_bucket (also using S3 operators). Each can be replaced by an operator later.

@john-jac john-jac force-pushed the add-mwaa-serverless-start-workflow branch 3 times, most recently from 2826f7d to f37f487 Compare April 29, 2026 17:28
Comment thread providers/amazon/tests/system/amazon/aws/example_mwaa_serverless.py Outdated
@john-jac
Copy link
Copy Markdown
Contributor Author

Good call @vincbeck — moved all @task functions outside the DAG context to module level, matching the pattern in example_batch.py and other system tests.

@john-jac john-jac force-pushed the add-mwaa-serverless-start-workflow branch 2 times, most recently from 62246d0 to 6906dfc Compare April 29, 2026 18:51
Comment thread providers/amazon/tests/system/amazon/aws/example_mwaa_serverless.py Outdated
@john-jac john-jac force-pushed the add-mwaa-serverless-start-workflow branch from 6906dfc to 6253aae Compare April 29, 2026 21:24
Comment thread providers/amazon/tests/system/amazon/aws/example_mwaa_serverless.py Outdated
Comment thread providers/amazon/tests/system/amazon/aws/example_mwaa_serverless.py Outdated
@john-jac john-jac force-pushed the add-mwaa-serverless-start-workflow branch 9 times, most recently from 3ef7a82 to 5dd71d2 Compare May 1, 2026 20:19
@john-jac john-jac force-pushed the add-mwaa-serverless-start-workflow branch from 5dd71d2 to a27308f Compare May 1, 2026 20:26
@vincbeck vincbeck merged commit b3343d3 into apache:main May 1, 2026
95 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants