Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MWAA module and example #585

Merged
merged 8 commits into from
Jul 8, 2022
Merged

Add MWAA module and example #585

merged 8 commits into from
Jul 8, 2022

Conversation

maiconrocha
Copy link
Contributor

@maiconrocha maiconrocha commented May 30, 2022

What does this PR do?

Demonstrates how to use Amazon Managed Workflows for Apache Airflow (MWAA) with Amazon EKS.
This example was originated from the steps provided on MWAA documentation on the link below:
mwaa-eks-example

Also, I have added a module to provision Amazon Managed Workflows for Apache Airflow (MWAA).

Motivation

Close #441

As an AWS Employee with experience with EKS, Terraform and MWAA, I want to provide an example to customers in how to use Amazon Managed Workflows for Apache Airflow (MWAA) with Amazon EKS.

More

  • Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
  • Yes, I have added a new example under examples to support my PR
  • Yes, I have created another PR for add-ons under add-ons repo (if applicable)
  • Yes, I have updated the docs for this feature
  • Yes, I ran pre-commit run -a with this PR

Note: Not all the PRs required examples and docs except a new pattern or add-on added.

For Moderators

  • E2E Test successfully complete before merge?

Additional Notes

Test Evidence showing the pod was schedule successfully by MWAA:
kubectl get pods -n mwaa
NAME READY STATUS RESTARTS AGE
mwaa-pod-test.2816b0b5b01047af947768d5cd6b889b 0/1 Completed 0 31s

examples/mwaa/main.tf Outdated Show resolved Hide resolved
Copy link
Contributor

@vara-bonthu vara-bonthu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for much needed PR 👍🏼 Few minor comments.
I haven't tested this myself. Please add the testing evidence to this PR.

examples/mwaa/dags/mwaa_pod_example.py Outdated Show resolved Hide resolved
examples/mwaa/aws-mwaa/README.md Outdated Show resolved Hide resolved
examples/mwaa/README.md Outdated Show resolved Hide resolved
@vara-bonthu
Copy link
Contributor

@maiconrocha Please update this PR to point to new MWAA module

examples/mwaa/main.tf Outdated Show resolved Hide resolved
@vara-bonthu
Copy link
Contributor

@maiconrocha Could you please update this PR as discussed?

@maiconrocha
Copy link
Contributor Author

@maiconrocha Could you please update this PR as discussed?
What other changes I need to make? I'm point already to mwaa module on aws-ia/mwaa/aws terraform registry, once we published the mwaa module there I can do one more final test before we can merge this PR.

@maiconrocha
Copy link
Contributor Author

@vara-bonthu I have update example to use mwaa module on the registry, it is tested and working as expected

@bryantbiggs
Copy link
Contributor

@maiconrocha I'm going to make a few updates but we should be able to get this merged today - thank you again for this contribution!


eks_cluster_id = module.eks_blueprints.eks_cluster_id

enable_metrics_server = false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want these enabled or should we just skip the addons for this example? I haven't delved into MWAA but seems like we'd want some autoscaling to support

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for now I'm enabling them but we can always adjust the example in subsequent PRs

@bryantbiggs bryantbiggs temporarily deployed to EKS Blueprints Test July 8, 2022 19:55 Inactive
@bryantbiggs bryantbiggs dismissed vara-bonthu’s stale review July 8, 2022 19:56

Updates made and validated in 7b8663d

Copy link
Contributor

@bryantbiggs bryantbiggs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this addition @maiconrocha and the module contribution 🎉

@bryantbiggs bryantbiggs merged commit e64e9d9 into aws-ia:main Jul 8, 2022
allamand pushed a commit to allamand/terraform-aws-eks-blueprints that referenced this pull request Dec 15, 2022
Co-authored-by: Bryant Biggs <bryantbiggs@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BLUEPRINT] - MWAA Workflow
4 participants