Self-managed Apache Airflow on EKS

[vara-bonthu]

What does this PR do?

• This PR deploys Apache Airflow Add-on on EKS
• Production ready deployment example with instructions
• Github Docs page

Closes the issue: #810

🛑 Please open an issue first to discuss any significant work and flesh out details/direction - we would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.

Motivation

Running Apache Airflow on Kubernetes is the common pattern for Data teams. This example will help customers to build production ready Apache Airflow on EKS.

This is also a request from few AWS customers

More

• Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
• Yes, I have added a new example under examples to support my PR
• Yes, I have created another PR for add-ons under add-ons repo (if applicable)
• Yes, I have updated the docs for this feature
• Yes, I ran pre-commit run -a with this PR

Note: Not all the PRs required examples and docs except a new pattern or add-on added.

For Moderators

• E2E Test successfully complete before merge?

Additional Notes

[vara-bonthu]

Agree. We should we move the MWAA example under Analytics folder and then rename the folders to managed-airflow and self-managed-airflow.

The main difference is that MWAA has nothing to do with EKS unlike self-managed airflow hosted on EKS. MWAA example is only to trigger the job on EKS

I have added additional information to Docs(apache-airflow.md) and Readme as well.

[vara-bonthu]

Check Markdown links / markdown-link-check error is expected as the link is not available in MAIN until this PR is merged

[bryantbiggs]

Looks great - few minor details to correct on the CI checks and then good to go !

[vara-bonthu]

Looks great - few minor details to correct on the CI checks and then good to go !

Plan error seems to be related to RAY example. Let me see if that needs fixing

Error: Error in function call

  on main.tf line 156, in module "eks_blueprints_kubernetes_addons":
 156:     values = [templatefile("${path.module}/helm-values/kube-stack-prometheus-values.yaml", {
 157:       hostname = var.eks_cluster_domain
 158:     })]
    ├────────────────
    │ path.module is "."
    │ var.eks_cluster_domain is null

Call to function "templatefile" failed:
./helm-values/kube-stack-prometheus-values.yaml:6,[27](https://github.com/aws-ia/terraform-aws-eks-blueprints/runs/7971539678?check_suite_focus=true#step:8:28)-[35](https://github.com/aws-ia/terraform-aws-eks-blueprints/runs/7971539678?check_suite_focus=true#step:8:36): Invalid template
interpolation value; The expression result is null. Cannot include a null
value in a string template..