[Feature] EMR on EKS with Apache YuniKorn

[vara-bonthu]

New EMR on EKS deployment pattern with custom scheduler (Apache YuniKorn)

• [TF] -> Terraform teamplate
• A new Terraform templates for deploying EMR on EKS with Cluster Autosclaer
• Managed Node group1: (Multi AZ) Core managed node groups uses the Cluster Autoscaler for critical add-ons
• Managed Node group2: (Single AZ) ON_DEMAND Node group for Spark Drivers
• Managed Node group3: (Single AZ) SPOT Node group for Spark Executors (Same as AZ as Drivers)
• Deploy Apache YuniKorn add-on
• Prometheus and Grafana for Monitoring
• Create Multiple Queue for each data team using YuniKorn
• Run two sample Spark jobs that submits to two different queues
• Add a new docs page using MDX format under -> https://github.com/awslabs/data-on-eks/tree/main/website/docs/amazon-emr-on-eks. Refer the existing examples for writing a doc. Feel free to add screenshots for test results

Use this example as a template and build on top of that.
https://github.com/awslabs/data-on-eks/tree/main/analytics/emr-eks-amp-amg

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

Describe the solution you would like

Describe alternatives you have considered

Additional context

[vara-bonthu]

EMR on EKS with YuniKorn pattern has been added now. Closing this issue.

Further work to add Spark samples to run Gang scheduling with YuniKorn and showing multiple queues and thresholds for each team