EMR on EKS example update

[vara-bonthu]

What does this PR do?

• This example updated with custom Add-on config to support Spark workloads
• Uses best practice to separate core node group and spark node group
• Deploys add-ons to core node-groups while the Spark workloads runs on separate node group
• CoreDNS Autoscaler added
• Prometheus and VPA uplifted to latest versions
• Some cleanup on existing examples

Motivation

• Customers looking for best practices and scalable solution to run Spark work loads using EMR on EKS
• Also ensure the blog is up-to-date https://aws.amazon.com/blogs/mt/monitoring-amazon-emr-on-eks-with-amazon-managed-prometheus-and-amazon-managed-grafana/

More

• Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
• Yes, I have added a new example under examples to support my PR
• Yes, I have created another PR for add-ons under add-ons repo (if applicable)
• Yes, I have updated the docs for this feature
• Yes, I ran pre-commit run -a with this PR

Note: Not all the PRs required examples and docs except a new pattern or add-on added.

For Moderators

• E2E Test successfully complete before merge?

Additional Notes

[bryantbiggs]

what is the reason behind the use of variables in the examples?

[vara-bonthu]

If we change the folder name in future then the blog that refers to this example will break with our current option of using basename(path.cwd) . I would like to ensure the name is always emr-on-eks for this example so added this default value to the variable.

Customers can also change the name if they want to by changing the variable or with tfvars without fiddling with the hardcoded values in main.tf