[Feat] Observability on Spark on EKS

[victorgu-github]

• Provide guidance on Spark history server
• Monitor spark applications with Prometheus (or AMP)

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

Describe the solution you would like

Describe alternatives you have considered

Additional context

[vara-bonthu]

We already have guidance for deploying Spark history server

data-on-eks/analytics/spark-k8s-operator/addons.tf

Line 115 in 8d3021d

enable_spark_history_server = true

. This blueprint also writes the metrics to AMP which can be consumed by Grafana dashboards.

Let me know if you want to see anything else

[victorgu-github]

@vara-bonthu it is mainly for documentation. I am thinking to create a web page to summarize how to monitor spark app using history server, spark web ui, prometheus and grafana (like snapshot below)
we can reuse the code in [spark-k8s-operator] and only check in some sample yaml code for demo purpose (like taxi driver one)
pls let me know your opinion. thx

[vara-bonthu]

Nice one! This will make it lot easier for users. Go for it

[victorgu-github]

implemented