K8s Spark Enhancements #3655
peridotml
started this conversation in
RFC Incubator
K8s Spark Enhancements
#3655
Replies: 1 comment
-
|
2023-11-09 Contributor's meetup notes: this idea needs a champion, otherwise it will have to be closed. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
A couple enhancements to the k8s spark plugin.
Choosing Machine Types
The k8s paradigm of abstracting away machine types adds additional complexity. Why? We need to know the machine types in order to reasoning through the executor and driver cpu / mem configuration. Getting this wrong can be costly too!
It has become much more common to request specific machine types for spark clusters. Users would need to specify a driver type and a worker type. I imagine that we should be able to generate the k8s spark config automatically.
Spark UI
Understanding the performance of a spark cluster is fairly complex. It requires a combination of logs and looking through the spark UI in real time. It would be great to reach parity with other spark offerings here.
Beta Was this translation helpful? Give feedback.
All reactions