[SPARK-24353][K8S][2.4] Support for specifying multiple custom tolerations to driver and executor pods

[ghost]

What changes were proposed in this pull request?

The PR allows a user to specify custom k8s tolerations when running Spark on kubernetes for both driver and executor pods.

The user can now set the key, effect, operator and value properties when running spark-submit, as long as they're prefixed by:

spark.kubernetes.driver.tolerations.n or spark.kubernetes.executor.tolerations.n

where n = a number starting from 0 that is incremented of 1.

Why not using the key field, as per original design doc attached to the JIRA issue?

As per kubernetes toleration v1 spec, the field key can be unset, thus disqualifying it as a unique identifier.

Some examples:

# driver pods tolerate the taint group:spark-driver registered to a specialised worker group
spark.kubernetes.driver.tolerations.0.key group
spark.kubernetes.driver.tolerations.0.operator Equal
spark.kubernetes.driver.tolerations.0.effect NoSchedule
spark.kubernetes.driver.tolerations.0.value spark-driver

# executor pods tolerate the taint group:spark-executor registered to another specialised worker group
spark.kubernetes.executor.tolerations.0.key group
spark.kubernetes.executor.tolerations.0.operator Equal
spark.kubernetes.executor.tolerations.0.effect NoSchedule
spark.kubernetes.executor.tolerations.0.value spark-executor

# driver pods tolerate when the key spot exists
spark.kubernetes.executor.tolerations.1.key spot
spark.kubernetes.executor.tolerations.1.operator Exists

Why are the changes needed?

Spark already supports using node selectors, which is helpful to pin driver/executor pods to a specific worker group. However, without tainting the workers and setting at least one toleration to the pods, the specialised worker group cannot ensure repelling non-spark pods, which could potentially lead to undesired, potentially unsuitable pods being scheduled in the specialised worker group.

This can be very important for many use cases, such as running spark jobs in AWS spot instances, which can be reclaimed at anytime by Amazon thus making it not suitable for every kind of workload.

Does this PR introduce any user-facing change?

No, it's a new feature.

How was this patch tested?

added both unit tests and k8s integration tests

[AmplabJenkins]

Can one of the admins verify this patch?

[vanzin]

• All changes must be made first in the master branch.
• You'll have to explain why pod templates don't cover this. All your feature step does is add stuff to the pod spec based on Spark config keys, which is what pod templates are supposed to cover (without the need for changing Spark code at all).

[ghost]

Hi @vanzin, thanks for taking the time.

About the pod templates feature, I can see documentation only in the master branch about it, but not in the 2.4 releases of spark. If 2.4 releases supports pod templates as well, I'm glad to try solving the issue without the code changes I proposed.

[vanzin]

2.4 has already been released. We don't add new features to existing releases.

[ghost]

I can confirm the pod template feature works for setting tolerations to pods in spark 3.0.0-preview, thus making this PR unnecessary (unless you're planning a 2.5 release, but that's another discussion). Thanks @vanzin for pointing it out.

Documentation can be found here (for those like me, how struggled to find it): https://spark.apache.org/docs/3.0.0-preview/running-on-kubernetes.html#pod-template