allow setting Kubernetes tolerations in KFP pipelines

[thesuperzapper]

Related to #2681

Background

It is common practice in Kubernetes to apply a taint to Nodes with accelerator hardware (like GPUs), so that resources are not wasted by Pods which do not require the accelerator.

Currently, there is no way to specify a Pod toleration in Elyra Kubeflow pipelines, this means that running Elyra pipelines with GPUs is effectively impossible on a well-designed Kubernetes cluster.

Solution

We can resolve this by allowing users to specify a list of "tolerations" in a similar way to our "volume mounts" (see PR #2799).

We only need to allow the user to set key, operator, value, and effect, because tolerationSeconds is intended for cluster management, rather than scheduling.

References:

• See the Taints and Tolerations k8s docs
• See kfp.dsl.BaseOp().add_toleration() for more information about adding tolerations in KFP
• See the Toleration v1 core spec for details about how a toleration is defined.

Examples

This toleration will match taints of key=my-key and value=my-value with effect=NoSchedule:

- key: "my-key"
  operator: "Equal"
  value: "my-value"
  effect: "NoSchedule"

This toleration will match taints of key=my-key and value=my-value with ANY effect:

- key: "my-key"
  operator: "Equal"
  value: "my-value"

This toleration will match taints of key=my-key and ANY value with ANY effect:

- key: "my-key"
  operator: "Exists"

[ptitzler]

Thanks for providing the detailed examples and resource links! They made it easy to investigate what it would take to support this. It looks like this should be a straightforward implementation. Since user-friendly GUI support depends on #2780, we might want to aim for two phase delivery if the prerequisite is not ready in time:

• phase 1 (3.11.0):
  • tolerations can be declared as pipeline default property and node property (generic nodes and custom nodes)
  • zero or more tolerations can be declared (similar to volumes, environment variables, ...)
  • tolerations must be specified as a single string (similar to what is currently required for volumes, environment variables, ...), e.g. key:operator:value:effect to be consistent
  • limited support for validation
• phase 2:
  • GUI support (TL;DR from https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#toleration-v1-core)
    • key is a text input field, which can be empty
    • operator is an enum selector (Exists (default selection), Equal )
    • value is a text input field, which can be empty
    • effect is an enum selector (e.g. <empty> (default selection), NoSchedule, NoExecute, PreferNoSchedule)

If possible we'll try to avoid a two phase delivery as that would require pipeline migration.

[kevin-bates]

If possible we'll try to avoid a two phase delivery as that would require pipeline migration.

If these are persisted in the "encoded" format (e.g. key:operator:value:effect) in both phases, would there be a need for migration?

[ptitzler]

If possible we'll try to avoid a two phase delivery as that would require pipeline migration.

If these are persisted in the "encoded" format (e.g. key:operator:value:effect) in both phases, would there be a need for migration?

You are right! No need in that case.