SHIP-0039: Build Scheduler Options #205
Conversation
Provisional proposal to add API options to control where build pods are scheduled. These will rely on core Kubernetes features related to pod scheduling. This proposal was partially motivated by the multi-arch feature discussions, where it was revealed that we currently have no means of controlling where build pods are scheduled. While these features may support a future proof of concept for multi-arch builds, orchestrating multi-arch builds end to end is out of scope.
pull-request-size
bot
added
the
size/L
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
|
||
| - Builds can be CPU/memory/storage intensive. Scheduling on larger worker nodes with additional | ||
| memory or compute can help ensure builds succeed. | ||
| - Clusters may have mutiple worker node architectures and even OS (Windows nodes). Container images |
There was a problem hiding this comment.
To not make false assumptions, shipwright builds that have a source or use image processing cannot run on Windows because we do not produce Windows images for our shipwright-injected steps.
There was a problem hiding this comment.
Noted in shipwright-io/build#1613
ships/0039-build-scheduler-opts.md
Outdated
| - Allow build pods to run on specific nodes using node selectors. | ||
| - Allow build pods to set node affinity/anti-affinity rules. | ||
| - Allow build pods to tolerate node taints. | ||
| - Allow node selection, pod affinity, and taint toleration to be set at the cluster level. |
There was a problem hiding this comment.
You mean on the Shipwright Build controller level as default? I like that, is possible so far by configuring the default pod template of Tekton, but that applies to all of Tekton.
Downstream we actually need it that users cannot configure any node selectors or tolerations (because they don't see nodes anyway but only what is in their namespace). Would a global configuration imply that settings on a build and buildrun are forbidden. Or would we add another configuration option for that? Or do we delegate forbidding things into policy tools or custom webhooks ?
There was a problem hiding this comment.
Would a global configuration imply that settings on a build and buildrun are forbidden. Or would we add another configuration option for that?
For idea/inspiration: in OpenShift we have two levels of configuration that were called "defaults" and "overrides." "Defaults" were applied if the user/developer did not specify them, "overrides" wrote over what the user specified. I'm inclined to prefer supporting only "defaults" at the cluster level.
Or do we delegate forbidding things into policy tools or custom webhooks ?
If the need is to forbid users from changing the default, my gut recommendation is to address this via an admission controller like OPA or Kyverno.
@siamaksade curious what your thought is here.
There was a problem hiding this comment.
I decided to drop the cluster defaults - this can be considered as a follow-up feature.
Regarding blocking usage of these fields - I added it as a non-goal, recommending this be blocked using an admission controler like OPA Gatekeeper or Kyverno.
- Removed the goal to set pod scheduling options at the cluster level. This adds significant scope since the build controller would need to support this configuration via a file or large set of environment variables. This goal is not ruled out completely - I'm recommending it as a follow up feature so the community can iterate and deliver quickly. - Added support for custom schedulers - this feature has been in Kubernetes for quite some time, though its adoption is somewhat limited due to the challenges of writing a custom scheduler. Signed-off-by: Adam Kaplan <adam.kaplan@redhat.com>
Affinity and anti-affinity is an incredibly rich and complex API in Kubernetes. Most of the other scheduler tuning APIs in this SHIP are simpler to understand and can be wired directly to underlying APIs (Tekton today, potentially another implementation in the future). Exposing the entire affinity rule API will make it challenging for developers to utilize. We should instead strive for a simpler API that makes it easy for developers to express their intent. This abstraction can then be translated to the underlying scheduler APIs. Such an effort is not trivial and warrants a separate feature/SHIP. Signed-off-by: Adam Kaplan <adam.kaplan@redhat.com>
Changes
Provisional proposal to add API options to control where build pods are scheduled. These will rely on core Kubernetes features related to pod scheduling.
This proposal was partially motivated by the multi-arch feature discussions, where it was revealed that we currently have no means of controlling where build pods are scheduled. While these features may support a future proof of concept for multi-arch builds, orchestrating multi-arch builds end to end is out of scope.
Submitter Checklist
See the contributor guide
for details on coding conventions, github and prow interactions, and the code review process.
Release Notes