management workload partitioning #703

dhellmann · 2021-03-19T17:20:45Z

This enhancement presents an alternative approach to replace #628.
It describes an approach to allow us to isolate the control plane services
to run on a restricted set of CPUs. This will be especially useful
for resource-constrained enviornments, such as single-node production
deployments, where the user wants to reserve most of the CPU resources
for their own workloads and needs to configure OpenShift to run on a
fixed number of CPUs within the host.

This enhancement presents an alternative approach to replace openshift#628. Signed-off-by: Doug Hellmann <dhellmann@redhat.com>

enhancements/management-workload-partitioning.md

smarterclayton · 2021-03-19T17:44:33Z

enhancements/management-workload-partitioning.md

+
+We want to give cluster administrators control over which workloads
+are run on the management CPUs. Therefore, only pods in namespaces
+labeled with `io.openshift.management: true` will be subject to


Namespaces aren't sufficient. We will have namespaces that have mixed management/not-management workloads and it's not appropriate for us to be forced to split those into different namespaces.

The RBAC behavior can be addressed simply by requiring high/admin privilege to set this on a pod, possibly by the same component that does the cpu request -> workload transition.

cc: @deads2k

Is it possible to say that someone needs a specific role in order to add a specific label to a pod?

What namespaces do we have with mixed workloads?

i agree.
openshift monitoring is a good example.
it has per-node components and per infra host components (prometheus).
in standalone deployments with workers, we only want to compress per-node components.

I'm not sure that what you're saying is incompatible with what is being proposed.

The idea with the pod label is to opt-in to being put on the management pool. The namespace label is about security, and giving the cluster admin control over what gets to use management pools at all.

Are we saying that we can't use the namespace because that gives anyone in that namespace the ability to run on the special pool?

we already do perms checks in pod admission, this is no different.

@smarterclayton are you opposed to the system of dual-opt-in using labels on both namespaces and pods? Or was it not clear that it was a dual system?

The text changed enough that this comment thread is detached from the diff view, but I don't think we can resolve it until we hear from @smarterclayton

enhancements/management-workload-partitioning.md

markmc

Awesome to see this proposal fleshed out so quickly, thanks!

enhancements/management-workload-partitioning.md

fix url to previous proposal Co-authored-by: Mark McLoughlin <markmc@redhat.com>

derekwaynecarr · 2021-03-19T19:26:15Z

/assign

enhancements/management-workload-partitioning.md

dhellmann · 2021-03-19T19:27:07Z

enhancements/management-workload-partitioning.md

+
+We want to give cluster administrators control over which workloads
+are run on the management CPUs. Therefore, only pods in namespaces
+labeled with `io.openshift.management: true` will be subject to


Is it possible to say that someone needs a specific role in order to add a specific label to a pod?

What namespaces do we have with mixed workloads?

enhancements/management-workload-partitioning.md

dhellmann · 2021-03-22T18:19:50Z

enhancements/management-workload-partitioning.md

+
+We want to give cluster administrators control over which workloads
+are run on the management CPUs. Therefore, only pods in namespaces
+labeled with `io.openshift.management: true` will be subject to


@smarterclayton are you opposed to the system of dual-opt-in using labels on both namespaces and pods? Or was it not clear that it was a dual system?

enhancements/management-workload-partitioning.md

Signed-off-by: Doug Hellmann <dhellmann@redhat.com>

…rver code UPSTREAM: <carry>: openshift-kube-apiserver: enabled conversion gen for admission configs UPSTREAM: <carry>: openshift-kube-apiserver/admission: fix featuregates resource name UPSTREAM: <carry>: openshift-kube-apiserver/admission: add missing FeatureSets UPSTREAM: <carry>: openshift-kube-apiserver: use github.com/openshift/apiserver-library-go/pkg/labelselector UPSTREAM: <carry>: openshift authenticator: don't allow old-style tokens UPSTREAM: <carry>: oauth-authn: support sha256 prefixed tokens UPSTREAM: <carry>: oauth-token-authn: switch to sha256~ prefix UPSTREAM: <carry>: oauth-token-authn: add sha256~ support to bootstrap authenticator UPSTREAM: <drop>: remove the openshift authenticator from the apiserver In 4.8, we moved the authenticator to be configured via webhookTokenAuthenticators to an endpoint in the oauth-apiserver, this should now be safe to remove. UPSTREAM: <carry>: set ResourceQuotaValidationOptions to true When PodAffinityNamespaceSelector goes to beta or GA this might affect how our ClusterResourceQuota might work UPSTREAM: <carry>: simplify the authorizer patch to allow the flags to function UPSTREAM: <carry>: eliminate unnecessary closure in openshift configuration wiring UPSTREAM: <carry>: add crdvalidation for apiserver.spec.tlsSecurityProfile UPSTREAM: <carry>: openshift-kube-apiserver: Add custom resource validation for network spec UPSTREAM: <carry>: stop overriding flags that are explicitly set UPSTREAM: <carry>: add readyz check for openshift apiserver availability UPSTREAM: <carry>: wait for oauth-apiserver accessibility UPSTREAM: <carry>: provide a new admission plugin to mutate management pods CPUs requests The ManagementCPUOverride admission plugin replaces pod container CPU requests with a new management resource. It applies to all pods that: 1. are in an allowed namespace 2. and have the workload annotation. It also sets the new management resource request and limit and set resource annotation that CRI-O can recognize and apply the relevant changes. For more information, see - openshift/enhancements#703 Conditions for CPUs requests deletion: 1. The namespace should have allowed annotation "workload.openshift.io/allowed": "management" 2. The pod should have management annotation: "workload.openshift.io/management": "{"effect": "PreferredDuringScheduling"}" 3. All nodes under the cluster should have new management resource - "management.workload.openshift.io/cores" 4. The CPU request deletion will not change the pod QoS class UPSTREAM: <carry>: Does not prevent pod creation because of no nodes reason when it runs under the regular cluster Check the `cluster` infrastructure resource status to be sure that we run on top of a SNO cluster and in case if the pod runs on top of regular cluster, exit before node existence check. UPSTREAM: <carry>: do not mutate pods when it has a container with both CPU request and limit Removing the CPU request from the container that has a CPU limit will result in the defaulter to set the CPU request back equals to the CPU limit. UPSTREAM: <carry>: Reject the pod creation when we can not decide the cluster type It is possible a race condition between pod creation and the update of the infrastructure resource status with correct values under Status.ControlPlaneTopology and Status.InfrastructureTopology. UPSTREAM: <carry>: add CRD validation for dnses Add an admission plugin that validates the dnses.operator.openshift.io custom resource. For now, the plugin only validates the DNS pod node-placement parameters. This commit fixes bug 1967745. https://bugzilla.redhat.com/show_bug.cgi?id=1967745 * openshift-kube-apiserver/admission/customresourcevalidation/attributes.go (init): Install operatorv1 into supportedObjectsScheme. * openshift-kube-apiserver/admission/customresourcevalidation/customresourcevalidationregistration/cr_validation_registration.go (AllCustomResourceValidators, RegisterCustomResourceValidation): Register the new plugin. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns.go: New file. (PluginName): New const. (Register): New function. Register the plugin. (toDNSV1): New function. Convert a runtime object to a versioned DNS. (dnsV1): New type to represent a runtime object that is validated as a versioned DNS. (ValidateCreate, ValidateUpdate, ValidateStatusUpdate): New methods. Implement the ObjectValidator interface, using the validateDNSSpecCreate and validateDNSSpecUpdate helpers. (validateDNSSpecCreate, validateDNSSpecUpdate): New functions. Validate a DNS, using the validateDNSSpec helper. (validateDNSSpec): New function. Validate the spec field of a DNS, using the validateDNSNodePlacement helper. (validateDNSNodePlacement): New function. Validate the node selector and tolerations in a DNS's node-placement parameters, using validateTolerations. (validateTolerations): New function. Validate a slice of corev1.Toleration. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns_test.go: New file. (TestFailValidateDNSSpec): Verify that validateDNSSpec rejects invalid DNS specs. (TestSucceedValidateDNSSpec): Verify that validateDNSSpec accepts valid DNS specs. * vendor/*: Regenerate. UPSTREAM: <carry>: prevent the kubecontrollermanager service-ca from getting less secure UPSTREAM: <carry>: allow SCC to be disabled on a per-namespace basis UPSTREAM: <carry>: verify required http2 cipher suites In the Apiserver admission, we need to return an error if the required http2 cipher suites are missing from a custom tlsSecurityProfile. Currently, custom cipher suites missing ECDHE_RSA_WITH_AES_128_GCM_SHA256 or ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 result in invalid http2 Server configuration causing the apiservers to crash. See: go/x/net/http2.ConfigureServer for futher information. UPSTREAM: <carry>: drop the warning to use --keep-annotations When a user runs the `oc debug` command for the pod with the management resource, we will inform him that he should pass `--keep-annotations` parameter to the debug command. UPSTREAM: <carry>: admission/managementcpusoverride: cover the roll-back case During the upgrade and roll-back flow 4.7->4.8->4.7, the topology related fields under the infrastructure can be empty because the old API does not support them. The code will equal the empty infrastructure section with the current one. When the status has some other non-empty field, and topology fields are empty, we assume that the cluster currently passes via roll-back and not via the clean install. UPSTREAM: <carry>: Remove pod warning annotation when workload partitioning is disabled UPSTREAM: <carry>: use new access token inactivity timeout field. UPSTREAM: <carry>: apirequestcount validation UPSTREAM: <carry>: Added config node object validation for extreme latency profiles UPSTREAM: <carry>: Add Upstream validation in the DNS admission check patches UPSTREAM: <carry>: Make RestrictedEndpointsAdmission check NotReadyAddresses UPSTREAM: <carry>: Make RestrictedEndpointsAdmission restrict EndpointSlices as well Moved SkipSystemMasterAuthorizers to the authorizer. UPSTREAM: <carry>: Add validation plugin for CRD-based route parity. UPSTREAM: <carry>: Add host assignment plugin for CRD-based routes. UPSTREAM: <carry>: Apply shared defaulters to CRD-based routes. Signed-off-by: Artyom Lukianov <alukiano@redhat.com> Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> Signed-off-by: Swarup Ghosh <swghosh@redhat.com> OpenShift-Rebase-Source: 932411e OpenShift-Rebase-Source: 1899555 OpenShift-Rebase-Source: 453583e OpenShift-Rebase-Source: bf7e23e UPSTREAM: <carry>: STOR-829: Add CSIInlineVolumeSecurity admission plugin The CSIInlineVolumeSecurity admission plugin inspects inline CSI volumes on pod creation and compares the security.openshift.io/csi-ephemeral-volume-profile label on the CSIDriver object to the pod security profile on the namespace. OpenShift-Rebase-Source: a65c34b UPSTREAM: <carry>: add icsp,idms,itms validation reject creating icsp with idms/itms exist Reject icsp with idms.itms resources exists. According to the discuusion resolution https://docs.google.com/document/d/13h6IJn8wlzXdiPMvCWlMEHOXXqEZ9_GYOl02Wldb3z8/edit?usp=sharing, one of current icsp or new mirror setting crd should be rejected if a user tries to use them on the same cluster. Signed-off-by: Qi Wang <qiwan@redhat.com> UPSTREAM: <carry>: node admission plugin for cpu partitioning The ManagedNode admission plugin makes the Infrastructure.Status.CPUPartitioning field authoritative. This validates that nodes that wish to join the cluster are first configured to properly handle workload pinning For more information see - openshift/enhancements#1213 Signed-off-by: ehila <ehila@redhat.com> UPSTREAM: <carry>: kube-apiserver: allow injection of kube-apiserver options UPSTREAM: <carry>: kube-apiserver: allow rewiring OpenShift-Rebase-Source: 56b49c9 OpenShift-Rebase-Source: bcf574c UPSTREAM: <carry>: STOR-1270: Admission plugin to deny deletion of storages.operator.openshift.io UPSTREAM: <carry>: support for both icsp and idms objects Revert: openshift#1310 Add support for ICSP and IDMS objects living at the same time. Signed-off-by: Qi Wang <qiwan@redhat.com> UPSTREAM: <carry>: openshift-kube-apiserver: add openshift-kube-apisever code

UPSTREAM: <carry>: openshift-kube-apiserver: enabled conversion gen for admission configs UPSTREAM: <carry>: openshift-kube-apiserver/admission: fix featuregates resource name UPSTREAM: <carry>: openshift-kube-apiserver/admission: add missing FeatureSets UPSTREAM: <carry>: openshift-kube-apiserver: use github.com/openshift/apiserver-library-go/pkg/labelselector UPSTREAM: <carry>: openshift authenticator: don't allow old-style tokens UPSTREAM: <carry>: oauth-authn: support sha256 prefixed tokens UPSTREAM: <carry>: oauth-token-authn: switch to sha256~ prefix UPSTREAM: <carry>: oauth-token-authn: add sha256~ support to bootstrap authenticator UPSTREAM: <drop>: remove the openshift authenticator from the apiserver In 4.8, we moved the authenticator to be configured via webhookTokenAuthenticators to an endpoint in the oauth-apiserver, this should now be safe to remove. UPSTREAM: <carry>: set ResourceQuotaValidationOptions to true When PodAffinityNamespaceSelector goes to beta or GA this might affect how our ClusterResourceQuota might work UPSTREAM: <carry>: simplify the authorizer patch to allow the flags to function UPSTREAM: <carry>: eliminate unnecessary closure in openshift configuration wiring UPSTREAM: <carry>: add crdvalidation for apiserver.spec.tlsSecurityProfile UPSTREAM: <carry>: openshift-kube-apiserver: Add custom resource validation for network spec UPSTREAM: <carry>: stop overriding flags that are explicitly set UPSTREAM: <carry>: add readyz check for openshift apiserver availability UPSTREAM: <carry>: wait for oauth-apiserver accessibility UPSTREAM: <carry>: provide a new admission plugin to mutate management pods CPUs requests The ManagementCPUOverride admission plugin replaces pod container CPU requests with a new management resource. It applies to all pods that: 1. are in an allowed namespace 2. and have the workload annotation. It also sets the new management resource request and limit and set resource annotation that CRI-O can recognize and apply the relevant changes. For more information, see - openshift/enhancements#703 Conditions for CPUs requests deletion: 1. The namespace should have allowed annotation "workload.openshift.io/allowed": "management" 2. The pod should have management annotation: "workload.openshift.io/management": "{"effect": "PreferredDuringScheduling"}" 3. All nodes under the cluster should have new management resource - "management.workload.openshift.io/cores" 4. The CPU request deletion will not change the pod QoS class UPSTREAM: <carry>: Does not prevent pod creation because of no nodes reason when it runs under the regular cluster Check the `cluster` infrastructure resource status to be sure that we run on top of a SNO cluster and in case if the pod runs on top of regular cluster, exit before node existence check. UPSTREAM: <carry>: do not mutate pods when it has a container with both CPU request and limit Removing the CPU request from the container that has a CPU limit will result in the defaulter to set the CPU request back equals to the CPU limit. UPSTREAM: <carry>: Reject the pod creation when we can not decide the cluster type It is possible a race condition between pod creation and the update of the infrastructure resource status with correct values under Status.ControlPlaneTopology and Status.InfrastructureTopology. UPSTREAM: <carry>: add CRD validation for dnses Add an admission plugin that validates the dnses.operator.openshift.io custom resource. For now, the plugin only validates the DNS pod node-placement parameters. This commit fixes bug 1967745. https://bugzilla.redhat.com/show_bug.cgi?id=1967745 * openshift-kube-apiserver/admission/customresourcevalidation/attributes.go (init): Install operatorv1 into supportedObjectsScheme. * openshift-kube-apiserver/admission/customresourcevalidation/customresourcevalidationregistration/cr_validation_registration.go (AllCustomResourceValidators, RegisterCustomResourceValidation): Register the new plugin. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns.go: New file. (PluginName): New const. (Register): New function. Register the plugin. (toDNSV1): New function. Convert a runtime object to a versioned DNS. (dnsV1): New type to represent a runtime object that is validated as a versioned DNS. (ValidateCreate, ValidateUpdate, ValidateStatusUpdate): New methods. Implement the ObjectValidator interface, using the validateDNSSpecCreate and validateDNSSpecUpdate helpers. (validateDNSSpecCreate, validateDNSSpecUpdate): New functions. Validate a DNS, using the validateDNSSpec helper. (validateDNSSpec): New function. Validate the spec field of a DNS, using the validateDNSNodePlacement helper. (validateDNSNodePlacement): New function. Validate the node selector and tolerations in a DNS's node-placement parameters, using validateTolerations. (validateTolerations): New function. Validate a slice of corev1.Toleration. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns_test.go: New file. (TestFailValidateDNSSpec): Verify that validateDNSSpec rejects invalid DNS specs. (TestSucceedValidateDNSSpec): Verify that validateDNSSpec accepts valid DNS specs. * vendor/*: Regenerate. UPSTREAM: <carry>: prevent the kubecontrollermanager service-ca from getting less secure UPSTREAM: <carry>: allow SCC to be disabled on a per-namespace basis UPSTREAM: <carry>: verify required http2 cipher suites In the Apiserver admission, we need to return an error if the required http2 cipher suites are missing from a custom tlsSecurityProfile. Currently, custom cipher suites missing ECDHE_RSA_WITH_AES_128_GCM_SHA256 or ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 result in invalid http2 Server configuration causing the apiservers to crash. See: go/x/net/http2.ConfigureServer for futher information. UPSTREAM: <carry>: drop the warning to use --keep-annotations When a user runs the `oc debug` command for the pod with the management resource, we will inform him that he should pass `--keep-annotations` parameter to the debug command. UPSTREAM: <carry>: admission/managementcpusoverride: cover the roll-back case During the upgrade and roll-back flow 4.7->4.8->4.7, the topology related fields under the infrastructure can be empty because the old API does not support them. The code will equal the empty infrastructure section with the current one. When the status has some other non-empty field, and topology fields are empty, we assume that the cluster currently passes via roll-back and not via the clean install. UPSTREAM: <carry>: Remove pod warning annotation when workload partitioning is disabled UPSTREAM: <carry>: use new access token inactivity timeout field. UPSTREAM: <carry>: apirequestcount validation UPSTREAM: <carry>: Added config node object validation for extreme latency profiles UPSTREAM: <carry>: Add Upstream validation in the DNS admission check patches UPSTREAM: <carry>: Make RestrictedEndpointsAdmission check NotReadyAddresses UPSTREAM: <carry>: Make RestrictedEndpointsAdmission restrict EndpointSlices as well Moved SkipSystemMasterAuthorizers to the authorizer. UPSTREAM: <carry>: Add validation plugin for CRD-based route parity. UPSTREAM: <carry>: Add host assignment plugin for CRD-based routes. UPSTREAM: <carry>: Apply shared defaulters to CRD-based routes. Signed-off-by: Artyom Lukianov <alukiano@redhat.com> Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> Signed-off-by: Swarup Ghosh <swghosh@redhat.com> OpenShift-Rebase-Source: 932411e OpenShift-Rebase-Source: 1899555 OpenShift-Rebase-Source: 453583e OpenShift-Rebase-Source: bf7e23e UPSTREAM: <carry>: STOR-829: Add CSIInlineVolumeSecurity admission plugin The CSIInlineVolumeSecurity admission plugin inspects inline CSI volumes on pod creation and compares the security.openshift.io/csi-ephemeral-volume-profile label on the CSIDriver object to the pod security profile on the namespace. OpenShift-Rebase-Source: a65c34b UPSTREAM: <carry>: add icsp,idms,itms validation reject creating icsp with idms/itms exist Reject icsp with idms.itms resources exists. According to the discuusion resolution https://docs.google.com/document/d/13h6IJn8wlzXdiPMvCWlMEHOXXqEZ9_GYOl02Wldb3z8/edit?usp=sharing, one of current icsp or new mirror setting crd should be rejected if a user tries to use them on the same cluster. Signed-off-by: Qi Wang <qiwan@redhat.com> UPSTREAM: <carry>: node admission plugin for cpu partitioning The ManagedNode admission plugin makes the Infrastructure.Status.CPUPartitioning field authoritative. This validates that nodes that wish to join the cluster are first configured to properly handle workload pinning For more information see - openshift/enhancements#1213 Signed-off-by: ehila <ehila@redhat.com> UPSTREAM: <carry>: kube-apiserver: allow injection of kube-apiserver options UPSTREAM: <carry>: kube-apiserver: allow rewiring OpenShift-Rebase-Source: 56b49c9 OpenShift-Rebase-Source: bcf574c UPSTREAM: <carry>: openshift-kube-apiserver: add kube-apiserver patches