OCPVE-379: fix: avoid checking resources for BestEffort pods #28006
Conversation
eggfoobar
changed the title
|
@eggfoobar: This pull request references OCPVE-379 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
the
jira/valid-reference
eggfoobar
force-pushed
the
fix-cpu-partitioning-flakes
branch
from
c062f0f
to
6359b46
Compare
|
The point of cpu affinity is to keep the platform from taking cpu from the workloads. Our platform pods, even best effort ones, need to be controlled for that purpose |
|
Oh sorry @deads2k, I wasn't clear what's happening, the desired CPU affinity is not being violated here. There are two things that happen for workload partitioning, a CPU affinity and a CPU share change. The flake was around the CPU Share itself. Since we checked that pods containing the workload partitioning annotations also had their resource requests altered to the custom resource field, this would flake on the tests that spun up BestEffort deployments and daemonsets because the default behavior is to apply the workload annotation with the minimum 2 CPU shares and not add/change resource requests since none are present at admission time. Unless we should require that all platform pods must contain resource requests, this would be safe to skip since the CPU affinity is still respected, and the workload get's a default CPUShare value of 2. What do you think? |
|
Adding some more context around CPU shares. A cpu share is a relative amount of time a process is granted on a CPU, in Kubernetes 1CPU = 1024CPU shares. When there is no CPU time contention, nothing happens and the process is allowed to use more, however if there is CPU time contention, then the cpu share will be enforced and processes with a higher CPU share will get that proportional percentage of time on the CPU. i.e two process with a 512 cpu share will each get 50% CPU time. In Kubernetes when a pod has a QoS of BestEffort, the kubelet will automatically give it a default CPU share of 2, meaning it will be scheduled pretty much anywhere since it's so small and if there's contention it will get the smallest amount of time on the CPU. If you need a minimum amount, consider supplying a Normally this is not too much of a problem. However, when using the workload pinning feature ( which allows cluster admins to limit the number of CPUs the platform pods have access to ), the probability of CPU time contention goes up and |
eggfoobar
force-pushed
the
fix-cpu-partitioning-flakes
branch
from
6359b46
to
748ea47
Compare
|
@eggfoobar: This pull request references OCPVE-379 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
eggfoobar
force-pushed
the
fix-cpu-partitioning-flakes
branch
from
748ea47
to
c503d91
Compare
Signed-off-by: ehila <ehila@redhat.com> feat: update to use exclude list instead of blanket ignore on qos Signed-off-by: ehila <ehila@redhat.com>
eggfoobar
force-pushed
the
fix-cpu-partitioning-flakes
branch
from
c503d91
to
0057332
Compare
|
the networking team ack'd these. The CPU slice is extremely small and bugs were opened so there's long term tracking as well. /lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: deads2k, eggfoobar The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/retest-required |
|
@eggfoobar: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
openshift-merge-robot
merged commit 21f0791
into
openshift:master
The reason the flakes have been occurring is due to some tests around the network-operator creating DaemonSets and Deployments that do not contain resources. Adding a skip list for resources that are okay being BestEffort and added comment to code to be explicit on the meaning of best effort in a workload partitioned cluster. This does not effect the CPU affinity for workload partitioning.
Test-Grid Flaks: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.14-informing#periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-cpu-partitioning&include-filter-by-regex=CPU%20Partitioning
Ref Resources on the network operator:
DaemonSet
Deployment
/assign @deads2k