P&F: enable mutating work estimator

[wojtek-t]

NONE

/sig api-machinery

[wojtek-t]

/test pull-kubernetes-kubemark-e2e-gce-big
/test pull-kubernetes-kubemark-e2e-gce-scale

[fedebongio]

/triage accepted

[wojtek-t]

/test pull-kubernetes-kubemark-e2e-gce-big
/test pull-kubernetes-kubemark-e2e-gce-scale

[wojtek-t]

/test pull-kubernetes-kubemark-e2e-gce-big
/test pull-kubernetes-kubemark-e2e-gce-scale

[wojtek-t]

/test pull-kubernetes-kubemark-e2e-gce-big
/test pull-kubernetes-kubemark-e2e-gce-scale

[wojtek-t]

/test pull-kubernetes-kubemark-e2e-gce-scale

[wojtek-t]

/test pull-kubernetes-kubemark-e2e-gce-scale

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apiserver/pkg/util/flowcontrol/OWNERS [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wojtek-t]

/test pull-kubernetes-kubemark-e2e-gce-scale

[MikeSpreitzer]

So, if I understand correctly, this is now intended to be reviewed and merged. The title of this PR should be updated to remove the outdated remarks to the contrary.

[k8s-triage-robot]

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

[MikeSpreitzer]

I did some looking around in https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/106085/pull-kubernetes-kubemark-e2e-gce-scale/1469204072497156096 . That tested the merge of https://github.com/kubernetes/kubernetes/tree/cc6f12583f2b611e9469a6b2e0247f028aae246b (master) and https://github.com/wojtek-t/kubernetes/tree/8f5ece0548b59b1f8706f79af814d9f7700acb5c (the PR).

I went looking for apiserver SLO violations, but mistakenly averaged over 1 minute instead of 5. The table columns are mean and max.

[MikeSpreitzer]

Here are the concurrency limits.

time series	value
apiserver_flowcontrol_request_concurrency_limit{endpoint="apiserver", instance="10.40.3.217:443", job="master", namespace="monitoring", priority_level="catch-all", service="master"}	14
apiserver_flowcontrol_request_concurrency_limit{endpoint="apiserver", instance="10.40.3.217:443", job="master", namespace="monitoring", priority_level="global-default", service="master"}	53
apiserver_flowcontrol_request_concurrency_limit{endpoint="apiserver", instance="10.40.3.217:443", job="master", namespace="monitoring", priority_level="leader-election", service="master"}	27
apiserver_flowcontrol_request_concurrency_limit{endpoint="apiserver", instance="10.40.3.217:443", job="master", namespace="monitoring", priority_level="node-high", service="master"}	105
apiserver_flowcontrol_request_concurrency_limit{endpoint="apiserver", instance="10.40.3.217:443", job="master", namespace="monitoring", priority_level="system", service="master"}	79
apiserver_flowcontrol_request_concurrency_limit{endpoint="apiserver", instance="10.40.3.217:443", job="master", namespace="monitoring", priority_level="workload-high", service="master"}	105
apiserver_flowcontrol_request_concurrency_limit{endpoint="apiserver", instance="10.40.3.217:443", job="master", namespace="monitoring", priority_level="workload-low", service="master"}	262

[MikeSpreitzer]

Only one priority level came close to using all its concurrency. That is workload-high.

[MikeSpreitzer]

The kube-apiserver had the highest CPU utilization, peaking around 44 CPUs.

[MikeSpreitzer]

Only one flow schema had high watch counts, kube-system-service-accounts (priority level workload-high).

[MikeSpreitzer]

Here's something odd: one sort of WATCH took less than a minute! It was resource=pods, scope=namespace.

[MikeSpreitzer]

More regarding apiserver SLO. Looks OK.

[dims]

@wojtek-t is this ready?

[MikeSpreitzer]

I myself would like to see this vetted in the context of #106725 first. Sadly, we do not yet have consensus on that PR.

[wojtek-t]

@MikeSpreitzer

Here's something odd: one sort of WATCH took less than a minute! It was resource=pods, scope=namespace.

This is expected - those are coming from test and we just set them up to wait for certain deployments to stabilize (we create them when the deployment is changed). This is expected it takes less than a minute a lot of times.

I myself would like to see this vetted in the context of #106725 first. Sadly, we do not yet have consensus on that PR.

That PR won't affect results of this PR at this point. The reason some version of it was needed initially was that kubemark was still using endpoints instead of endpointslices so we have thousands of watches on endpoints. After #106740, this is no longer true, so that PR won't really change results for our tests with this PR.
This is ready for stamping.

[MikeSpreitzer]

/LGTM