flowschema for etcd operator traffic

[tkashem]

Traffic from control plane operators are important (kas-o, oas-o, auth operator, etcd operator), they have a dedicated concurrency pool now.

See openshift/cluster-kube-apiserver-operator#966 for more details.

[hexfusion]

/hold
@tkashem can you walk us through this. I understand that we have a dedicated concurrency pool.

questions:

• does this pool have a limit?
• if yes what happens if we exceed it?
• if yes how can the operator understand the limit thresholds and how close we are to it.
• if yes did we have a limit before?
• if yes what happened if we exceeded it?

[hexfusion]

maybe a link to docs would help :)

[tkashem]

https://kubernetes.io/blog/2020/04/06/kubernetes-1-18-feature-api-priority-and-fairness-alpha/

does this pool have a limit?

yes, all control plane operators (kas-o, etcd-o, oauth-o, oas-o) share the same concurrency pool, right now its share is 10. By default (max-mutating=1000 and max-readonly=3000) it translates to about 175 concurrent requests.

if yes what happens if we exceed it?

the apiserver will reject the request with a 429.

if yes how can the operator understand the limit thresholds and how close we are to it.

There are metrics in place to see if there are requests in queue for this particular request flow (all requests originating from etcd operator)

if yes did we have a limit before?

The previous limit was not as fine grained as this. Previously we have max inflight requests allowed (readonly=3000 and mutating=1000)

if yes did we have a limit before?

same, the apiserver would reject it with a 429. The difference now is that if a bad actor (a different user) floods the apiserver with requests it won't have any affect on the etc operator traffic.

[tkashem]

@hexfusion this shows how many requests per second etcd operator is generating. :)

[hexfusion]

@hexfusion this shows how many requests per second etcd operator is generating. :)

ok so plenty of room to expand! thanks for details

[hexfusion]

/hold cancel

cc @retroflexer @ironcladlou

[deads2k]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, tkashem

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@tkashem: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-disruptive	272fc26	link	/test e2e-disruptive

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[tkashem]

/retest