bug 1885358: protect openshift traffic by using dedicated flowschema #966
Conversation
openshift-ci-robot
added
the
do-not-merge/work-in-progress
|
/retest |
tkashem
changed the title
openshift-ci-robot
removed
the
do-not-merge/work-in-progress
tkashem
changed the title
| - '*' | ||
| verbs: | ||
| - '*' | ||
| subjects: |
There was a problem hiding this comment.
etcd, kas, kcm, ks, auth, oas
There was a problem hiding this comment.
kcm(kube controller manager) andks(scheduler) are already covered by the default set of flowschema shipped.- by auth you mean
openshift-authentication-operator? This is included here. - oas (openshift apiserver) can't be here, it has a higher priority and will be assigned to
aggregated-api-delegated-authpriority configuration.
There was a problem hiding this comment.
found openshift/cluster-openshift-apiserver-operator#398 I like it.
| apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1 | ||
| kind: PriorityLevelConfiguration | ||
| metadata: | ||
| name: openshift-system |
There was a problem hiding this comment.
This should be allowed more requests than openshift-control-plane-operators. Even though it's one server, really don't want to run out.
Also, rename aggregated-api-delegated-auth I think. We'll want to tie the oauth-apiserver to it too, right?
There was a problem hiding this comment.
Also, rename aggregated-api-delegated-auth
sounds good, I am adding a prefix openshift to it.
We'll want to tie the oauth-apiserver to it too, right?
I have added oauth apiserver.
This should be allowed more requests than openshift-control-plane-operators. Even though it's one server, really don't want to run out.
Yes, makes sense. we expect the operators to have a steady call rate (unless there is a bug that causes a hot loop or something) and openshift api servers are tied to the user traffic. I doubled the concurrency share to 20 and reduced for the operators to 10.
| type: Queue | ||
| type: Limited | ||
| --- | ||
| apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1 |
There was a problem hiding this comment.
I'd rather keep rules like this local to operators that use them.
There was a problem hiding this comment.
@deads2k Yes, I thought about it. the only concern I have is the matchingPrecedence field. This defines the matching order, so we need to keep in mind all the other values to make sure the right order is maintained.
There was a problem hiding this comment.
@deads2k Yes, I thought about it. the only concern I have is the
matchingPrecedencefield. This defines the matching order, so we need to keep in mind all the other values to make sure the right order is maintained.
We did this with conventions in the CVO. Our payloads are local. I'm ok moving these in 4.7 instead of 4.6.
tkashem
force-pushed
the
flowschema
branch
2 times, most recently
from
0e4eea7
to
8dfa337
Compare
tkashem
changed the title
openshift-ci-robot
added
bugzilla/severity-urgent
|
@tkashem: This pull request references Bugzilla bug 1885358, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
tkashem
changed the title
openshift-ci-robot
removed
the
bugzilla/severity-urgent
|
@tkashem: No Bugzilla bug is referenced in the title of this pull request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
removed
the
bugzilla/valid-bug
| name: openshift-aggregated-api-delegated-auth | ||
| spec: | ||
| limited: | ||
| assuredConcurrencyShares: 20 |
There was a problem hiding this comment.
bump this to 100 for me? It should only matter when we start hitting pressure.
There was a problem hiding this comment.
bump this to 100 for me? It should only matter when we start hitting pressure
actually, when looking at it, this is an ok starting point.
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: deads2k, tkashem The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
lgtm
|
/bugzilla refresh |
mfojtik
changed the title
|
@tkashem: All pull requests linked via external trackers have merged: Bugzilla bug 1885358 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Define dedicated flowschema and priority configuration that will protect openshift specific traffic.
SARandtokenreviewsfrom bothoasandoauthare very importantoasandoauthrequests,/metricsrequests from openshift-monitoring is pretty importantkcm.workloads-lowgoes below the traffic defined above.Question:
cvoin the control plane operators, or are there traffic from any other critical operators that we should protect?New configuration:
After applying the above configuration, we can see that the number of requests from
workload-lowdrops. This implies that traffic from oas,/metricscall from openshift monitoring, traffic from control plane operators are not being throttled anymore.