BUG 1782516: Disable client side rate limiting in Azure. #3259
Conversation
|
@enxebre: This pull request references Bugzilla bug 1782516, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
bugzilla/valid-bug
enxebre
changed the title
enxebre
force-pushed
the
azure-perms
branch
2 times, most recently
from
bf86eb2
to
3858e8a
Compare
can you expand a little? also today we have a SP per operator created by the cred-minter, where cred-minter itself uses the SP used to install. _ we can leave overlook this here as the configuration only controls the k8s-cloud-provider_ So since we are using managed identity for control plane VMs we are okay to remove the limits? The original limits were from AKS, was AKS using shared SP that they use the client side limits? |
|
Secondly, how do we get our already existing clusters to follow this new limits? |
|
also @enxebre it would be very helpful to move the PR desc into the commit message too. |
Client side rate limiting is being problematic for fresh installs and scaling operations [1] Azure ARM throttling is applied at subscription level, so client side rate limiting helps to prevent cluster sharing the same subscription from disrupting each other. However there's lower limits which apply at the SP/tenant and resource level e.g ARM limits the number of write calls per service principal to 1200/hour [2]. Since we ensure particular SPs per cluster via Cloud Credential Operator it should be relatively safe to disable the client rate limiting Orthogonally to this some improvements on the rate limiting and back off mechanisms are being added to the cloud provider. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1782516. [2] https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling [3] kubernetes-sigs/cloud-provider-azure#247
My understanding is that we mostly do tenant-level operations and limits are scoped to SP. Since we have granular SP as you described above we wouldn't likely throttle the subscription. https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling#subscription-and-tenant-limits
I know they set it to prevent subscription level throttling but I'm not 100% sure they share a SP. FWIW ARO has rate limiting turned off.
Good point. afaik nothing owns |
|
The changes introduced in this PR are contained to Azure, so aws|libvirt|openstack tests should not be failing. /test e2e-openstack |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: abhinavdahiya The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
|
@enxebre I can lgtm this if it helps? |
|
https://bugzilla.redhat.com/show_bug.cgi?id=1826069 created for 4.4, please backport. https://bugzilla.redhat.com/show_bug.cgi?id=1826073 created for 4.3, please backport. |
|
@abhinavdahiya do the |
|
Note that this fix needs to be backported to 4.4 and 4.3. See https://bugzilla.redhat.com/show_bug.cgi?id=1782516#c24 for details. |
|
@enxebre can you sync with @abhinavdahiya to get this PR in a state to get merged? |
|
/test e2e-azure |
1 similar comment
|
/test e2e-azure |
|
/lgtm |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@enxebre: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@enxebre: All pull requests linked via external trackers have merged: openshift/installer#3259. Bugzilla bug 1782516 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cherrypick release-4.4 |
|
@michaelgugino: new pull request created: #3616 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Client side rate limiting is being problematic for fresh installs and scaling operations [1]
Azure ARM throttling is applied at subscription level, so client side rate limiting helps to prevent cluster sharing the same subscription from disrupting each other.
However there's lower limits which apply at the SP/tenant and resource level e.g ARM limits the number of write calls per service principal to 1200/hour [2]. Since we ensure particular SPs per cluster via Cloud Credential Operator it should be relatively safe to disable the client rate limiting
Orthogonally to this some improvements on the rate limiting and back off mechanisms are being added to the cloud provider.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1782516.
[2] https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling
[3] kubernetes-sigs/cloud-provider-azure#247