Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1785023: Increase pull secrets controller QPS to 100 #84

Conversation

adambkaplan
Copy link
Contributor

@adambkaplan adambkaplan commented Mar 30, 2020

Increase the k8s client QPS limit to 100, 200 burst for the service account pull secrets controller. This ensures that pull secrets for the registry are created as quickly as the tokens are generated.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 30, 2020
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 30, 2020
@@ -99,7 +107,7 @@ func NewControllerContext(

ClientBuilder: OpenshiftControllerClientBuilder{
ControllerClientBuilder: controller.SAControllerClientBuilder{
ClientConfig: rest.AnonymousClientConfig(clientConfig),
ClientConfig: rest.AnonymousClientConfig(kubeClientConfig),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smarterclayton from what I can gather, this client config is used when the controllers call ctx.ClientBuilder.ClientOrDie(...). Every controller in OCM calls this.

My fear is that by using a higher QPS across all controllers, we can DoS the kube apiserver.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adambkaplan adambkaplan force-pushed the fix-sa-pullsecrets-ratelimit branch 2 times, most recently from 02548c6 to 6bd9f1d Compare April 1, 2020 14:38
@adambkaplan adambkaplan changed the title WIP - Increase k8s client QPS to 100 Bug 1765294: Increase pull secrets controller QPS to 100 Apr 1, 2020
@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Apr 1, 2020
@openshift-ci-robot
Copy link
Contributor

@adambkaplan: This pull request references Bugzilla bug 1765294, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.5.0) matches configured target release for branch (4.5.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1765294: Increase pull secrets controller QPS to 100

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link
Contributor

@adambkaplan: This pull request references Bugzilla bug 1765294, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.5.0) matches configured target release for branch (4.5.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1765294: Increase pull secrets controller QPS to 100

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@adambkaplan
Copy link
Contributor Author

/assign @smarterclayton

/cc @bparees @deads2k @mfojtik

@adambkaplan
Copy link
Contributor Author

Test to ensure we don't regress this timing: openshift/origin#24776

Increase the k8s client QPS limit to 100, 200 burst for the service account
pull secrets controller. This ensures that pull secrets for the registry are created
as quickly as the tokens are generated.
@adambkaplan adambkaplan changed the title Bug 1765294: Increase pull secrets controller QPS to 100 Bug 1785023: Increase pull secrets controller QPS to 100 Apr 1, 2020
@openshift-ci-robot
Copy link
Contributor

@adambkaplan: This pull request references Bugzilla bug 1785023, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.5.0) matches configured target release for branch (4.5.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1785023: Increase pull secrets controller QPS to 100

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@adambkaplan
Copy link
Contributor Author

/retest

@adambkaplan
Copy link
Contributor Author

/retest

Failed to acquire lease

@smarterclayton
Copy link
Contributor

/test e2e-aws

@smarterclayton
Copy link
Contributor

Tested, seems to resolve the flake

/lgtm
/approve

@smarterclayton smarterclayton added the lgtm Indicates that a PR is ready to be merged. label Apr 5, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adambkaplan, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit ee8f767 into openshift:master Apr 5, 2020
@openshift-ci-robot
Copy link
Contributor

@adambkaplan: Some pull requests linked via external trackers have merged: . The following pull requests linked via external trackers have not merged:

In response to this:

Bug 1785023: Increase pull secrets controller QPS to 100

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sttts
Copy link
Contributor

sttts commented Apr 7, 2020

@adambkaplan we need a backport for 4.4. This is bleeding.

@sttts
Copy link
Contributor

sttts commented Apr 7, 2020

/cherry-pick release-4.4

@openshift-cherrypick-robot

@sttts: new pull request created: #89

In response to this:

/cherry-pick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@adambkaplan
Copy link
Contributor Author

/cherry-pick release-4.3

@openshift-cherrypick-robot

@adambkaplan: new pull request created: #91

In response to this:

/cherry-pick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants