Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remove apiserver loopback client QPS limit #80465

Merged
merged 1 commit into from Sep 26, 2019
Merged

remove apiserver loopback client QPS limit #80465

merged 1 commit into from Sep 26, 2019

Conversation

answer1991
Copy link
Contributor

@answer1991 answer1991 commented Jul 23, 2019 •
edited by liggitt

What type of PR is this?

/kind bug

What this PR does / why we need it:

The loopback QPS limit will influence API Server performance, we need to remove limit.

Which issue(s) this PR fixes:

related to #78766

Special notes for your reviewer:

The loopback QPS limit will influence API Server performance, we need to remove limit.

Does this PR introduce a user-facing change?:

Resolves bottleneck in internal API server communication that can cause increased goroutines and degrade API Server performance

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

None

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Jul 23, 2019
@k8s-ci-robot
Copy link
Contributor

Hi @answer1991. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 23, 2019
@k8s-ci-robot k8s-ci-robot added area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 23, 2019
@answer1991
Copy link
Contributor Author

@hormes @wojtek-t PTAL

wojtek-t
wojtek-t reviewed Jul 23, 2019
View reviewed changes
staging/src/k8s.io/apiserver/pkg/server/config_selfclient.go
Burst: 100,
Host: "https://" + net.JoinHostPort(host, port),
// Do not limit loopback client QPS.
QPS: -1,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@answer1991 Can you explain a bit more here regarding why we shouldn't limit loopback client QPS? Thanks!

Copy link
Contributor Author

@answer1991 answer1991 Aug 5, 2019 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure.

We found in our API Server lots of goruntine stuck at limiter.Accpet() and cost lots of time.

API Server will use this client configuration in inner controller, admission client and other client to visit API Server itself. Current configuration will limit the QPS of this client and influence the performance of API Server.

As API Server has already had --max-mutating-requests-inflight and --max-requests-inflight to prevent API Server overload from client requests, and API Server use this client in most scenes when client request is accepted. So I think we should remove this client QPS limit to improve API Server throughout capacity.

Any way, to set QPS -1 is tricky, but there is no any other ways to remove the client QPS before we refactor lots of codes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related issue: #78766

@roycaihw
Copy link
Member

/cc @aaron-prindle

@lavalamp
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 25, 2019
@liggitt
Copy link
Member

liggitt commented Sep 25, 2019

pull-kubernetes-node-e2e failure is expected, the job has a known problem with an unavailable image at the moment

@lavalamp
Copy link
Member

/retest

@lavalamp
Copy link
Member

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 25, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: answer1991, lavalamp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 25, 2019
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@liggitt
Copy link
Member

liggitt commented Sep 25, 2019

/retest

@liggitt
Copy link
Member

liggitt commented Sep 25, 2019

/kind bug
/priority important-soon

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Sep 25, 2019
@cheftako
Copy link
Member

/test pull-kubernetes-kubemark-e2e-gce-big

@liggitt
Copy link
Member

liggitt commented Sep 25, 2019

/retest

@k8s-ci-robot k8s-ci-robot merged commit 45f7f70 into kubernetes:master Sep 26, 2019
@k8s-ci-robot k8s-ci-robot added this to the v1.17 milestone Sep 26, 2019
This was referenced Sep 26, 2019
Merged
Merged
Merged
Merged
@lavalamp
Copy link
Member

@answer1991 Many thanks for sending this PR, apologies that it fell through the cracks until now.

@lavalamp lavalamp mentioned this pull request Sep 26, 2019
Closed
@answer1991
Copy link
Contributor Author

@lavalamp :-)

k8s-ci-robot added a commit that referenced this pull request Sep 27, 2019
@k8s-ci-robot
Merge pull request #83140 from liggitt/automated-cherry-pick-of-#8046…
a5301e3 
…5-upstream-release-1.13

Automated cherry pick of #80465: remove apiserver loopback client QPS limit
k8s-ci-robot added a commit that referenced this pull request Sep 27, 2019
@k8s-ci-robot
Merge pull request #83142 from liggitt/automated-cherry-pick-of-#8046…
db4dd7e 
…5-upstream-release-1.15

Automated cherry pick of #80465: remove apiserver loopback client QPS limit
k8s-ci-robot added a commit that referenced this pull request Sep 27, 2019
@k8s-ci-robot
Merge pull request #83143 from liggitt/automated-cherry-pick-of-#8046…
a3b5227 
…5-upstream-release-1.16

Automated cherry pick of #80465: remove apiserver loopback client QPS limit
@liggitt liggitt mentioned this pull request Sep 27, 2019
Merged
@lavalamp lavalamp mentioned this pull request Sep 27, 2019
Closed
k8s-ci-robot added a commit that referenced this pull request Sep 27, 2019
@k8s-ci-robot
Merge pull request #83141 from liggitt/automated-cherry-pick-of-#8046…
ac609db 
…5-upstream-release-1.14

Automated cherry pick of #80465 #83250: remove apiserver loopback client QPS limit
This was referenced Oct 8, 2019
Merged
Closed
@lavalamp lavalamp mentioned this pull request Oct 17, 2019
Closed
@mikedanese mikedanese mentioned this pull request Dec 4, 2019
Closed
This was referenced Oct 23, 2020
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wojtek-t wojtek-t wojtek-t left review comments

@lavalamp lavalamp Awaiting requested review from lavalamp

@sttts sttts Awaiting requested review from sttts

@aaron-prindle aaron-prindle Awaiting requested review from aaron-prindle

Assignees

@lavalamp lavalamp

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.17
Development

Successfully merging this pull request may close these issues.

None yet
9 participants
@answer1991 @k8s-ci-robot @roycaihw @lavalamp @liggitt @fejta-bot @cheftako @aaron-prindle @wojtek-t