Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate API Throttling in Node Controller #14

Open
andrewsykim opened this issue Mar 6, 2019 · 13 comments
Open

Investigate API Throttling in Node Controller #14

andrewsykim opened this issue Mar 6, 2019 · 13 comments
Assignees
@cheftako @aoxn @andrewsykim
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. P3 Priority 3
Milestone
Next

Comments

@andrewsykim
Copy link
Member

ref: kubernetes/kubernetes#75016

For large clusters, we're seeing API throttling from providers becoming more common. Taking node-controller as an example, it will call a "get node" api request per node on every sync loop. For a 1000 node cluster that's could be 1000 get requests per minute which can result in users running out of API quotas.
@andrewsykim andrewsykim added this to the Next milestone Mar 6, 2019
@andrewsykim andrewsykim assigned aoxn and unassigned aoxn Mar 6, 2019
@andrewsykim
Copy link
Member Author

/assign @aoxn @andrewsykim

@andrewsykim andrewsykim modified the milestones: Next, v1.15 Mar 6, 2019
@hogepodge
Copy link

SIG OpenStack has seen similar issues, but we're addressing in our own implementations. We would welcome a generic solution if one would be appropriate, but we're fine with adding cloud-specific tuning variables on our provider.

@aoxn
Copy link

aoxn commented Mar 21, 2019

@hogepodge Cool work. Could you share your solutions with us?

@andrewsykim andrewsykim added the P3 Priority 3 label Mar 28, 2019
@feiskyer feiskyer mentioned this issue May 5, 2019
Closed
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 26, 2019
@feiskyer
Copy link
Member

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 27, 2019
@andrewsykim andrewsykim added P1 Priority 1 and removed P3 Priority 3 labels Jul 10, 2019
@andrewsykim andrewsykim modified the milestones: Next, v1.16 Jul 10, 2019
@andrewsykim
Copy link
Member Author

Investigate severity of API throttling for v1.16

@andrewsykim
Copy link
Member Author

If folks have user stories on how bad the API throttling is for clusters that would be really helpful in determining the priority for this issue.

@feiskyer
Copy link
Member

This would be the issue for Azure. Refer #30.

@andrewsykim andrewsykim added P3 Priority 3 and removed P1 Priority 1 labels Oct 2, 2019
@andrewsykim andrewsykim modified the milestones: v1.16, Next Oct 2, 2019
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 31, 2019
@cheftako
Copy link
Member

cheftako commented Jan 2, 2020

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 2, 2020
@cheftako
Copy link
Member

cheftako commented Jan 2, 2020

/lifecycle frozen

@k8s-ci-robot k8s-ci-robot added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label Jan 2, 2020
@andrewsykim
Copy link
Member Author

/help

@k8s-ci-robot
Copy link
Contributor

@andrewsykim:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Apr 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cheftako cheftako

@aoxn aoxn

@andrewsykim andrewsykim

Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. P3 Priority 3
Projects
None yet
Milestone
Next
Development

No branches or pull requests
7 participants
@hogepodge @feiskyer @cheftako @aoxn @andrewsykim @k8s-ci-robot @fejta-bot