Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better mechanism to decide Retry-After header #117734

Closed
linxiulei opened this issue May 2, 2023 · 4 comments
Closed

Better mechanism to decide Retry-After header #117734

linxiulei opened this issue May 2, 2023 · 4 comments
Assignees
@wojtek-t
Labels
kind/feature Categorizes issue or PR as related to a new feature. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@linxiulei
Copy link
Contributor

What would you like to be added?

Make the value for Retry-After header of APF-rejected responses dynamic or user-adjustable instead of being constant 1

Why is this needed?

When kube-apiserver APF rejects a request, it sets the HTTP response header with Retry-After of value 1 (ie. retry in 1 second). There was a TODO in https://github.com/kubernetes/apiserver/blob/master/pkg/server/filters/maxinflight.go#L38 to improve this since many years ago.

Retry-After being 1 seems less reasonable to many cases when the kube-apiserver is heavily overloaded.

As suggested in the TODO:

  • one naive way to improve this is simply making it user-adjustable .
  • alternatively, it can be dynamic by recording count of 429s in past X mins and decide the Retry-After value up to what users have specified. The implementation needs more thoughts if we think the naive one is not good enough.
@linxiulei linxiulei added the kind/feature Categorizes issue or PR as related to a new feature. label May 2, 2023
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 2, 2023
@aojea
Copy link
Member

aojea commented May 2, 2023

I think that this has been discussed more times in another context but I can't find the history

/sig api-machinery
/cc @MikeSpreitzer @wojtek-t

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 2, 2023
@wojtek-t
Copy link
Member

wojtek-t commented May 4, 2023

I have a WIP PR where I'm working on it as we speak:

#117547

:)

@wojtek-t wojtek-t self-assigned this May 4, 2023
@jpbetz
Copy link
Contributor

jpbetz commented May 4, 2023

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 4, 2023
@MikeSpreitzer MikeSpreitzer mentioned this issue May 5, 2023
Open
@linxiulei
Copy link
Contributor Author

It should be closed by #117547

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wojtek-t wojtek-t

Labels
kind/feature Categorizes issue or PR as related to a new feature. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jpbetz @linxiulei @aojea @wojtek-t @k8s-ci-robot