Priority and Fairness for API Server Requests

[yue9944882]

Enhancement Description

• One-line enhancement description (can be used as a release note): Priority and Fairness for API Server Requests

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness

• Primary contact (assignee): @yue9944882 @MikeSpreitzer

• Responsible SIGs: sig-api-machinery

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y) 1.18
  • Beta release target (x.y) 1.20
  • Stable release target (x.y) 1.28

• Stable:

  • KEP: KEP-1040: Add borrowing by exempt to APF GA criteria #3991
  • Code:
  • Docs:

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

[kacole2]

@yue9944882 the 1.15 enhancement freeze was on 4/30. The KEP is still in a provisional state and still requires graduation criteria. This will not make it in 1.15 without filing an exception and fixing what I had just mentioned.

[kacole2]

/milestone v1.15
/sig api-machinery
/stage alpha

[makoscafee]

Hey, @yue9944882 👋 I'm the v1.15 docs Lead.
Does this enhancement require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[kacole2]

Hi @yue9944882 @MikeSpreitzer . Code Freeze is Thursday, May 30th 2019 @ EOD PST. All enhancements going into the release must be code-complete, including tests, and have docs PRs open.

Please list all current k/k PRs so they can be tracked going into freeze. If the PRs aren't merged by freeze, this feature will slip for the 1.15 release cycle. Only release-blocking issues and PRs will be allowed in the milestone.

If you know this will slip, please reply back and let us know. Thanks!

[kacole2]

Hi @yue9944882 @MikeSpreitzer , today is code freeze for the 1.15 release cycle. I do not see a reply for any k/k PRs to track for this merge. It's now being marked as At Risk in the 1.15 Enhancement Tracking Sheet. If there is no response, or you respond with PRs to track and they are not merged by EOD PST, this will be dropped from the 1.15 Milestone. After this point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[kacole2]

/milestone clear

[evillgenius75]

Hi @MikeSpreitzer and @yue9944882 , I'm a 1.16 Enhancement Shadow. Is this feature going to be graduating to alpha stage in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

I see you have PR kubernetes/kubernetes#78966 referenced above, please include any other relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[yue9944882]

hey, yes please track the feature in the 1.16 milestone. matter of fact this feature is literally active from 1.15 on, as you see in the pull above there's ~300 discussions over the api model since april this year. there're also other poc repos under development:

• https://github.com/aaron-prindle/fq-inflight-rate-limit
• https://github.com/aaron-prindle/fq-apiserver
• https://github.com/mars1024/shuffle-sharding

we will get into 1.16 this time 👏🏻

[kacole2]

/milestone v1.16

[sethmccombs]

Hey @yue9944882 @MikeSpreitzer,

I'm one of the v1.16 docs shadows.
Does this enhancement (or the work planned for v1.16) require any new docs (or modifications to existing docs)? If not, can you please update the 1.16 Enhancement Tracker Sheet (or let me know and I’ll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday, August 23rd, it can just be a placeholder PR at this time. Let me know if you have any questions!

[MikeSpreitzer]

@evillgenius75 : Yes, this feature needs docs. I do not see how to update the tracking spreadsheet myself; is that something I leave to you?

@sethmccombs : please see my plea for help on the sig-docs Slack channel of the Kubernetes workspace

[Atharva-Shinde]

Hello @yue9944882 @MikeSpreitzer 👋, Enhancements team here.

Just checking in as we approach Enhancements freeze on Thursday, 16th June 2023.

This enhancement is targeting for stage stable for 1.28 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.28
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would need to update the following:

• Update the KEP to be in compliance with the latest KEP readme template
• Update kep.yaml with up-to-date graduation criteria and latest milestone information.
• Add the acknowledgement for the Test Plan.
• Add related information/links to all tests under the test-plan section of this KEP.

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[Atharva-Shinde]

Hey @yue9944882 @MikeSpreitzer,
We would just need to update the latest-milestone field under kep.yaml to 1.28, apart from that everything else seems to be addressed. Please make the change within 24 hours else you would have to file an exception request.
Thanks :)

[Atharva-Shinde]

Hello @MikeSpreitzer @yue9944882, 1.28 Enhancements Lead here.
This enhancement unfortunately do not satisfy the requirements for v1.28 enhancements freeze.
Feel free to file an exception request to add this back to the release tracking process. Thanks!

[Atharva-Shinde]

/milestone clear

[taniaduggal]

Hello @yue9944882 1.28 Docs Shadow here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[Rishit-dagli]

@taniaduggal This is removed from 1.28 release so I have marked this as "removed form release" and there is no need for 1.28 docs PR

[MikeSpreitzer]

@Rishit-dagli and others. Eek. I got overwhelmed with other stuff 2 weeks ago. Not sure I understand the current status. Earlier we were thinking of going to stable in 1.28 but have since realized this feature will not be ready. We are still doing work on it. I see that the KEP needs to be updated to the latest template. We can work on that. Are you saying that this feature will be removed in 1.28 or stay in its current beta status?

[wojtek-t]

/milestone v1.29

[rayandas]

Hello @yue9944882 @MikeSpreitzer 👋, v1.29 Enhancements team here.

Just checking in as we approach enhancements freeze on 01:00 UTC, Friday, 6th October, 2023.

This enhancement is targeting for stage stable for v1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.29. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[katcosgrove]

Hey there @yue9944882 👋, v1.29 Docs Lead here.
Does this enhancement work planned for v1.29 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023.
Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[katcosgrove]

Hi @yue9944882 and @MikeSpreitzer! The deadline to open a placeholder PR against k/website for required documentation is Thursday, 19 October. Could you please update me on the status of docs for this enhancement? Thank you!

[MikeSpreitzer]

Metrics missing from doc:

• current_inqueue_seats
• current_executing_seats
• epoch_advance_total

The debug endpoint doc looks current.

Will open a PR.

[MikeSpreitzer]

See kubernetes/website#43549

[tengqm]

Metrics missing from doc:

* current_inqueue_seats

* current_executing_seats

* epoch_advance_total

The debug endpoint doc looks current.

Will open a PR.

When working on an observability tool, I found that the APF is exposing a lot of metrics, most of which are more like developer oriented ones. The number of metrics almost amounts to that for the API server itself. From a user's perspective, I don't want to be bothered by the epoch_advance_toal or *_r_* or *_s_* metrics. I don't even know how to interpret them even after spending time reading the documents. I only care whether APF is working properly. There are my gut feeling when checking these metrics. Sorry for being a little skeptic on the metrics design.

[MikeSpreitzer]

@tengqm: I agree with your observation. We can certainly consider changes to the design.

One concern that I bring is that some of these metrics are like logs in this sense: while users would not normally look at them and interpret them, having them available to someone trying to debug something that happened in the past is valuable.

[logicalhan]

@tengqm: I agree with your observation. We can certainly consider changes to the design.

One concern that I bring is that some of these metrics are like logs in this sense: while users would not normally look at them and interpret them, having them available to someone trying to debug something that happened in the past is valuable.

Do we need all the P&F metrics or can we truncate some of them? Alternatively, we can gate P&F metrics behind a flag, or maybe even a dedicated endpoint.

[MikeSpreitzer]

A special endpoint is likely to not be scraped in many environments.
I wonder about control through -v. This is not strictly "logging", but it is serving the same underlying concerns and would be a knob that users generally want to turn in tandem.

[rayandas]

Hey again @yue9944882 @MikeSpreitzer 👋, 1.29 Enhancements team here,

Just checking in as we approach code freeze at 01:00 UTC Wednesday 1st November 2023: .

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).

• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

Please update the Issue description to include all the related PRs of this KEP under the stable section in the Github Issue description. The status for this KEP is currently at risk for code freeze.

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.
As always, we are here to help if any questions come up. Thanks!

[a-mccarthy]

Hi @yue9944882 @MikeSpreitzer, 👋 from the v1.29 Release Team-Communications! We would like to check if you have any plans to publish a blog for this KEP regarding new features, removals, and deprecations for this release.

If so, you need to open a PR placeholder in the website repository.
The deadline will be on Tuesday 14th November 2023 (after the Docs deadline PR ready for review)

Here is the 1.29 calendar

[npolshakova]

Hey @yue9944882 @MikeSpreitzer, Enhancements lead here. Are both of these PRs tracking the code changes for this KEP?

• [WIP] promote APIPriorityAndFairness to GA kubernetes#118521
• apiserver: set APF featuregate to stable kubernetes#121638

[tkashem]

@npolshakova yes, all the code PRs have merged, thanks!

• apiserver: set APF featuregate to stable kubernetes#121638
• Promote APF API to v1 kubernetes#121089
• apiserver: set APF featuregate to stable kubernetes#121638

and the conformance test pr is being tracked here: kubernetes/kubernetes#121718

[tkashem]

draft PR for doc update: kubernetes/website#43946