dynamic resource allocation

[pohly]

Enhancement Description

• One-line enhancement description: dynamic resource allocation

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3063-dynamic-resource-allocation

• Discussion Link: CNCF TAG Runtime Container Device Interface (COD) Working Group meeting(s)

• Primary contact (assignee): @pohly

• Responsible SIGs: SIG Node

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.27
  • Beta release target (x.y): 1.29
  • Stable release target (x.y): 1.31

• Alpha (1.26)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063: dynamic resource allocation #3064
    • KEP-3063: dynamic resource allocation updates for 1.26 #3502
    • KEP-3063 Update kep.yaml #3868
  • Code (k/k) update PR(s): dynamic resource allocation kubernetes#111023
  • Docs (k/website) update PR(s):
    • blog: add dynamic resource allocation feature blog post website#37434
    • dynamic resource allocation concepts website#37766

• Alpha (1.27)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063 Update kep.yaml #3868
    • KEP 3063: Update to reflect changes merged into v1.27 #3802
  • Code (k/k) update PR(s):
    • api: resource.k8s.io v1alpha1 -> v1alpha2 kubernetes#116299
    • Update resource.AllocationResult with a slice of ResourceHandlers kubernetes#116332
    • dra: PodScheduling -> PodSchedulingContext kubernetes#116556
    • Update kubeletplugin API for DRA to v1alpha2 kubernetes#116558
    • optional: api: extend validation of dynamic resource allocation fields in PodSpec kubernetes#116576
  • Docs (k/website) update PR(s):
    • dynamic resource allocation: update for 1.27 website#40086

• Alpha (1.28)

  • KEP (k/enhancements) update PR(s):
    • KEP-3063: dra: pre-scheduled pods #4063
    • KEP-3063: dra: continue as alpha in 1.28 #4096
  • Code (k/k) update PR(s):
    • node authorizer: limit kubelet access to ResourceClaim objects kubernetes#116254
    • kubelet: support batched prepare/unprepare in v1alpha3 DRA plugin API kubernetes#119012
    • dra: pre-scheduled pods kubernetes#118209
    • DRA: generated resource claim names kubernetes#117351
    • dra: scheduler queueing hints kubernetes#119078
    • DRA controller: batch resource claims for Allocate kubernetes#118862
    • DRA: improve handling of completed pods kubernetes#118817
    • DRA: for delayed allocation, deallocate when no longer used kubernetes#118936
    • dynamic resource allocation: Improve code coverage of state checkpoint kubernetes#119058
    • DRA: don't query claims from API server kubernetes#119307
    • DRA: report NodePrepareResource errors kubernetes#118578
    • add unit test for dra/manager.go kubernetes#118711
    • Increase timeout for DRA kubelet plugin client kubernetes#119414
    • feature(scheduler): implement ClusterEventWithHint to filter out useless events kubernetes#118551
    • feature(scheduling_queue): track events per Pods kubernetes#118438
    • dynamic resource allocation: reuse gRPC connection kubernetes#118619 (v1.29)
  • Docs (k/website) update PR(s): dra: update for Kubernetes 1.28 website#41856

• Alpha (1.29)

  • KEP (k/enhancements) update PR(s):
    • DRA: handle non graceful node shutdowns #4260
    • KEP-3063: dynamic resource allocation: update kubelet plugin RPC versions supported. #4164
  • Code (k/k) update PR(s):
    • DRA: notify ResourceClaim or ResourceClass of params error kubernetes#121065
    • dra scheduler: fall back to SSA for PodSchedulingContext updates kubernetes#120534
    • dynamic resource allocation: reuse gRPC connection kubernetes#118619
    • dra e2e: fix "reallocation works" flake kubernetes#119195
    • DRA: Pass CDI devices with a new CRI field kubernetes#115891
    • DRA: call plugins for claims even if exist in cache kubernetes#120550
  • Docs (k/website) update PR(s):
    • Add Resource Drivers section to extending Kubernetes documentation website#43742
    • Document impact of DRA on scheduling website#43907

[pohly]

/assign @pohly
/sig node

[ahg-g]

do we have a discussion issue on this enhancement?

[pohly]

@ahg-g: with discussion issue you mean a separate issue in some repo (where?) in which arbitrary comments are welcome?

No, not at the moment. I've also not seen that done elsewhere before. IMHO at this point the open KEP PR is a good place to collect feedback and questions. I also intend to come to the next SIG-Scheduling meeting.

[ahg-g]

@ahg-g: with discussion issue you mean a separate issue in some repo (where?) in which arbitrary comments are welcome?

Yeah, this is what I was looking for, the issue would be under k/k repo.

No, not at the moment. I've also not seen that done elsewhere before.

That is actually the common practice, one starts a feature request issue where the community discusses initial ideas and the merits of the request (look for issues with label kind/feature). That is what I would expect in the discussion link.

IMHO at this point the open KEP PR is a good place to collect feedback and questions. I also intend to come to the next SIG-Scheduling meeting.

But the community have no idea what this is about yet, so better to have an issue discusses "What would you like to be added?" and "Why is this needed" beforehand. Also, meetings are attended by fairly small groups of contributors, having an issue tracking the discussion is important IMO.

[pohly]

In my work in SIG-Storage I've not seen much use of such a discussion issue. Instead I had the impression that the usage of "kind/feature" is discouraged nowadays.

https://github.com/kubernetes/kubernetes/issues/new?assignees=&labels=kind%2Ffeature&template=enhancement.yaml explicitly says

Feature requests are unlikely to make progress as issues. Please consider engaging with SIGs on slack and mailing lists, instead. A proposal that works through the design along with the implications of the change can be opened as a KEP.

This proposal was discussed with various people beforehand, now we are in the formal KEP phase. But I agree, it is hard to provide a good link to those prior discussions.

[ahg-g]

We use that in sig-scheduling, and it does serve as a very good place for initial rounds of discussions, discussions on slack and meetings are hard to reference as you pointed out.

I still have no idea what this is proposing, and I may not attend the next sig meeting for example...

[gracenng]

Hi @ ! 1.24 Enhancements team here.
Checking in as we approach enhancements freeze in less than a week on 18:00pm PT on Thursday Feb 3rd
Here’s where this enhancement currently stands:

• Updated KEP file using the latest template has been merged into the k/enhancements repo. KEP-3063: dynamic resource allocation #3064
• KEP status is marked as implementable for this release with latest-milestone: 1.24
• KEP has a test plan section filled out.
• KEP has up to date graduation criteria.
• KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is track as at risk.
Thanks!

[gracenng]

The Enhancements Freeze is now in effect and this enhancement is removed from the release.
Please feel free to file an exception.

/milestone clear

[gracenng]

Hi Patrick, `tracked/yes` will be applied when the KEP is merged and all requirements for Enhancement Freeze are met. We are definitely keeping an eye on this! Feel free to ping me once it's merged

…

On Tue, Mar 1, 2022 at 11:17 AM Patrick Ohly ***@***.***> wrote: @gracenng <https://github.com/gracenng> : an exception was requested and granted for this enhancement to move to GA in 1.24: https://groups.google.com/g/kubernetes-sig-release/c/sUpd2H1wxnk/m/lL_I6GT-BwAJ Can you label it again as "tracked/yes"? — Reply to this email directly, view it on GitHub <#3063 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCRLO46IYCBRTXXZE6EHGTU5ZUM3ANCNFSM5JCIMTQA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kerthcet]

/remove-lifecycle stale

[marosset]

Hello @pohly 👋, 1.25 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PST on Thursday June 16, 2022.

For note, This enhancement is targeting for stage alpha for 1.25 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has a updated detailed test plan section filled out
• KEP has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

It looks like #3064 will address everything in this list.

For note, the status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[marosset]

Hello @pohly 👋, just a quick check-in again, as we approach the 1.25 enhancements freeze.

Please plan to get #3064 reviewed and merged before enhancements freeze on Thursday, June 23, 2022 at 18:00 PM PT.

For note, the current status of the enhancement is atat-risk. Thank you!

[alculquicondor]

@pohly can you link the PRs about scheduling hints?

[klueska]

@Atharva-Shinde

These are all of the PRs related to DRA that landed this cycle:

• node authorizer: limit kubelet access to ResourceClaim objects kubernetes#116254
• kubelet: support batched prepare/unprepare in v1alpha3 DRA plugin API kubernetes#119012
• dra: pre-scheduled pods kubernetes#118209
• DRA: generated resource claim names kubernetes#117351
• dra: scheduler queueing hints kubernetes#119078
• DRA controller: batch resource claims for Allocate kubernetes#118862
• DRA: improve handling of completed pods kubernetes#118817
• DRA: for delayed allocation, deallocate when no longer used kubernetes#118936
• dynamic resource allocation: Improve code coverage of state checkpoint kubernetes#119058
• DRA: don't query claims from API server kubernetes#119307
• DRA: report NodePrepareResource errors kubernetes#118578
• add unit test for dra/manager.go kubernetes#118711
• Increase timeout for DRA kubelet plugin client kubernetes#119414

This one was planned but didn't make it:

• dynamic resource allocation: reuse gRPC connection kubernetes#118619

[Atharva-Shinde]

Thanks @klueska for linking all the code PRs associated with this KEP, as all these PRs associated with the v1.28 milestone were merged/ were in merge-ready state by the code-freeze, I am adding this KEP back to the v1.28 milestone and marking it as tracked for the code freeze :)
/milestone v1.28

[alculquicondor]

Please also add kubernetes/kubernetes#118551 and kubernetes/kubernetes#118438

[Rishit-dagli]

Hello @pohly wave: please take a look at Documenting for a release - PR Ready for Review to get your docs PR ready for review before Tuesday 25th July 2023. Thank you!

Ref: kubernetes/website#41856

[pohly]

@klueska: thanks for collecting the PR list while I was on vacation.

Out of curiosity, how did that get into the issue description? Does the release team have permission to edit issues?

[npolshakova]

Hello 👋, 1.29 Enhancements Lead here.

If you wish to progress this enhancement in v1.29, please have the SIG lead opt-in your enhancement by adding the lead-opt-in and milestone v1.29 labels before the Production Readiness Review Freeze.

/remove-label lead-opted-in

[pacoxu]

This one was planned but didn't make it:

• dynamic resource allocation: reuse gRPC connection kubernetes#118619

This PR is finally merged in v1.29. I commented on the issue description directly. Do we still target beta for v1.29?

• BTW, KEP-3063: dra: pre-scheduled pods #4063 is still not merged as well.

Until now, below are merged PRs in the v1.29 release cycle:

• dynamic resource allocation: optimize class.SuitableNodes usage kubernetes#114685
• dynamic resource allocation: reuse gRPC connection kubernetes#118619
• dra: scheduler_perf test case with multiple claims per pod kubernetes#119963
• dra test: enhance performance of test driver controller kubernetes#119819

Some ongoing PR:

• DRA: helper logging improvements kubernetes#120340
• scheduler_perf: use different log names for different DRA drivers kubernetes#120339
• dra helper: skip allocated claims during UnsuitableNodes calculation kubernetes#120338
• dra: resourceclass missing kubernetes#120213
• ...

[pohly]

We still target beta for 1.29. I'm currently doing a full pass over the KEP to ensure that everything is covered and documented properly.

[alculquicondor]

Was there any progress in the autoscaling story?

[pohly]

I discussed it at KubeCon EU this year with Maciek. We tossed around a few ideas on how autoscaler could get extended by vendors to cover their business logic, but in the end we'll need a plugin mechanism for it, similar to what is in the KEP. I still need to come up with specific code for it. That'll be the biggest work item for 1.29.

I got sidetracked by the scheduler issues, otherwise I would have started already on it 😰

Here's the PR with the beta KEP update: #4181

[SergeyKanzhelev]

/milestone v1.29
/stage beta
/label lead-opted-in

[npolshakova]

Hello @pohly 👋, 1.29 Enhancements team here!

Just checking in as we approach enhancements freeze on 01:00 UTC, Friday, 6th October, 2023.

This enhancement is targeting for stage beta for 1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.29. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

It looks like https://github.com/kubernetes/enhancements/pull/4181/files will address most of these issues! Make sure to also update the stage in the kep.yaml

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[npolshakova]

Hi @pohly, just checking in once more as we approach the 1.29 enhancement freeze deadline this week on 01:00 UTC, Friday, 6th October, 2023. The status of this enhancement is marked as at risk for enhancement freeze.

It looks like when #4181 merges in it will address most of the requirements. Let me know if I missed anything. Thanks!

[npolshakova]

Hello @pohly 👋, 1.29 Enhancements Lead here.
Unfortunately, this enhancement did not meet requirements for v1.29 enhancements freeze.
Feel free to file an exception to add this back to the release tracking process.

It looks like there is still ongoing discussion for this KEP. Let me know if you have any questions!

Thanks!

/milestone clear

[sftim]

Tracking something to highlight it for graduation reviews: I had a concern that we might not be planning to document the resource driver API well enough for people to learn about and use it.