DRA: Device Binding Conditions

[pohly]

Enhancement Description

• One-line enhancement description: Some network- or fabric-attached devices need to be attached to a node before a pod using them can be scheduled.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/5007-device-attach-before-pod-scheduled/README.md
• Discussion Link: DRA: structured parameters: network-attached resource kubernetes#124042 (comment)
• Primary contact (assignee): @pohly, @KobayashiD27
• Responsible SIGs:
  /sig scheduling
  /wg device-management
• Enhancement target (which target equals to which milestone):
  • Alpha release target: 1.33
  • Beta release target: TBD
  • Stable release target: TBD
• Alpha
  • KEP (k/enhancements) update PR(s): KEP-5007: DRA Device Binding Conditions #5012
  • Code (k/k) update PR(s): Implement DRA Device Binding Conditions (KEP-5007) kubernetes#130160
  • Docs (k/website) update PR(s): DRA Device Binding Conditions #5007 : placeholder PR website#49814

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[pohly]

/assign @KobayashiD27

As discussed in kubernetes/kubernetes#124042 (comment).

/sig scheduling
/wg device-management

[k8s-ci-robot]

@pohly: GitHub didn't allow me to assign the following users: KobayashiD27.

Note that only kubernetes members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @KobayashiD27

As discussed in kubernetes/kubernetes#124042 (comment).

/sig scheduling
/wg device-management

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[KobayashiD27]

Thank you for creating the issue. I will post a draft KEP as soon as possible.

[KobayashiD27]

@pohly

To facilitate the discussion on the KEP, we would like to share the design of the composable controller we are considering as a component utilizing the fabric-oriented scheduler function. By sharing this, we believe we can deepen the discussion on the optimal implementation of the scheduler function. Additionally, we would like to verify whether the controller design matches the DRA design.

Background

Our controller's philosophy is to efficiently utilize fabric devices. Therefore, we prefer to allocate devices directly connected to the node over attached fabric devices. (e.g., Node-local devices > Attached fabric devices > Pre-attached fabric devices)

Design Overview

This design aims to efficiently utilize fabric devices, prioritizing node-local devices to improve performance. The composable controller manages fabric devices that can be attached and detached. Therefore, it publishes a list of fabric devices as ResourceSlices.

The structure we are considering is as follows:

# composable controller publish this pool
kind: ResourceSlice
pool: composable-device
driver: gpu.nvidia.com
nodeSelector: fabric1
devices:
  - name: device1
  ...
  - name: device2
  ...

The vendor's DRA kubelet plugin will also publish the devices managed by the vendor as ResourceSlices.

# vendor DRA kubelet plugin publish this pool
kind: ResourceSlice
pool: Node1
driver: gpu.nvidia.com
nodeName: Node1
devices:
  - name: device3
  ...

Here, when the scheduler selects the fabric device device1, it waits for the attachment of the fabric device during PreBind. The composable controller performs the attachment operation by checking the flag of the ResourceClaim. After successful attachment, the composable controller changes the flag of the ResourceClaim.

We are considering the following two methods for handling ResourceSlices upon completion of the attachment. We would like to hear your opinions and feasibility on these two composable controller proposals.

Proposal 1: The composable controller publishes ResourceSlices with NodeName set within the pool

Multiple ResourceSlices are published with the same pool name. One indicates the devices included in the fabric, and the other indicates the devices attached to the node.

# composable controller publish this pool
kind: ResourceSlice
pool: composable-device
driver: gpu.nvidia.com
nodeSelector: fabric1
devices:
  - name: device2
  ...
---
kind: ResourceSlice
pool: composable-device
driver: gpu.nvidia.com
nodeName: Node1
devices:
  - name: device1
  ...

If the vendor's plugin responds to hotplug, device1 will appear in the ResourceSlice published by the vendor.

# vendor DRA kubelet plugin publish this pool
kind: ResourceSlice
pool: Node1
driver: gpu.nvidia.com
nodeName: Node1
devices:
  - name: device3
  ...
  - name: device1
  ...

This may cause device duplication issues between ResourceSlices. To prevent multiple ResourceSlices from publishing duplicate devices, we plan to define a deny list and standardize it with DRA.

Advantages

• No need to change the allocationResult by the scheduler or composable controller.
• Can distinguish attached fabric devices and maintain prioritization.

Disadvantages

• ResourceSlices created by the composable controller may not be understood by the vendor kubelet plugin. (NVIDIA drivers use internal information, so cooperation is needed)
• Attached and unattached fabric devices are mixed in one pool. (DRA: structured parameters: network-attached resource kubernetes#124042 (comment))
• A mechanism to prevent device duplication is needed (e.g., deny list).

Proposal 2: Attached devices are published by the vendor's plugin

In this case, devices are removed from the composable-device pool.

# composable controller publish this pool
kind: ResourceSlice
pool: composable-device
driver: gpu.nvidia.com
nodeSelector: fabric1
devices:
  - name: device2
  ...

If the vendor's plugin responds to hotplug, device1 will appear in the ResourceSlice published by the vendor.

# vendor DRA kubelet plugin publish this pool
kind: ResourceSlice
pool: Node1
driver: gpu.nvidia.com
nodeName: Node1
devices:
  - name: device3
  ...
  - name: device1
  ...

This breaks the linkage between ResourceClaim and ResourceSlice. Therefore, it is necessary to modify the AllocationResult of the ResourceClaim.

Advantages

• Simplifies device management.
• Centralizes management as the vendor's plugin directly publishes devices.
• No need for mechanisms to prevent device duplication (e.g., deny list).

Disadvantages

• Cannot distinguish attached fabric devices, making prioritization difficult.
• Requires modification of the linkage between ResourceClaim and ResourceSlice (expected to be done by the scheduler or DRA controller. Which is more appropriate?).
• Until the linkage is fixed, the device being used may be published as a ResourceSlice and reserved by other Pods.

We would appreciate your feedback and insights on these proposals to ensure the optimal implementation of the scheduler function and alignment with the DRA design.

[pohly]

Let's keep the discussion in this issue shorter. You now can put all of this, including the alternatives, into the KEP document.

[KobayashiD27]

@pohly
Could you please link this PR #5012 to "KEP update PR"?

[johnbelamaric]

/retitle DRA: Device Binding Conditions

[johnbelamaric]

Looks like this was never opted into the release. Given Aldo's approval, I am assuming it's OK to do so. Please correct me if not...

/label lead-opted-in
/milestone v1.33

[dipesh-rawat]

Hello @pohly, @KobayashiD27 👋, 1.33 Enhancements team here,

With PR #5012 has been merged, all the KEP requirements are in place and merged into k/enhancements.

Before the enhancement freeze, it would be appreciated if following nits could be addressed:

• Update issue description to add direct link to KEP README.md.

  • Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/5007-device-attach-before-pod-scheduled/README.md

• Update issue description to add the KEP update PR for alpha stage

  • Alpha
    • KEP (k/enhancements) update PR(s): KEP-5007: DRA Device Binding Conditions #5012

Aside from the minor nits mentioned above, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is now marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!
(cc: @fykaa)

/stage alpha
/label tracked/yes

[johnbelamaric]

Before the enhancement freeze, it would be appreciated if following nits could be addressed:

done! thank you

[Urvashi0109]

Hello @pohly @KobayashiD27 👋, v1.33 Docs Shadow here.

Does this enhancement work planned for v1.33 require any new docs or modification to existing docs?

If so, please follow the steps here to open a PR against dev-1.33 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 27th February 2025 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[KobayashiD27]

@Urvashi0109
cc @pohly
I opened a placeholder PR.
kubernetes/website#49814

There's nothing in it yet, but let me know if there are any corrections that need to be made.

[fykaa]

Hey again @KobayashiD27 👋, v1.33 Enhancements team here,

Just checking in as we approach Code Freeze at 02:00 UTC Friday 21st March 2025 / 19:00 PDT Thursday 20th March 2025.

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs need to be merged before code freeze (and we need to update the Issue description to include all the related PRs of this KEP):

• Implement DRA Device Binding Conditions (KEP-5007) kubernetes#130160

If you anticipate missing code freeze, you can file an exception request in advance.

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.

The status of this enhancement is marked as At risk for code freeze.

As always, we are here to help if any questions come up. Thanks!

[Udi-Hofesh]

Hey @pohly & @KobayashiD27 👋 -- this is Udi (@Udi-Hofesh) from the 1.33 Communications Team!

For the 1.33 release, we are currently in the process of collecting and curating a list of potential feature blogs, and we'd love for you to consider writing one for your enhancement!

As you may be aware, feature blogs are a great way to communicate to users about features that fall into (but are not limited to) the following categories:

• This introduces some breaking change(s)
• This has significant impacts and/or implications to users
• ...Or this is a long-awaited feature, which would go a long way to cover the journey more in detail 🎉

To opt in to write a feature blog, could you please let us know and open a "Feature Blog placeholder PR" (which can be only a skeleton at first) against the website repository by Wednesday, 5th March, 2025? For more information about writing a blog, please find the blog contribution guidelines 📚

Tip

Some timelines to keep in mind:

• 02:00 UTC Wednesday, 5th March, 2025: Feature blog PR freeze
• Monday, 7th April, 2025: Feature blogs ready for review
• You can find more in the release document

Note

In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[Udi-Hofesh]

Hi @pohly & @KobayashiD27 👋, It's @Udi-Hofesh from 1.33 Comms Team here again!

This is a gentle reminder for the feature blog deadline mentioned above, which is 02:00 UTC Wednesday, 5th March, 2025. This is in just several hours! To opt in, please let us know and open a Feature Blog placeholder PR against k/website by the deadline. If you have any questions, please feel free to reach out to us!

Tip

Some timeline to keep in mind:

• 02:00 UTC Wednesday, 5th March, 2025: Feature blog PR freeze
• Monday, 7th April, 2025: Feature blogs ready for review
• You can find more in the release document

Note

In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.