DRA: Handle extended resource requests via DRA Driver

[klueska]

Enhancement Description

• One-line enhancement description (can be used as a release note):
  Allow DRA drivers to honor requests made via the extended resource API (e.g. nvidia.com/gpu: 2) rather than requiring a standard device plugin be used.

• Kubernetes Enhancement Proposal:

  • https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5004-dra-extended-resource

• Discussion Link:

  • https://youtu.be/fKhX_lHK8Z0?si=gq5kIFHP3ve2TXyE&t=1822

• Primary contact (assignee):
  @klueska, @pohly, @johnbelamaric

• Responsible SIGs:
  /sig node
  /wg device-management

• Enhancement target (which target equals to which milestone):

  • Alpha release target: 1.34
  • Beta release target: 1.35
  • Stable release target: 1.36

• Alpha (1.34)

  • KEP (k/enhancements) update PR(s):
    • Add KEP for DRA: Extended Resource #5136
  • Code (k/k) update PR(s):
    • kubelet and scheduler for extended resource backed by DRA kubernetes#130653
  • Docs (k/website) update PR(s):
    • [KEP-5004] DRA Extended resource documentation website#51710
  • Known issues where fixes won't be backported:
    • resource claims created by scheduler for extended resources are not de-allocated when pod is completed kubernetes#134311

• Beta (1.35)

  • KEP (k/enhancements) update PR(s):
    • KEP-5004: DRA Extended Resource: graduate to beta #5537
  • Code (k/k) update PR(s):
    • DRA Extended Resourece promote to Beta in 1.35 kubernetes#135048
  • Docs (k/website) update(s):
    • TBD

<--

• Stable
  • KEP (k/enhancements) update PR(s):
    • TBD
  • Code (k/k) update PR(s):
    • TBD
  • Docs (k/website) update(s):
    • TBD
      -->

[johnbelamaric]

+1 yes please!

[johnbelamaric]

We need to sort out the requirements. A few initial questions:

1. For newly created pods, I think it's clear we want this to be transparent. Existing manifests that use the extended resource API should continue to work as before, without modification.
2. Can we handle this invisibly in the driver layer, or do we need to have DRA invoked at the control plane level and select the specific devices? If we don't, we will likely have a race condition - unless the scheduler can do some magical accounting (which seems possible).
3. How do we handle upgrades? If we have a node running device plugin, and we switch to the DRA driver (or we upgrade to a driver that supports both), do you have to delete the pods? Do they automatically adopt the devices? If so, how do we write those back to the allocation logic (since no DRA claim exists).
4. What happens if there are pods in a deployment, and some land on nodes with device plugin and some with DRA drivers?
5. We talked about letting specific device classes be advertised as specific extended resources. This could mean the existing resource names get mapped to specific device classes by the admin. It could also mean we have a convention like deviceclass.k8s.io/foo: 4 for extended resource names. How do these choices interplay with the questions above?

[lengrongfu]

Can each dra-driver implement a webhook to create a ResourceClaimTemplate after creating a pod and modify the application method of resources in the pod?

[klueska]

@lengrongfu that is what this KEP would be designed to avoid. There would be integrated scheduler support for all drivers, rather than requiring each DRA driver to provide a webhook.

[alculquicondor]

Open questions (from SIG Scheduling meeting):

• How to handle resource quotas
• Scheduling throughput (API requests and overall processing).

[ffromani]

/cc

[yliaog]

/cc

[johnbelamaric]

/sig scheduling