DRA: ResourceSlice Status for Device Health Tracking

[johnbelamaric]

Enhancement Description

Devices like GPUs sometimes fail. At the recent Maintainers Summit in London, during the unconference we asked what Kubernetes is still missing for AI/ML workloads. Handling GPU failure was high on the list.

Currently, it is the responsibility of the DRA driver to exclude unhealthy devices from the ResourceSlice. However, this means that only the driver is well positioned to mitigate those failures. For example, often a failure requires only a reset of the GPU device to repair. This can be done without even rebooting the node. Different organizations are building different tooling for managing this mitigation. A single API to surface device issues would help those efforts, and may even enable integration with existing tooling such as the pluggable Node Problem Detector.

This issue proposes a ResourceSlice.Status API that drivers can optionally use to publish health information about their devices. This can be used by maintenance tooling to attempt mitigation, or alert administrators, or even reschedule larger multi-node jobs into new placement groups.

/sig node
/wg device-management
/cc @pohly @klueska @SergeyKanzhelev @asm582 @tardieu

• One-line enhancement description (can be used as a release note): Enable DRA drivers to store device health and other device status in the ResourceSlice
• Kubernetes Enhancement Proposal: TBD
• Discussion Link: https://docs.google.com/document/d/1Zz_xhPemY28EqpcKSLPl-S7tWnOHPU_IDLVmZOoGS5k/edit?tab=t.0#bookmark=id.j08fcqbp6h8d
• Primary contact (assignee): @johnbelamaric
• Responsible SIGs: Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.34
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[SergeyKanzhelev]

cc @ArangoGutierrez

[kannon92]

Is #4680 related to this?

For #4680, it seems we never did implement the DRA health status.

[johnbelamaric]

It is related. We need to reconcile if that one is enough, or if we want both. With that one, I could achieve a similar result to this enhancement if I grabbed all devices with an admin access request. That could work for tooling/automation but is not a good UX for user troubleshooting.

So, I can see a place for both of these. We just need to decide.

[ArangoGutierrez]

Given that #4680 is halfway through (Device plugin part merged), I think it makes sense to continue that work, and just see how we can integrate this proposal

[ArangoGutierrez]

@johnbelamaric Once we decide how this KEP would align with #4680 I would like to volunteer to take on this KEP implementation, if that's ok with you.

[johnbelamaric]

@johnbelamaric Once we decide how this KEP would align with #4680 I would like to volunteer to take on this KEP implementation, if that's ok with you.

Awesome, thanks.

[johnbelamaric]

@ArangoGutierrez are you interested in working on the design/KEP and working with @Jpsassine to figure out how this KEP and #4680 fit together? Or are you more focused on the implementation side?

[johnbelamaric]

@ArangoGutierrez are you interested in working on the design/KEP and working with @Jpsassine to figure out how this KEP and #4680 fit together? Or are you more focused on the implementation side?

This KEP currently has no owner, see https://docs.google.com/presentation/d/1COH8dG8qZ9jEMXzQVdco74XmroZQTMYcUgwAES8FYWU/edit?resourcekey=0-RCm3FFt8pxiK-y-_YHUkOg&slide=id.g353812bec9d_0_26#slide=id.g353812bec9d_0_26