Distinguish between NUMA memory onlined as CPU memory vs. GPU memory

[klueska]

Enhancement Description

• One-line enhancement description (can be used as a release note):
  Build a comprehensive solution to ensure Kubernetes can distinguish between NUMA nodes brought online with CPU memory vs. GPU memory (or more generally any device memory). This will likely take the form of changes to cadvisor at the lowest layers.

• Kubernetes Enhancement Proposal:

  • TBD

• Discussion Link:

  • Support TopologyManager in Large NUMA System (NVIDIA GB200) kubernetes#135581 (comment)
  • https://developer.nvidia.com/blog/understanding-memory-management-on-hardware-coherent-platforms/#how_numa_impacts_kubernetes

• Primary contact (assignee):
  @klueska, @PiotrProkop, @fanzhangio

• Responsible SIGs:
  /sig node

• Enhancement target (which target equals to which milestone):

  • Alpha release target: TBD
  • Beta release target: TBD
  • Stable release target: TBD

[ffromani]

/cc

[zvonkok]

/cc

[fanzhangio]

Adding some data points from NVIDIA’s GB200 (Grace) bring-up that makes this KEP worth tracking.

Observed problem:

On Grace-class systems the host firmware/ACPI SRAT exposes a large NUMA node set (e.g., 34 nodes on GB200 trays, including CPU-less / memory-less “device” nodes). kubelet/cAdvisor surfaces this topology to TopologyManager, and TM’s hint evaluation can scale combinatorially with NUMA node count, leading to kubelet appearing “stuck” for TopologyManager(TM)-dependent workloads.

Impact:

This blocks TM-dependent workloads (e.g., GPU passthrough / high-perf placement scenarios, including KubeVirt-style containerized virtualization with toplogy-awareness allocation), even in cases where the NVIDIA driver/CDMM is not present on the host.

Short-term mitigation (validated):

• Excluding non-CPU / zero-memory nodes from the TM search space unblocks kubelet admission on Grace-system (GB200).
• References:
  This PR Support TopologyManager in Large NUMA System (NVIDIA GB200) kubernetes#135581 is a verified WAR fix on GB200.
  and follow-up fix in fix: exclude fully reserved NUMA nodes consistently from topology hints, not just memory manager decisions kubernetes#137280 (It uses MemoryManager to consistently exclude NUMA nodes but has limitation)

Why this KEP matters long-term:

We need a consistent contract for “eligible” NUMA nodes for topology decisions on heterogeneous-memory platforms (CPU memory vs device memory vs memory-less nodes), so TM/hint providers don’t treat every enumerated NUMA ID as a scheduling dimension.

Asks:

• Can we identify SIG Node / Scheduling reviewers to confirm whether this is a TM eligibility rule vs cadvisor reporting rule?
• Is there agreement that ‘NUMA nodes with zero CPU + fully reserved/zero memory’ should be excluded from TM consideration behind a config/option?

[ffromani]

* References:
  This PR [kubelet TopologyManager Stalls on Large NUMA Systems (e.g. Nvidia GB200 Compute Tray with 34 NUMA nodes) kubernetes#135541](https://github.com/kubernetes/kubernetes/issues/135541) is a verified WAR fix on GB200.
  and follow-up fix in [fix: exclude fully reserved NUMA nodes consistently from topology hints, not just memory manager decisions kubernetes#137280](https://github.com/kubernetes/kubernetes/pull/137280) (It uses MemoryManager to consistently exclude NUMA nodes but has limitation)

Will review shortly. I see no obvious reasons not to merge these fixes.

Asks:

* Can we identify SIG Node / Scheduling reviewers to confirm whether this is a TM eligibility rule vs cadvisor reporting rule?

I'm willing to have a look and coment

* Is there agreement that ‘NUMA nodes with zero CPU + fully reserved/zero memory’ should be excluded from TM consideration behind a config/option?

I'm supportive of TM/kubelet changes in this area, but this would require SIG-wide discussion and more acks.