NUMA-aware allocation: opt-in design and failure modes

[thameem-abbas]

Context

Removed automatic NUMA MatchAttribute constraints from the webhook (commit 624fbee). Webhook now generates plain DeviceRequests — scheduler picks devices freely. NUMA affinity is not enforced unless explicitly requested.

Problem with automatic NUMA constraints

With "4/4" (4 pairs, all same NUMA), the webhook generated:

constraints:
- requests: [pair-0, pair-1, pair-2, pair-3]
  matchAttribute: composite/numaNode

If no single NUMA zone had 4 free pairs (but 4 existed across 2 zones), the scheduler hard-failed. The constraint is not best-effort — it is a hard requirement. Pod stays Pending even though devices exist.

The old webhook avoided this by scanning ResourceSlices and falling back to cross-NUMA when a single zone was insufficient. We explicitly do not scan ResourceSlices.

Options for NUMA support (when we add it back)

Option 1: Explicit annotation format

composite.dra/gpu-nic-pairs: "4"
composite.dra/numa-policy: "preferred"  # or "required" or "none"

• none (default): no constraints, scheduler picks freely
• preferred: add constraints, but use separate claims so partial failure is tolerable
• required: hard MatchAttribute — fail if not satisfiable

Option 2: NUMA-ordered ResourceSlices (driver-side)

Publish one ResourceSlice per NUMA zone. Allocator uses first-fit → naturally packs same-NUMA when capacity allows. No constraints needed. Heuristic, not guaranteed.

Option 3: Per-NUMA DeviceClasses + extended resources

Separate DeviceClasses per NUMA zone with extendedResourceName. User explicitly requests from a NUMA zone. Guaranteed but requires user to know topology.

Option 4: Webhook scans ResourceSlices

Add ResourceSlice awareness back to the webhook to determine feasible NUMA grouping before generating constraints. Adds coupling and complexity — moves toward the old webhook model.

Current position

Do not manage NUMA unless explicitly asked for. Simple count-based allocation covers the common case. NUMA optimization is an advanced feature that should be opt-in and fail-safe.

Questions

1. Which option (or combination) should we pursue?
2. Is best-effort NUMA (option 2, slice ordering) sufficient for most workloads?
3. Should the webhook support a numa-policy annotation, or should NUMA be a DeviceClass concern?

[thameem-abbas]

On Kube 1.36+, with the KEP-4816, The user can still request for devices with NUMA attribute match as a preferred constraint. This is potentially expressible today as :

firstAvailable + matchAttribute on composite/numaNode.

With composite driver:

  resourceClaim:
    spec:
      devices:
        firstAvailable:
          - # prefer: same NUMA
            requests:
              - name: pair-0
                deviceClassName: composite.dra.io
              - name: pair-1
                deviceClassName: composite.dra.io
            constraints:
              - requests: [pair-0, pair-1]
                matchAttribute: "composite/numaNode" <Or a new attribute name we surface/create>
          - # fallback: any NUMA
            requests:
              - name: pair-0
                deviceClassName: composite.dra.io
              - name: pair-1
                deviceClassName: composite.dra.io

[thameem-abbas]

consumesCounters + TAS/Gang Scheduling Integration

Beyond the firstAvailable approach for preferred NUMA constraints, there's a broader integration story with Kueue's Topology Aware Scheduling (TAS) and gang scheduling that affects how we think about NUMA and capacity.

consumesCounters (KEP-5075, beta K8s 1.36)

The consumesCounters field on a Device in a ResourceSlice declares how much of a shared capacity pool (CounterSet) the device uses when allocated. Devices and sharedCounters live in separate ResourceSlice objects but share the same pool.name.

The composite driver can leverage this for consumable capacity (CPUs, memory, bandwidth) in two modes:

Passthrough mode: Composite device's consumesCounters references the underlying driver's CounterSet. Multiple composite devices sharing a topology group share the same counter. Scheduler manages contention via CounterSet deduction.

Locked-in mode: Each composite device gets its own CounterSet with pre-divided capacity (e.g., quarter-partition = 32 of 128 cores). Consumed entirely on allocation.

Why this matters for TAS

Kueue TAS currently has no DRA awareness — documented limitation: "DRA resources are not accounted for in TAS capacity calculations." However, with KueueDRAIntegrationPartitionableDevices feature gate, Kueue reads consumesCounters from ResourceSlice devices for quota accounting.

The composite driver's per-node ResourceSlices are the right primitive for this integration:

• TAS needs per-node device capacity → composite RS provides this directly
• Gang scheduling (KEP-4671) needs cross-node paired-device inventory → composite RS visible natively
• KEP-5732 (DRA-aware workload scheduling) will consume standard ResourceSlices → composite RS is directly usable

Publishing consumesCounters on composite devices enables Kueue to do accurate capacity-aware quota for paired devices — e.g., "this workload consumes 4 GPU-NIC pairs, each consuming 20Gi GPU memory and 100Gbps NIC bandwidth."

Intra-node bin packing

This connects to the NUMA bin packing discussion — if composite devices carry both topology attributes (composite/numaNode, composite/coupling) AND consumesCounters, the scheduler + Kueue can make informed decisions about:

1. Where to place (TAS — which node/rack)
2. What to pair (DRA scheduler — which composite devices, preferring same-NUMA via firstAvailable)
3. How much capacity is consumed (counter-based quota)

References:

• KEP-5075: DRA Consumable Capacity
• Kueue DRA integration
• KEP-5732: Topology-Aware Workload Scheduling