v0.16.0

Added

• Added a bounded scenario generator portfolio for reclaim, preempt, and consolidation search, with SchedulingShard.spec.scenarioSearchBudgets time-budget configuration and production scenario-search metrics.
• Added an opt-in deviceaccess admission plugin (--block-nvidia-visible-devices, config field admission.blockNvidiaVisibleDevices, default disabled) that (1) rejects pods overriding the NVIDIA_VISIBLE_DEVICES environment variable with values other than void/none (or via a valueFrom reference), and (2) injects NVIDIA_VISIBLE_DEVICES=void into containers that do not request a GPU, blocking their access to GPUs on the node.
• Added support for configuring admission Pod Disruption Budget via Helm values (admission.podDisruptionBudget) #1490 dttung2905
• Added an opt-in hamicore binder plugin (depends on gpusharing) to write the HAMI-core GPU memory limit (CUDA_DEVICE_MEMORY_LIMIT) for fractional GPU pods.
• Added global.podSecurityContext, global.resourceReservation.namespaceLabels, nodescaleadjuster.labels, crdupgrader.resources, topologyMigration.resources, and postCleanup.resources to the Helm. chart.
• Skill to capture and run snapshots
• Added kaiConfigDeployer.enabled Helm value (default true) to allow disabling the post-install/post-upgrade hook that applies the kai-config CR, for managing the CR outside of the chart.
• Added defaultShard.enabled Helm value (default true) to allow installing KAI without deploying the chart-managed default SchedulingShard CR.
• Added NUMA-aware scheduling (v1). A new numa scheduler plugin predicts the kubelet Topology Manager's admission verdict for the single-numa-node and restricted policies from NodeResourceTopology (NRT) data, to prevent TopologyAffinityErrors. The plugin is opt-in per shard (enable the numa plugin in a SchedulingShard). Shipped alongside it is an optional per-node NUMA placement exporter DaemonSet that exports the kubelet podresources API to each pod's observed NUMA placement. This is v1 — feasibility filtering and per-zone correctness; node scoring/optimization is future work. See the design for details: NUMA-Aware Scheduling via NodeResourceTopology.

Changed

• Scoped admission runtimeClassName injection to GPU fraction pods only; whole-GPU pods are no longer mutated. admission.gpuPodRuntimeClassName is deprecated in favor of admission.gpuFractionRuntimeClassName. Reservation pod runtimeClassName now defaults to empty. #1543 davidLif
• Removed redundant PodDisruptionBudgetImplemented guard from operator PDB creation helper #1613 dttung2905
• Updated Go toolchain and base build images to v1.26.3.
• Breaking: The podgroup produced for JobSet is now produces as a single PodGroup per JobSet with a two-level SubGroup hierarchy (one parent SubGroup per replicatedJob, one leaf SubGroup per replica) regardless of startupPolicyOrder. The kai.scheduler/batch-min-member annotation on the JobSet now overrides the root minSubGroup; the same annotation on replicatedJobs[].template.metadata.annotations overrides the leaf minMember (defaulting to template.spec.parallelism). #1617 davidLif

Fixed

• Reduced scheduler heap retention after scheduling cycles by clearing completed session snapshots and callback references, and by releasing the node scoring pool without waiting for finalizers.
• Fixed Helm chart prometheus RBAC always being installed when prometheus.enabled is false, and the kai-prometheus ClusterRoleBinding referencing the prometheus ServiceAccount in hardcoded kai-scheduler namespace instead of the Helm release namespace #1684 dttung2905
• Fixed post-delete cleanup hook hardcoding kai-scheduler namespace instead of Helm release namespace on helm uninstall #1619 dttung2905
• Improved solver performance in some large reclaim scenarios #1627 itsomri
• Grove grouper now sets minSubGroup (equal to the number of child SubGroups) instead of minMember=0 on parent SubGroups generated from topologyConstraintGroupConfigs #1639 davidLif
• Fixed Helm chart not wiring podgrouper.queueLabelKey into spec.global.queueLabelKey on the Config CR, so custom queue label keys were ignored at install time #1655 dttung2905
• Fixed scheduler nil-pointer panic in the preempt scenario builder when a (partial) job has no tasks to allocate (NewIdleGpusFilter dereferenced a nil scenario); added the missing nil-guard matching the sibling filters #1664 sam-huang1223
• Fixed default node-scale-adjuster image name (node-scale-adjuster → nodescaleadjuster) so it matches the image published to GHCR
• Fixed duplicate GPU reservation pods being created for a single gpu-group on a node (each reserving a different physical GPU), which corrupted the scheduler's fractional-GPU accounting and left devices unschedulable. Reservation pods are now named deterministically per (node, gpu-group) and treat AlreadyExists as success, so concurrent or retried binds collide on one object instead of duplicating #1673