v0.17.0

Added

• Added preemption delay: a kai.scheduler/preemption-delay pod annotation (or preemptionDelay PodGroup spec field, metav1.Duration format) defines a minimal pending time before a workload may trigger eviction of others via preempt, reclaim or consolidation — giving cluster autoscalers a window to provision nodes first. The window re-arms after each eviction (kai.scheduler/last-eviction-timestamp annotation); allocation into free capacity and the workload's own evictability are unaffected (docs, design). #1832
• Added global.resourceReservation.createServiceAccount Helm value (default true) to allow disabling creation of the resource-reservation ServiceAccount, for embedding KAI in a parent chart that creates the ServiceAccount itself.
• Added defaultPriorityClasses.enabled Helm value (default true) for installations that manage KAI PriorityClasses externally.
• Added GitOps/ArgoCD install support (guide): kaiConfig.render Helm value (default false) renders the kai-config Config CR inline as a tracked release resource (mutually exclusive with kaiConfigDeployer.enabled), openshift value (default false) forces OpenShift mode where lookup auto-detection is unavailable under offline rendering, and ArgoCD PostDelete hook and Prune=false annotations (requires ArgoCD >= 2.10). #1794 #1751
• Added topology level aliases: a Topology level may declare an alias (e.g. rack), usable in place of the raw node label key in a workload's requiredTopologyLevel/preferredTopologyLevel. Aliases are one-to-one (unique within the Topology and must not collide with a nodeLabel, enforced by a new Topology validating webhook) and may be edited freely (the levels immutability rule now freezes only the nodeLabel structure). When a level has no alias, behavior is unchanged and raw label keys keep working. #1498
• Added a Karta fallback podgrouper plugin that lets workload owners define gang-scheduling behavior declaratively via Karta definitions, without writing a native KAI plugin. The plugin translates a Karta gangScheduling.podGroup instruction into a KAI PodGroup with optional SubGroups and topology constraints; native KAI plugins take precedence, and the alpha podGroups instruction format remains supported for compatibility. #1877 davidLif
• Publish FIPS-enabled image variants (<version>-fips) for every release, built with the Go toolchain's native FIPS 140-3 mode (GOFIPS140), and added a global.fips Helm value (default false) that appends -fips to every resolved image tag (guide). #1867
• Added global.resourceReservation.createNamespace Helm value (default true) to allow disabling creation of the resource-reservation namespace, for embedding KAI in a parent chart that creates the namespace itself.
• Reservation pods inherit fractional pod tolerations
• Helm value global.priorityClassName sets a PriorityClass on all KAI control-plane pods
• NUMA-aware node scoring preferring fewest-NUMA-zone placement
• Added support for configuring scheduler Pod Disruption Budget via Helm values (scheduler.podDisruptionBudget) when running multiple replicas per scheduling shard. #1624 dttung2905
• override stale gang eviction grace period #1913 thebhdn
• Added support for configuring pod-grouper Pod Disruption Budget via Helm values (podgrouper.podDisruptionBudget) when running multiple replicas #1477 dttung2905
• Added support for DRA-backed extended resources (KEP-5004). Pods can request a DeviceClass's extendedResourceName without a ResourceClaim, and the scheduler routes it through DRA. See the design DRA-backed extended resources. #1943 gshaibi

Changed

• Reduced scheduler memory allocations and improved performance on clusters using NUMA topology alignment.
• make changelog accepts KIND and BODY vars for non-interactive use by agents
• Scenario search now skips re-simulating equivalent victim-set candidates that already failed simulation for the same pending job (reclaim, preempt, consolidation). Skipped candidates are recorded as state="duplicate" in scenario_search_scenarios_total. #1719
• Podgrouper now preserves an existing PodGroup's topology constraint when the workload does not specify one, so an externally-assigned topology is not overwritten. Workload topology annotations still take precedence when present.
• Removed unused queuecontroller.certSecretName and admission.certSecretName Helm values; webhook TLS secrets are created and managed by the operator (queue-webhook-tls-secret, kai-admission-webhook-tls-secret). #1791 dttung2905

Fixed

• Scenario search no longer leaks a rejected scenario's victim nodes into the probe's feasible-node set: the solver now rolls back feasible-node additions after validator-rejected and errored simulations, not only after cleanly unsolved ones. #1719
• Reduced scheduler memory use during large reclaim operations by removing redundant per-job-pair min-runtime protection caching; effective min-runtime durations remain cached per queue pair. #1808
• Podgrouper now rejects negative PyTorch replica indexes and LWS worker indexes, and caps the number of subgroups created for block-level segmentation at 10000 to avoid unbounded PodGroup fan-out. davidLif
• Fixed extended resources present on only a subset of nodes being reported as unavailable cluster-wide: ResourceVector.SetMax now grows the accumulator to the longer vector's length instead of silently dropping resource indices discovered after the first-iterated node, which caused pods requesting such resources to be rejected as unschedulable ("No node in the node-pool has X resources") depending on node map iteration order. #1851
• In the fractional admission checks, check that the fractional value can be parsed as a quantity. #1798 davidLif
• Scoped the operator's informer cache for Pods, Leases and EndpointSlices to the KAI namespace and stripped managed fields from cached objects. Since v0.15.0 the operator cached every such object in the cluster, so its memory grew with cluster size and exceeded the default 256Mi limit on large clusters. #1780
• Reduced transient scheduler allocations during large reclaim operations by comparing proportion queue state and cached resource vectors directly instead of repeatedly materializing resource maps.
• Fixed scheduler panic during reclaim when building eviction messages for jobs in root-level queues (ParentQueue empty) that reclaim across hierarchy branches. #1863
• Block NaN value for fraction in the pod admission #1798 davidLif
• Reduced allocation churn in the scheduler hot path: cached Schedulable() result as a package-level singleton and lazily formatted logNodeSetsPluginResult node names only when verbose logging is enabled.
• Use the maximum gpu size ine the cluster rather then the minimum when checking a potential overLimit or isNonPreemptebleOverquota for a pod. #1792 davidLif
• Fix the MinNodeGPUMemoryMiB calculation in the scheduler. This affected allocations for fractional pod requesting gpu "gpu-memory". #1792 davidLif
• Scheduler cache now filters terminal Pods at watch time to reduce memory use, while still watching Pods bound by other schedulers so their resource usage is counted in allocatable calculations. #1645 enoodle
• Saturated the DRA GPU device-count accumulation so a ResourceClaim requesting an oversized device count can no longer overflow the queue controller's int64 GPU total to a negative value. #1873 thc1006
• Clean deleted Pod status updates
• Prevent some scheduler OOM kills by setting Go's memory limit
• Clear stale PodGroup UnschedulableOnNodePool conditions after workloads schedule.
• NUMA plugin no longer counts non-integral container CPU toward NUMA alignment
• NUMA plugin now aligns non-Guaranteed pods that request GPUs, matching the kubelet's device manager (device alignment is QoS-independent).
• Status-updater drops status/patch updates for deleted PodGroups instead of retrying NotFound errors indefinitely, stopping log floods. #1945
• Restricted Helm post-delete cleanup to KAI operator-managed Deployments and preserved externally managed kai-config resources when kaiConfigDeployer.enabled=false.
• Fixed reclaim abandoning valid over-quota victims when an unrelated under-deserved queue appeared earlier in victim ordering. #1750
• Fixed GPU-sharing pods with dotted pod names generating invalid ConfigMap-backed volume names. Volume names are now sanitized to valid DNS labels while preserving original ConfigMap references used for shared-GPU injection.
• Scheduler now exits on 401 Unauthorized API responses instead of retrying indefinitely with a stale ServiceAccount token. #1817
• Removed unnecessary pod-affinity pre-score allocation churn
• Avoid repeated victim candidate allocations during reclaim, preempt, and unlimited-candidate consolidation searches #1850
• Segmented elastic PyTorchJob now respects minReplicas instead of requiring all worker segments
• Upgrade test suite retries queue creation when admission webhook is not yet ready
• Reduce retained scheduler memory with compact fit-error counts and on-demand details. #1827 enoodle
• Fixed admission and scheduler PodDisruptionBudget reconciliation, including drift recovery and cleanup when no longer desired. #1477 dttung2905
• Segmented PyTorch and LWS PodGrouper now uses minSubGroup on parent SubGroups, fixing admission webhook rejection. #1927