You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
(No, really, you MUST read this before you upgrade)
Kubelet: fixed several bugs in the ID range handling for pods using user namespaces
(spec.hostUsers: false).
The kubelet no longer fails to start when the kubelet user exists but the getsubids binary
is not installed. It now falls back to the default ID range, as documented.
Pods are no longer assigned the ID range ending at 4294967295 (2^32-1). The kernel treats that
ID as invalid and refuses to map it, so a pod assigned that range failed to start. The default
range now ends one idsPerPod block lower.
the kubelet now refuses to start if the subordinate ID range configured for the
kubelet user in /etc/subuid and /etc/subgid ends at or past 4294967295, since such a range
either contains the unmappable ID or extends past the end of the 32-bit ID space. Nodes
configured this way, for example kubelet:65536:4294901760, must reduce the count by at least
one idsPerPod (65536 by default) before upgrading.
if a pod on the node was assigned the invalid ID range by an earlier kubelet,
that range is still recorded in the pod's directory, and the upgraded kubelet refuses to start:
create user namespace manager: record pod mappings for existing pod "<uid>": id 4294901760 is out of range
Delete any pod stuck in ContainerCreating with a FailedCreatePodSandBox event, or drain the
node, before upgrading the kubelet. (#141345, @halaney) [SIG Node]
Changes by Kind
API Change
Add garbage collection finalizer protection (scheduling.k8s.io/compositepodgroup-protection) for CompositePodGroup objects to ensure hierarchical bottom-up cleanup of pod groups. (#141021, @jdzikowski) [SIG API Machinery, Apps, Auth, Scheduling and Testing]
DRA consumable capacity: requesting capacity had to be done with the capacity name exactly as published by drivers, i.e. typically without the driver name prefix. When requesting with a fully-qualified name, the request failed. Now both unqualified and fully-qualified names work. A related, less likely problem existed when a DRA driver changed how it published capacity and some capacity was allocated already. (#142202, @pohly) [SIG API Machinery, Apps, Node and Scheduling]
Fixed JSONSchemaProps comment to properly render JSON Schema URL as a valid link (#136159, @kfess) [SIG API Machinery]
Implemented status updates for CompositePodGroup objects. (#140670, @jdzikowski) [SIG API Machinery, Scheduling and Testing]
Update TokenReview internal type to improve cast speed from latest external version (#142750, @michaelasp) [SIG Auth]
Feature
Added the Alpha CloudNodeAdditionalLabelsReconciliation feature gate, disabled by default. When enabled in the cloud-controller-manager, the cloud node controller periodically adds or updates node labels returned through InstanceMetadata.AdditionalLabels. Provider values overwrite conflicting changes to the same label keys, including changes made by other controllers. Initial node labeling is unchanged, and labels no longer returned by the provider are not removed. (#141616, @stblatzheim) [SIG API Machinery and Cloud Provider]
Hack/local-up-cluster.sh now supports quantum crypto keys and certs for testing (#142401, @rphillips) [SIG Architecture]
Promoted Windows support for CPU and memory affinity (feature gate WindowsCPUAndMemoryAffinity) to Beta, and made it enabled by default. (#142500, @zylxjtu) [SIG Node and Windows]
Separate deleteGet and updateGet etcd operation in etcd_request_* metrics and add reason label to error. (#142484, @serathius) [SIG API Machinery, Etcd and Instrumentation]
The feature gate KubeProxyNFTablesLocalhostNodePorts is now beta and enabled by default (#142490, @AustinAbro321) [SIG Network]
Update node local DNS to version 1.27.1 (#142686, @bowei) [SIG Cloud Provider and Network]
Bug or Regression
API server returns 400 instead of 500 status code on GET request with invalid revision (#142591, @serathius) [SIG API Machinery, Etcd and Testing]
Creating or updating a resource.k8s.io object with a qualified name that contains more than one slash, such as a device attribute key a/b/c, is now rejected. Objects that already store such a name remain updatable as long as the field holding it is left unchanged; for a ResourceSlice, as long as spec.devices is unchanged, since any change to that atomic list revalidates every device in it. (#141644, @krishhna24) [SIG API Machinery, Node and Testing]
DRA: fixed a deadlock when an ErrorHandler calls Update, or a kubelet plugin's HandleError calls PublishResources, in response to pools that do not match ReconcilePoolWithName. Such resources now replace the desired state instead of being rejected; the other pools are reported to the error handler and are not published. Initial resources with other pools no longer cause a panic. Also fixed a deadlock when a kubelet plugin's HandleError calls PublishResources while the plugin stops. (#141108, @junnncct1106) [SIG Node]
Fix: kubectl create ingress panics on rules with extra equals signs (#142640, @weizhoublue) [SIG CLI]
Fixed API warnings on ReplicationController updates, which described the previous pod template instead of the new one. (#142443, @thc1006) [SIG Apps]
Fixed ResourceSlice validation to reject a per-device nodeSelector with more than one node selector term, as it already does for the slice-level nodeSelector. Allocating a device with such a selector could make pod scheduling fail with an error. A ResourceSlice that already stores such a selector can still be updated as long as spec.devices is unchanged. (#141213, @thc1006) [SIG Node]
Fixed kubectl kuberc set leaving a stale credential plugin allowlist when the policy is changed to AllowAll or DenyAll, which caused kubectl to fail. (#142564, @kfess) [SIG CLI and Testing]
Fixed a panic in kubectl kuberc view when no kuberc file exists and the user declines to generate a default one. (#141099, @tancheng33) [SIG CLI]
Fixed k8s.io/apimachinery/pkg/util/yaml silently dropping the last line of a YAML stream when it has no trailing newline and its length is an exact multiple of the reader buffer size. (#142527, @SLoeuillet) [SIG API Machinery]
Fixed kubelet so that pods using dnsPolicy: Default no longer inherit scoped IPv6 link-local nameservers (for example fe80::1%3) from the node resolver configuration unless the pod uses the host network namespace. (#140880, @OchukoWH) [SIG Network and Node]
Fixed refcounting leak in the node authorizer graph for pods which reference secrets, configmaps, or pvcs multiple times (#142574, @liggitt) [SIG Auth]
Fixed the HorizontalPodAutoscaler utilization and usage calculations wrapping to a wrong or negative current value when pod metric or request totals exceeded int64 or the utilization percentage exceeded int32; the reported percentage now saturates at the int32 limit. (#142192, @KR-Ravindra) [SIG Apps and Autoscaling]
Fixed the ResourceClaim controller repeatedly restoring terminal Pod entries in status.reservedFor for allocated claims that remain in use by other consumers. (#141276, @neBM) [SIG Apps and Node]
Fixed the taint_eviction_controller_pod_deletion_duration_seconds metric, which recorded nanoseconds scaled by 1e9 instead of seconds. Latencies below about 9.2 seconds fell into the +Inf bucket, and longer ones overflowed and were recorded as arbitrary, often negative, values. (#141742, @imusmanmalik) [SIG Apps and Scheduling]
Fixed the field path in CSIDriver validation errors for spec.attachRequired, which was previously reported as spec.attachedRequired — a field that does not exist in the API. (#141597, @krishhna24) [SIG Storage]
Kube-apiserver and other servers built on k8s.io/apiserver now accept up to 16384 request header values. Without this, Go 1.27's default of 500 rejects impersonated or proxied users with about 500 or more groups with HTTP 431. Releases built with Go 1.26 had no separate header-count limit. (#142607, @cwedgwood) [SIG API Machinery and Auth]
Kube-controller-manager: Informer metrics now use correctly pluralized resource names (#141566, @neoLsH) [SIG API Machinery]
Kubelet now reports allocated resource health statuses in deterministic order, preventing unnecessary Pod status updates when resource health is unchanged. (#142340, @photoszzt) [SIG API Machinery, Apps, Auth, Cloud Provider, Node, Release, Scheduling, Storage and Testing]
Kubelet: fixed a regression on Windows where the container log file could not be read when the container log directory is a volume mount point (reparse point) onto a secondary disk, because symbolic-link resolution cannot traverse such mount points. (#141699, @MartinForReal) [SIG Node and Windows]
The apiequality.Semantic.DeepDerivative function fixes slice comparisons to treat a RHS slice with additional items compared to a non-empty LHS slice as different. For example, [1] and [1,2,3] were previously considered identical, and are now considered different. (#137264, @ShaanveerS) [SIG API Machinery]
kubectl label now prints messages about nonexistent labels to stderr instead of stdout, so they no longer break JSON or YAML output. (#142436, @jwc1love) [SIG CLI]
Other (Cleanup or Flake)
Add consistency store to client-go and deprecate in pkg/controller (#142537, @michaelasp) [SIG API Machinery, Apps, Autoscaling and Node]
Applyconfiguration-gen now honors +k8s:openapi-model-package when looking up OpenAPI definitions, and fails if the schema has no definition for a root type. (#142697, @yongruilin) [SIG API Machinery]
Graduate HPA observedGeneration to GA. The HPAGeneration feature gate is now locked to enabled. (#142386, @adrianmoisey) [SIG Apps and Autoscaling]
Ktesting: the former internal packages are now available as k8s.io/ktesting (core package + API) and k8s.io/client-go/ktesting (additional APIs for testing against an apiserver). (#142146, @pohly) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Release, Scheduling, Storage, Testing and Windows]
Kubelet: gRPC probes fail immediately when the connection is refused instead of retrying until the probe timeout. The failure message for a server that never answers is now "timeout: health rpc did not complete within ". (#142457, @dims) [SIG Node and Testing]
Kubelet: the pods API gRPC server now limits concurrent streams per connection to 100 and applies its request rate limit to WatchPods stream openings. (#142458, @dims) [SIG Node]
Kubemark has been removed from the kubernetes repository (#142700, @upodroid) [SIG API Machinery, Cloud Provider, Instrumentation, Network, Scalability, Storage and Testing]
The MatchLabelKeysInPodAffinity feature gate has been removed. (#141744, @wasylkowski-a) [SIG Node, Scheduling and Testing]
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
(No, really, you MUST read this before you upgrade)
Out-of-tree kube-scheduler plugins using the NominatedPodsForNode method must now pass logger klog.Logger as the first argument. (#142118, @jdzikowski) [SIG Scheduling and Testing]
Removed the PodSchedulingReadiness feature gate from kube-scheduler and related components as the feature is permanently enabled. (#141741, @wasylkowski-a) [SIG Scheduling]
Removed the SeparateTaintEvictionController feature gate from kube-controller-manager. Remove this gate from existing --feature-gates configuration before upgrading. Taint-based eviction is now permanently implemented by the standalone taint eviction controller, which can still be disabled using --controllers=-taint-eviction-controller. (#141789, @wasylkowski-a) [SIG API Machinery, Apps, Node, Scheduling and Testing]
Kubernetes is now built with Go 1.27.1 (#141663, @liggitt) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling and Storage]
Updated google.golang.org/grpc to v1.82.2 to address CVE-2026-84445. (#141976, @harshitgupta31415) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Network, Node and Scheduling]
Updated google.golang.org/grpc to v1.84.0 to pick up the fix for CVE-2026-84304. (#142410, @dims) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Scheduling and Storage]
API Change
Added resource.MaxMilliQuantity(), which returns the largest quantity whose MilliValue and ScaledValue(resource.Milli) do not overflow an int64. For a non-negative quantity, compare it against the returned bound with Quantity.Cmp before calling MilliValue or ScaledValue(resource.Milli). Comparing Quantity.Value against resource.MaxMilliValue for this is unreliable, because Value itself overflows for a quantity larger than int64. (#140674, @thc1006) [SIG API Machinery]
Adds (Beta) support for requesting pod certificates to be signed using ML-DSA algorithms. This updates both PodCertificateRequest and the projected volume type. (#142108, @everettraven) [SIG API Machinery, Apps, Auth, Autoscaling, Node and Scheduling]
Adds support for using PKCS#10 signing requests signed with ML-DSA keys in CertificateSigningRequest, gated by the CertificateSigningRequestMLDSA feature-gate. (#142196, @everettraven) [SIG API Machinery, Apps and Auth]
Allow CompositePodGroup.Spec.SchedulingPolicy.Gang.MinGroupCount to be mutable, and automatically re-enqueue pods in the scheduler when MinGroupCount decreases. (#141023, @jdzikowski) [SIG API Machinery, Scheduling and Testing]
CRD validation failures for minItems and minProperties now report Too few: <count>: must have at least N items instead of the misleading Invalid value: <count>. (#141641, @yongruilin) [SIG API Machinery]
Clearing node.spec.podCIDRs on update now reports "field cannot be cleared once set" instead of "node updates may not change podCIDR except from "" to valid". (#141260, @krishhna24) [SIG API Machinery, Apps and Testing]
Clients will now receive resource versions on delete when deleting objects. This resource version is either the actual delete or a version when the object was observed to be deleted at. (#138724, @michaelasp) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]
Exposed PlacementFeasible as an extension point in kube-scheduler, allowing for out-of-tree PodGroup placement feasibility evaluation. (#141314, @macsko) [SIG Scheduling]
Kube-scheduler: TAS placement selection now respects NominatedNodeName. This does not yet apply to PodGroups belonging to a CompositePodGroup; see #140863. (#139472, @mboersma) [SIG API Machinery, Scheduling and Testing]
Kubelet: KubeletConfiguration now supports selecting ECDSA, RSA, or ML-DSA key algorithms for client and server certificate requests. The default remains ECDSA-P256. (#142046, @rphillips) [SIG API Machinery, Auth, Node and Scheduling]
Preemption-capable scheduler extenders must now handle candidates with empty victim lists. (#135486, @FouoF) [SIG Scheduling]
Removed locked GA feature gates CustomResourceFieldSelectors and CRDValidationRatcheting. (#141207, @Jefftree) [SIG API Machinery and Testing]
Resource.Quantity: calling String or encoding a quantity is now guaranteed to not mutate the instance. Previously it did in some cases. If you know that an instance will need the string representation multiple times, use CacheString to pre-populate the value at a time when the caller owns the instance and can safely mutate it. This is not necessary for instances coming from ParseQuantity, only for instances created differently or modified through math operations. (#142277, @pohly) [SIG API Machinery]
The OpenAPI schema for ValidatingAdmissionPolicyBinding and MutatingAdmissionPolicyBinding now correctly marks spec.paramRef.parameterNotFoundAction as required, matching long-standing apiserver validation. Client-side validators (e.g. kubeconform) can now catch the missing field before apply. (#141527, @grosser) [SIG API Machinery]
The DynamicResourceAllocation feature gate, which has been locked to default on since v1.35, has been removed. Dynamic resource allocation is now unconditionally enabled. (#141946, @troychiu) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
The endpoint_slice_controller_changes metric is deprecated in favor of endpoint_slice_controller_changes_total. Both metrics are emitted during the deprecation period. (#140447, @cipheraxat) [SIG Architecture, Instrumentation and Network]
The apidiscovery.k8s.io/v2beta1 types and the AggregatedDiscoveryRemoveBetaType feature gate are removed, including the k8s.io/api/apidiscovery/v2beta1 Go package. (#141208, @Jefftree) [SIG API Machinery]
The deprecated and unused metrics.k8s.io/v1alpha1 API has been removed. Clients still using it should migrate to metrics.k8s.io/v1 or metrics.k8s.io/v1beta1. (#140862, @dalaoqi) [SIG Instrumentation]
The documentation of Node.status.volumesAttached[].devicePath now states the platform-specific semantics: on Linux it is the host block-device node, on Windows it carries the CSI VolumeID. (#141861, @abhinav-phi) [SIG API Machinery, Apps, CLI, Storage and Windows]
Updated the authorization API group to improve OpenAPI schema correctness for fields being optional or required. (#139728, @krishhna24) [SIG Auth]
Updated the batch API group to improve OpenAPI schema correctness for fields being optional or required. (#140010, @krishhna24) [SIG Apps]
resource.Quantity gains AsScaledInt64 and AsMilliInt64, which report int64 overflow with a boolean. Value, MilliValue and ScaledValue now saturate to math.MinInt64 or math.MaxInt64 on overflow instead of returning a wrapped value. A kubelet CPU reservation no longer wraps to a negative value when the configured quantity exceeds that range. (#141305, @thc1006) [SIG API Machinery and Node]
Feature
Add a library for generating unique names for workloads and podgroups with builder library (#141557, @kannon92) [SIG Apps and Scheduling]
Added auto complete support to kubectl explain. (#140058, @oscrx) [SIG CLI]
Added the cacher_queue_latency stage to the apiserver_watch_events_dispatch_duration_seconds metric to measure watch event queuing latency in the cacher. (#141895, @richabanker) [SIG API Machinery and Instrumentation]
Added the watcher_queue_latency stage to the apiserver_watch_events_dispatch_duration_seconds metric to measure watch event queuing latency in individual watch buffers. (#141896, @richabanker) [SIG API Machinery and Instrumentation]
Client-go: add support for ML-DSA keys from the crypto/mldsa package of Go 1.27 (#142038, @neolit123) [SIG API Machinery and Auth]
Conformance test Pods, completes the lifecycle of a Pod and the PodStatus now cover resetting the PodReadyToStartContainers condition (#141460, @Priyankasaggu11929) [SIG Node and Testing]
DRA: the helper code for ResourceSlice publishing now validates that drivers use the shorter attribute and capacity names without the driver prefix. DRA driver authors can opt out of this check. (#142218, @pohly) [SIG Node]
DRADeviceTaints and DRADeviceTaintRules feature gates are now locked to true (GA) in 1.38. Users with these gates explicitly set to false in --feature-gates should remove those entries as they will be ignored. Both gates will be removed in 1.41. (#140256, @SamWangEng) [SIG Apps and Node]
Extends the scheduler's queue metrics to support CompositePodGroup (#141800, @dom4ha) [SIG Instrumentation and Scheduling]
Graduated container env files to GA. The EnvFiles feature gate is now locked to enabled. (#139115, @HirazawaUi) [SIG Node]
KEP-4858 "IP/CIDR Validation improvements" is now GA, and the stricter rules
for validating IP and CIDR values in Kubernetes API types are now always in
effect. (#141979, @danwinship) [SIG API Machinery, Apps, Network and Testing]
Kube-apiserver: Requests to unknown /apis/... paths and final delegation 404s now return a JSON Status object with Content-Type: application/json instead of a plain-text "404 page not found" body. (#138219, @yashsingh74) [SIG API Machinery and Testing]
Kube-apiserver: added the alpha ManagedFieldsOptOut feature gate (off by default). When it's enabled, clients can omit metadata.managedFields from responses by adding drop=metadata.managedFields to the Accept header, e.g. Accept: application/json;drop=metadata.managedFields. This works for all verbs including watch, for built-in and custom resources, in every supported media type. (#139561, @yongruilin) [SIG API Machinery and Testing]
Kube-controller-manager: promoted metric 'volume_operation_errors_total' from ALPHA to BETA stability. (#141536, @obarisk) [SIG Apps, Instrumentation and Storage]
Kubeadm: add support for the ML-DSA encryption algorithm. "ML-DSA-44", "ML-DSA-65" and "ML-DSA-87" are now allowed values for ClusterConfiguration.EncryptionAlgorithm for new clusters using the v1beta4 and the still disabled (WIP) v1 API. Note that while kubeadm is adding this support, support in other components like kube-apiserver and etcd might be added a bit later and until then a kubeadm default setup with ML-DSA might fail. (#142037, @neolit123) [SIG Cluster Lifecycle]
Kubectl debug: The sysadmin profile now auto-detects Windows nodes and configures debug pods as Windows Host Process Containers with NT AUTHORITY\SYSTEM access. (#138182, @rzlink) [SIG CLI and Windows]
Kubernetes is now built using Go 1.27.1 (#142176, @cpanato) [SIG Release and Testing]
Moves node authorization processing outside of listener callback, preventing lock contention. (#140803, @michaelasp) [SIG API Machinery and Auth]
Return resource version in status on pod binding (#142327, @michaelasp) [SIG API Machinery and Node]
Return resource version in status on pod eviction (#142197, @michaelasp) [SIG Node]
Validation-gen: +k8s:minimum and +k8s:maximum support time.Duration fields with quoted Go duration strings, such as +k8s:minimum="1s". Integer payloads on time.Duration fields are now rejected. (#142368, @yongruilin) [SIG API Machinery]
SelfSignedCertKeyOptions in k8s.io/client-go/util/cert accepts a KeyGenerator, allowing self-signed certificates to be generated with a key algorithm other than RSA. The default remains a 2048-bit RSA key. (#142350, @antcybersec) [SIG API Machinery and Auth]
kubectl now prints a warning when creating or updating a DeviceTaintRule whose deviceSelector
is present but empty, since that selector matches every device from every driver in the cluster.
Scripts using kubectl ... --warnings-as-errors against an intentional "select all" rule will see a
non-zero exit code where they previously didn't, even though the object is still created
successfully — the same behavior PodSecuritywarn-level violations already have. (#141831, @yogeshbendre)
validation-gen now accepts a --validation-extensions-file flag, taking a YAML file that declares project-specific +k8s:format=<name> values backed by a regular expression. (#142116, @yongruilin) [SIG API Machinery]
Documentation
Kube-scheduler metrics that carry a profile label now document that label in their help text. (#140800, @alimaazamat) [SIG Instrumentation and Scheduling]
Kubectl top pod: add example for filtering metrics by node with --field-selector (#141656, @Pratik-Redhat-Tech) [SIG CLI]
The /flagz plain-text response now notes that it reports command-line flag values (or defaults), not the effective runtime configuration, and points to /configz where available. (#140233, @ashishpatel26) [SIG API Machinery and Instrumentation]
Bug or Regression
Both distribute-cpus-across-cores=true and align-by-socket=true options were enabled, even when a single socket had sufficient capacity. (#141238, @Chunxia202410) [SIG Node]
CEL cost estimation for CRD validation rules using URL accessor functions (getScheme, getHostname, getHost, getPort, getEscapedPath, getQuery) now propagates the result size bound from the input URL. Previously, chaining a URL accessor into a string operation like matches() treated the intermediate result as unbounded, causing legitimate validation rules to be rejected for exceeding the cost budget. (#141984, @jubittajohn) [SIG API Machinery]
CPUManager now preserves TopologyManager's multi-NUMA affinity when using the
distribute-cpus-across-numa policy option, ensuring CPU allocation stays
aligned with device topology (e.g., GPUs, NICs) across NUMA nodes. (#139456, @yong-jie-gong) [SIG Node]
Client-go: a non-nil NegotiatedSerializer in rest.Config is now used when constructing clientsets instead of being unconditionally overwritten (#140317, @npinaeva) [SIG API Machinery, Auth, CLI and Instrumentation]
Client-go: restrict CA key encipherment usage to RSA keys. Key encipherment usage is no longer included for non RSA keys in self signed CA certificates generated from client-go (#142273, @adoi) [SIG API Machinery, Apps, Auth, Node and Testing]
Conntrack entries for a deleted UDP Service's frontend IPs are now cleared on deletion instead of waiting to expire. (#141371, @kiarashazarnia) [SIG Network and Testing]
DRA: Fixed scheduling failures for AllocationMode: All requests when a resource pool on a candidate node is incomplete or invalid, allowing the scheduler to continue trying other nodes. (#141898, @divyanshuprakas-h) [SIG Node]
Decoding a JSON or CBOR null into a resource.Quantity that had already been serialized left its cached text in place, so the quantity read as zero while String, JSON, CBOR and unstructured conversion kept returning the previous value. The cached text is now cleared with the rest of the value. (#141980, @thc1006) [SIG API Machinery]
Device-plugin: skip DRA resources in GetDeviceRunContainerOptions (#141426, @jackfrancis) [SIG Node]
EndpointSliceMirroring: Ignore NotFound Errors When Deleting EndpointSlices (#141608, @weizhoublue) [SIG Apps and Network]
Fix a bug in scheduler where a PodGroup and CompositePodGroup sharing the same name may cause lookup collisions. (#141864, @iomarsayed) [SIG Scheduling]
Fix drifts of CPUManager gauge metrics: CPUManagerExclusiveCPUsAllocationCount, CPUManagerSharedPoolSizeMilliCores, CPUManagerAllocationPerNUMA from true state. (#141262, @lukaszwojciechowski) (#141261, @lukaszwojciechowski) [SIG Node]
Fix incorrect ExtendedResourceCache mappings when multiple DeviceClasses use the same extended resource name, which could cause stale or incorrect resource-to-DeviceClass mappings after DeviceClass updates or deletion. (#141110, @anshulchikhale30-p) [SIG Node]
Fix kubectl top pod --sum column alignment when --no-headers flag is specified. (#140874, @0xff-dev) [SIG CLI]
Fix: Keep Zero-Exponent BinarySI Values on Integer Serialization Path (#141817, @weizhoublue) [SIG API Machinery]
Fix: silently dropped DNS resolution latency metric in restclient (#141493, @weizhoublue) [SIG Architecture and Instrumentation]
Fixed CPUManager's distribute-cpus-across-cores policy option to select CPUs using CPU topology instead of relying on logical CPU numbering, improving CPU distribution across physical cores on systems where sibling threads have contiguous CPU IDs. (#140766, @OchukoWH) [SIG Node]
Fixed CompositePodGroup validation in the scheduling cycle to ensure that child CompositePodGroups and PodGroups match the root group's priority and preemptionPolicy. (#141930, @macsko) [SIG Node and Scheduling]
Fixed DRA kubelet plugin helper rolling updates for drivers with long valid names by shortening automatic DRA service socket paths when needed. (#142003, @ryo-whaletech) [SIG Node]
Fixed Memory Manager pod-scope topology hint restoration for multi-container Pods after kubelet restart. (#141070, @AutuSnow) [SIG Node]
Fixed StatefulSet scale-down becoming blocked with Parallel Pod management when a failed scale-up leaves missing desired Pods and ResourceQuota prevents their creation. (#141057, @suii2210) [SIG Apps]
Fixed FailedToCreateEndpoint/FailedToUpdateEndpoint events from the endpoints controller being misfiled into the default namespace instead of the service's namespace. (#141509, @inspirit941) [SIG Apps and Network]
Fixed PersistentVolumeClaimstatus.capacity accepting a negative quantity whose Value() overflows (for example -9.5Gi or -1e30), and stopped a positive capacity whose Value() wraps negative (for example 9223372036854775808) from being wrongly rejected. (#141169, @semx) [SIG API Machinery, Apps and Storage]
Fixed kubectl top pod --sum printing the separator and totals rows misaligned by one column when --show-swap was also specified. (#141621, @0xff-dev) [SIG CLI]
Fixed pv_collector_total_pv_count metric reporting plugin_name="N/A" for CSI-provisioned PersistentVolumes. The label now correctly shows kubernetes.io/csi:<driver-name> (e.g. kubernetes.io/csi:ebs.csi.aws.com). (#139356, @dfajmon) [SIG API Machinery, Apps and Storage]
Fixed resource.Quantity.AsApproximateFloat64 returning NaN instead of 0 for a zero-valued quantity with a large scale. (#141937, @jpbetz) [SIG API Machinery]
Fixed resource.Quantity.String() (and MarshalJSON/ToUnstructured) silently dropping the magnitude for DecimalSI quantities larger than 10^18 (for example 1000E), and for BinarySI quantities grown past Ei via arithmetic (for example 2^70), which corrupted the value and broke round-tripping through String/JSON/YAML. Such quantities now serialize to an exact base-10 representation: decimal exponent notation where trailing zeros allow it (for example 1e21), otherwise the full decimal integer (for example 2^70 becomes 1180591620717411303424). (#140459, @semx) [SIG API Machinery]
Fixed resource.Quantity.Value() and MilliValue() to round negative values away from zero, as documented. Previously, small negative quantities could round toward positive infinity (e.g. ParseQuantity("-.484785E-7466").Value() returned 1 instead of -1). (#138510, @qflen) [SIG API Machinery]
Fixed a DRA scheduling bug where a device with a live shared allocation could also be allocated exclusively after a ResourceSlice changed it to disallow multiple allocations. (#140798, @thc1006) [SIG Node]
Fixed a bug in kubectl create ingress where malformed trailing data in --rule passed validation and silently dropped TLS configuration. (#140926, @bhuvan-somisetty) [SIG API Machinery and CLI]
Fixed a bug in the PodTopologySpread scheduler plugin where a topology spread constraint with an empty (non-nil) labelSelector (labelSelector: {}) was silently ignored instead of matching all pods in the namespace. (#141340, @0xff-dev) [SIG Scheduling]
Fixed a bug in the HorizontalPodAutoscaler: on amd64, an extremely large metric value could scale a workload down to minReplicas instead of up, and an extreme Percent scaling policy could block scaling up entirely. Replica calculations are now capped at the int32 maximum, so the HPA scales up toward maxReplicas, matching the existing behavior on all other architectures. (#141397, @ChihweiLHBird) [SIG Apps and Autoscaling]
Fixed a bug in the attach-detach controller that could delay volume detach by up to one node-status update interval (~5min) after a pod is deleted, when the volume attach was uncertain or kubelet had not yet reported the volume in Node.Status.VolumesInUse. (#141385, @huww98) [SIG Apps and Storage]
Fixed a bug where Kubelet device manager may assign the same device ID to multiple pods simultaneously. (#137534, @trozet) [SIG Node and Testing]
Fixed a bug where NoExecute device taints (DeviceTaintRule) did not evict pods using DRA-backed extended resources (pod.Status.ExtendedResourceClaimStatus). (#142289, @troychiu) [SIG Apps, Auth, Node and Scheduling]
Fixed a bug where resource.ParseQuantity failed to use the fast integer path for values near math.MaxInt64 (19-digit quantities), causing AsInt64() to incorrectly return false for valid int64 values. (#141938, @jpbetz) [SIG API Machinery]
Fixed a bug where a DeviceTaintRule with no spec.deviceSelector incorrectly matched every device cluster-wide and made all DRA devices unschedulable for NoSchedule or NoExecute taints. The rule now correctly matches no devices as documented. Use an explicit empty selector (deviceSelector: {}) to target all devices. (#141648, @marwan562) [SIG Node]
Fixed a bug where a pod could be permanently lost from the scheduling
queue when a concurrent Pop picked it up before Done() was called in
AddUnschedulablePodIfNotPresent. (#139450, @rsd-darshan) [SIG Scheduling]
Fixed a bug where binding a PersistentVolumeClaim to a PersistentVolume by name could misjudge whether the volume was large enough when the capacity or the request was larger than a 64-bit integer. (#141618, @thc1006) [SIG Apps and Storage]
Fixed a bug where kubelet logged --manifest-url-header credential values (e.g., Authorization tokens) to runtime logs at startup. Only header key names are now logged. (#140220, @rithwik-01) [SIG Node]
Fixed a bug where pods requesting extended resources backed by DRA failed to schedule when the DRA devices also contributed node allocatable resources (with the alpha DRANodeAllocatableResources feature gate enabled). (#141556, @harche) [SIG Apps, Node, Scheduling and Testing]
Fixed a bug where the kubelet could hang during Pod admission when the CPU manager distribute-cpus-across-numa and full-pcpus-only policy options were both enabled and per-NUMA CPU availability was not a multiple of the hardware thread count per core. (#140730, @kiarashazarnia) [SIG Node]
Fixed a kubelet panic when validating an in-place decrease of a container memory limit while pod-level memory usage stats are unavailable. (#141100, @tancheng33) [SIG Node]
Fixed a panic in DRA drivers that use k8s.io/dynamic-resource-allocation/client to Get a ResourceClaim and then call UpdateStatus on it, once the client has selected resource.k8s.io/v1beta2. That selection happens when a driver built with the v1-based helper runs against a Kubernetes 1.33 API server, and also when an object-level NotFound makes the client fall back from v1 to v1beta2. (#141106, @thc1006) [SIG Node]
Fixed a panic in the Windows kube-proxy (winkernel) that could terminate kube-proxy when
an HNS load balancer without port mappings was enumerated. (#141489, @prince-melvin) [SIG Network and Windows]
Fixed a performance degradation in scheduling queue, when PodGroups are waiting for quorum and large quantity of events arrive to scheduler. (#141577, @macsko) [SIG Scheduling and Testing]
Fixed a race in client-go MutationCache indexed lookups that could temporarily hide a recently created replacement object. (#141736, @jackfrancis) [SIG API Machinery]
Fixed a rare race in the PersistentVolume controller that could leave a PV stuck in the Bound phase and never released after concurrent updates. (#140691, @huww98) [SIG Apps, Scheduling and Storage]
Fixed a regression in 1.38 where resource quantities written by a 1.37 or older apiserver with a decimal exponent outside the int32 range, such as 1e4294967296, could not be decoded, which failed reads of the whole collection. (#142395, @antcybersec) [SIG API Machinery]
Fixed a zero-division overflow/underflow issue in the HPA replica calculator when the replica count is 0. (#140958, @KengoA) [SIG Apps and Autoscaling]
Fixed an issue where HorizontalPodAutoscaler incorrectly included pod overhead in the request calculation for pods using pod-level resources. (#142154, @sophieliu15) [SIG Apps and Autoscaling]
Fixed an issue where a Job recreated with the same name as a recently deleted Job could remain unreconciled because it inherited pending Pod expectations from the previous Job. (#142247, @6democratickim9) [SIG Apps and Testing]
Fixed an issue where a container that was recreated while the kubelet could not reach the API server was reported Ready after a kubelet restart, before its own readiness probe had run. (#141487, @toVersus) [SIG Node]
Fixed an issue where liveness and readiness probes could run before the startup probe succeeded after a container restart. (#141342, @HirazawaUi) [SIG Node and Testing]
Fixed an issue where the in-tree iSCSI detach of a block volume could fail permanently when the /dev/disk/by-path link was already gone (e.g. the iSCSI session was lost before teardown), leaving the volume stuck in node.status.volumesInUse and blocking attach on other nodes. (#141863, @abhinav-phi) [SIG Storage]
Fixed apiserver_watch_cache_events_dispatched_total to not count bookmark events. (#141333, @Jefftree) [SIG API Machinery]
Fixed false fractional-byte warnings for integer resource quantities larger than the int64 range. (#142131, @6democratickim9) [SIG API Machinery]
Fixed kubectl describe service printing "+ 0 more..." after the endpoint list when exactly three ready endpoints are followed by not-ready endpoints. (#142214, @reckless-sherixx) [SIG CLI]
Fixed logic around PlacementFeasible status propagation so that the unschedulable plugins for a pod are correctly stored. PodGroup statuses were improved to be more readable. (#141253, @macsko) [SIG Node, Scheduling and Testing]
Fixed preemption algorithm in kube-scheduler so that it no longer ignores a PodDisruptionBudget with an empty selector. (#141785, @weizhoublue) [SIG Scheduling]
Fixed quantity parsing to reject decimal exponents that are too large to represent as an internal scale, instead of silently narrowing them to an unrelated value (for example 1e4294967297 no longer parses as 1e1). (#141203, @thc1006) [SIG API Machinery]
Fixed the DRA ResourceSlice controller to preserve TimeAdded for every unchanged device taint when updating a ResourceSlice. (#141107, @thc1006) [SIG Node]
Fixed the NodeRestriction admission plugin so the request-serviceaccounts-token-audience authorization check is still consulted when resolving a pod's CSI volume sources fails (e.g. a referenced CSIDriver, PVC, or PV is not found). This restores the authorizer bypass on the error path when the ServiceAccountNodeAudienceRestriction feature is enabled. (#140607, @aramase) [SIG Auth and Testing]
Fixed the kubelet CPU manager so that pods with pod-level (QoS-class) CPU allocations are no longer re-allocated from scratch after a kubelet restart. The checkpointed pod-level CPU set and container assignments are now restored as-is, preventing running pods from being rejected or silently migrated to different CPUs, which previously leaked exclusive CPUs. (#140989, @lukaszwojciechowski) (#141041, @lukaszwojciechowski) [SIG Node and Testing]
Fixed the kubelet rejecting a negative CSI volume expansion size. Sizes whose int64 conversion overflowed were previously accepted because the sign was lost. (#141181, @ashvinctrl) [SIG Node and Storage]
Fixes a panic in kubectl auth reconcile if an error was encountered reconciling a rolebinding or clusterrolebinding (#141035, @liggitt) [SIG Auth and CLI]
Integer resource quantities (such as extended resources) that are whole numbers large enough to overflow an int64 milli projection are no longer incorrectly rejected as "must be an integer". (#141349, @thc1006) [SIG Apps]
Kube-proxy no longer tries to set the nf_conntrack_max sysctl if its
value is already higher than the value kube-proxy wants it to be. This
fixes a regression in 1.36 on systems where kube-proxy does not have
permission to modify the sysctls, since the new maximum value for
nf_conntrack_max could cause kube-proxy to now want a different
(lower) value than it had before. (#142035, @danwinship) [SIG Network]
Kube-scheduler no longer logs the extender TLS private key (tlsConfig.keyData) when logging its component configuration at verbosity >=2. (#140519, @UditDewan) [SIG Scheduling]
Kube-scheduler: a ResourceClaim whose device request counts add up past the int range no longer crashes the scheduler or skips the firstAvailable fallback in the DRA allocator. A request over the per-claim device limit is rejected with an error naming it, and an oversized firstAvailable alternative falls through to a smaller one. (#141307, @thc1006) [SIG Node]
Kubeadm: fixed a preflight check on OpenRC systems that reported a service as enabled whenever rc-update show default mentioned its name anywhere, including as part of another service's name. (#141981, @thc1006) [SIG Cluster Lifecycle]
Kubeadm: no longer persists a node-specific systemd-resolved resolvConf path in the cluster-wide kubelet-config ConfigMap during init. (#141337, @HirazawaUi) [SIG Cluster Lifecycle]
Kubectl describe node: per-pod resource percentages no longer print -9223372036854775808% on nodes that have not reported allocatable resources, and the "Allocated resources" totals now use the same resource accounting as the per-pod rows. (#142215, @reckless-sherixx) [SIG CLI]
Kubectl explain no longer prints the KIND/VERSION header to stdout when the command fails because of an invalid field path; the error is still written to stderr and the exit code is unchanged. (#141534, @ManafovM) [SIG CLI]
Kubectl get --sort-by now treats resources that both lack the selected sort field as equal. (#141613, @DevaanshPathak) [SIG CLI]
Kubectl: kubectl label and kubectl annotate now return an error if modification arguments are passed alongside the --list flag. (#141589, @Roaimkhan) [SIG CLI]
Kubectl: defaults are now matched by full command path instead of the last path segment. (#140230, @Mujib-Ahasan) [SIG CLI]
Kubelet: a Pod with an unknown or empty seccompProfile.type now fails closed, the container fails to start with CreateContainerConfigError instead of silently running as Unconfined. Both values are already rejected by API validation on Pod create and update, and by the kubelet's own validation for static Pods, so no Pod accepted by a supported API server is affected. This only changes behavior for Pods that reached the kubelet without passing API validation, or for a kubelet older than an API server that supports a seccomp profile type the kubelet does not know. The error message for a Localhost profile without localhostProfile no longer ends with a trailing period. (#141958, @saschagrunert) [SIG Auth and Node]
Kubelet: fixed a panic in the DRA device-health goroutine when the on-disk health checkpoint file decoded to a nil map — either the top-level map (null) or a driver's inner Devices map (e.g. {"driverA":{"Devices":null}}). Both cases are now handled on load, and the next health update proceeds normally instead of panicking. (#140904, @bart0sh) [SIG Node]
LimitRanger no longer re-validates the resource requests and limits a pod resize leaves unchanged. Only the values a resize changes are checked against the LimitRange, so an existing pod is no longer rejected on resize by a constraint it already violates. (#142171, @semx) [SIG API Machinery and Node]
LimitRanger no longer rejects an update of a PersistentVolumeClaim because of a request that the update leaves unchanged, even when that request is outside the LimitRange minimum or maximum. A changed, added or removed request is checked against both the minimum and the maximum, as on create, so a stored request outside the range is rejected as soon as an update changes it. (#142170, @thc1006) [SIG API Machinery and Storage]
LimitRanger now compares resource quantities exactly. A LimitRange min, max, or limit/request ratio is enforced for quantities large enough to overflow an int64 projection instead of being silently skipped, and a ratio constraint no longer rejects an equal request and limit above 2^63. Requests and limits within one milli-unit of a bound, such as a request of 0.9999 against a minimum of 1, are now rejected where the previous projection rounded them onto the bound. This applies to pod creation and to every PVC create and update, so a PVC already holding such a value has to have its request corrected before it can be updated. Where the stored request sits above a maximum rather than below a minimum, correcting it would mean decreasing the request, which PVC validation refuses; those claims need the LimitRange widened instead. (#141348, @thc1006) [SIG API Machinery]
Resource.Quantity.Cmp and CmpInt64 now return the correct result when the two quantities' scales differ by more than an int32 can represent. (#142013, @thc1006) [SIG API Machinery]
ResourceQuota updates no longer fail validation on a spec.hard, status.hard or status.used value that the object already holds, so a quota holding a value an older release accepted can still be updated and its status still synced. (#142190, @thc1006) [SIG API Machinery and Apps]
RuntimeClass admission plugin will admit Pods with an empty spec.runtimeClassName. (#140108, @soltysh) [SIG Node]
The EndpointSlice controller no longer retries reconciliation when an EndpointSlice selected for deletion has already been removed. (#141304, @jfremy-openai) [SIG Network]
When a Service and its corresponding Pods are rapidly deleted and recreated, ensure the associated EndpointSlice remains correct and up-to-date. (#141593, @yangjunmyfm192085) [SIG Apps, Network and Testing]
resource.Quantity: for int64-backed quantities whose scales, and for subtraction whose scale alignment, fit the decimal backend's scale range, Neg() of math.MinInt64 and Sub() with a math.MinInt64 subtrahend now promote to the decimal backend instead of overflowing while negating the int64 operand. The exact value is preserved (for example, -math.MinInt64 becomes 2^63) and AsInt64() returns ok=false for the decimal-backed result. Unsupported extreme-scale combinations retain their pre-existing behavior. The unchecked Value(), MilliValue(), and ScaledValue() overflow behavior is unchanged and tracked separately. (#141264, @thc1006) [SIG API Machinery]
Other (Cleanup or Flake)
Apiserver_request_total now counts requests to the aggregated /openapi/v2 endpoint under subresource="openapi/v2". (#140888, @Jefftree) [SIG API Machinery]
Bump etcd SDK to v3.7.1 (#141033, @humblec) [SIG API Machinery, Auth, Cloud Provider, Node and Scheduling]
Kubeadm: added a preflight warning when getent is not found in PATH. kubelet uses it to resolve the kubelet account when setting up user namespace ID mappings. (#141968, @thc1006) [SIG Cluster Lifecycle]
Kubelet is built by default without using CGO across all platforms. If you need CGO support, please use KUBE_CGO_OVERRIDES env variable when building kubernetes. (#135870, @dims) [SIG Architecture, Node and Testing]
Promote apiserver_flowcontrol_priority_level_seat_utilization to BETA (#141161, @LiomSV) [SIG API Machinery and Instrumentation]
Removed Permit extension point implementation from GangScheduling plugin. Similar functionality is already covered by the PlacementFeasible, making the Permit redundant. (#141182, @macsko) [SIG Scheduling]
Removed the KubeletTracing feature gate, which has been GA and locked to default since v1.34. (#141714, @dashpole) [SIG Instrumentation and Node]
Removed the deprecated WindowsHostNetwork feature gate (KEP-3503, withdrawn). The gate no longer guarded any code paths. (#142018, @Adarsh-Me) [SIG Node and Windows]
Removed the generally available feature gate ServiceAccountTokenPodNodeInfo, which was locked and enabled since 1.32. (#135338, @carlory) [SIG Auth, Node and Testing]
Rename the label "cache_to_watcher" in apiserver_watch_events_dispatch_duration_seconds ALPHA metric to "watcher_to_client_handler" (#141175, @richabanker) [SIG API Machinery and Instrumentation]
Updated the default etcd version to 3.7.1 (#141032, @humblec) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Etcd and Testing]
Updates the bundled CoreDNS version to v1.14.7. (#141462, @bzsuni) [SIG Cloud Provider and Cluster Lifecycle]