v2.28.0

Announcement

⚠️ This is the last version of RHEL 8 that we support, and it will be deprecated in the next release, see #11872 for a discussion and reasons why.
⚠️ We have removed the Weave CNI test in previous versions and will remove it in the next release because the project has been deprecated.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  Krew installation support is removed (#11824, @VannTen)
• Action required
  You should remove the leading 'v' of all explicit version of components deployed by kubespray (most notably kube_version) (#11890, @VannTen)
• Action required
  etcd_kubeadm_enabled (was deprecated) is removed. You should remove it from your inventory (#11901, @VannTen)
• gateway_api_experimental_channel is deprecated, please use gateway_api_channel and set experimental. (#11763, @tico88612)

Changes by Kind

Feature

• Add Kubernetes 1.32.x hash (#12161, @tmurakam) (#11885, @yankay) (#12003, @mzaian) (#12052, @0ekk)
• Add containerd 2.0.x hash (#11845, @mzaian) (#12011, @mzaian)
• Update runc binary to v1.2.4
  Set containerd_limit_open_file_num to 1048576 so it's configurable. (#11845, @mzaian)
• Update runc binary to v1.2.5 (#12011, @mzaian)
• Make nerdctl 2.0.3 default (#11913, @mzaian)
• Add deploy_coredns: bool (true by default), to let kubespray deploy or not coredns in kube-system (#12218, @ant31)
• Add option ubuntu_stop_unattended_upgrades to stop Ubuntu unattended upgrades (#12174, @0ekk)
• Add support for ranges: (start‑stop or single start) as an additional way to define Cilium LoadBalancer IP pools, alongside the existing cidrs: field. (#12140, @Kimcheolhui)
• Adds the script controb/offline/upload2artifactory.py for offline environments. (#11886, @bbaassssiiee)
• ArgoCD updated to version 2.14.5 to maintain compatibility with Kubernetes version 1.31. (#12041, @farshadasadpour)
• Automatically publish ingress-nginx service address if manual address is not specified and ingress-nginx is not using host network (#11879, @ThisIsQasim)
• Bump node-local-dns (k8s-dns-node-cache) image (#11981, @sathieu)
• Cilium CNI installation replaces Jinja template with Cilium CLI
  cilium_agent_custom_args and cilium_operator_custom_args are deprecated, please use cilium_agent_extra_args and cilium_operator_extra_args.
  cilium_identity_allocation_mode default change to crd.
  cilium_enable_host_legacy_routing default change to false.
  Add CIlium hubble export advanced flow log settings (cilium_hubble_export_file_max_backups, cilium_hubble_export_file_max_size_mb, cilium_hubble_export_dynamic_enabled and cilium_hubble_export_dynamic_config_content)
  Deprecated cilium_ipsec_node_encryption, replace it with cilium_encryption_node_encryption (#12101, @tico88612)
• Default etcd snapshot count to 10000 (#11997, @ErikJiang)
• Enable_dual_stack_networks deprecated, refact network stack with separate ipv4 and ipv6 (#11953, @borislitv)
• Ensure metrics port exists for nodelocaldns/nodelocaldns-second daemonsets (#11998, @Rickkwa)
• Fix cilium network plugin config issue deploying cilium 1.17 (#11986, @pedro-peter)
• For RHEL hosts, checking for subscription status timeout after rh_subscription_check_timeout (default to 3 minutes) (#12115, @VannTen)
• Gateway API can be brought forward before the CNI installation. (#12189, @tico88612)
• Improve ntp package conflict handling (#12212, @ErikJiang)
• Increase the control plane memory requirement to 2GB (#11864, @yankay)
• Network: Fix calico-kube-controller can't list the tiers resources (#12169, @cyclinder)
• Setting up a Docker image service for offline installation on a Mac (#11960, @diguage)
• Support containerd registry mirror certificate configuration (#11857, @KubeKyrie)
• Support kube-proxy nftables mode (#12060, @yankay)
• Terraform upcloud: Add possibility to setup cluster using nodes with no public IPs (#11696, @Xartos)
• Terraform: Added support for UpCloud routers and gateways (#11386, @Xartos)
• The external_cloud_provider support manual option lets users install the cloud controller manager themselves. (#11883, @tico88612)
• Tolerations of cilium-operator deployments can be defined using the cilium_operator_tolerations group_var (#12200, @felipe88alves)
• Update default crio capabilities to allow rancher to start (#11989, @jvkassi)
• Update CI test from AlmaLinux8 to AlmaLinux9 (#11889, @yankay)
• Update kube-vip to v0.8.9 (#11983, @sathieu)
• Upgrade OpenStack Cloud Controller Manager to v1.32.0 (#12121, @tico88612)
• Upgrade ingress-nginx to version v1.12.1 to resolve critical vulnerabilities (CVE-2025-1974 and others) and webhook certgen to v1.5.2. (#12075, @farshadasadpour)
• Upgrade kube-router to 2.1.1 (#12066, @VannTen)
• Upgrade load balancers image version to Nginx 1.27, Haproxy 3.1. (#11928, @guoard)
• Upgrade the default Docker version to 28.0 (#12070, @tico88612)
• Users can now configure hubble-export-file-max-backups and hubble-export-file-max-size-mb through the Kubespray inventory. (#12072, @ErmolenkoMaxim)
• [calico] Update default calico to v3.29.2 (#12012, @mzaian)
• [kubernetes/control-plane] Added support for structured AuthorizationConfiguration files. (#11852, @chadswen)

Documentation

• Fix documentation for offline usage by adding the 'v' prefix in download urls (#12166, @tmurakam)
• Fix path to facts.yml in node facts refresh section (#12177, @guoard)
• Fix sample inventory for the reserved resource (#11895, @anshuman-agarwala)
• No longer reserve outdated cephfs-provisioner installation and documentation (#12113, @tico88612)
• No longer reserve outdated rbd-provisioner installation and documentation (#12114, @tico88612)
• Our CRI-O default capabilities remove NET_RAW and SYS_CHROOT. (#12018, @tico88612)

Failing Test

• Add dns_autoscaler_affinity and remove in-place values. (#12165, @tico88612)
• Fix CI by exclude the .ansible in .ansible-lint
  Remove ctr image pull workaround for nerdctl (#11948, @yankay)

Bug or Regression

• Add support for control plane reconfiguration on upgrades
  Add support for kubeadm-config v1beta4 UpgradeConfiguration.apply and UpgradeConfiguration.node
  Use kubeadm upgrade node during secondary control plane node upgrades (#12015, @chadswen)
• Enable NRI by default on containerd (following containerd defaults) (#12152, @ShinyaIshitobi)
• File download.url's are masked unless the extra var unsafe_show_logs is true. (#11959, @bbaassssiiee)
• Fix a bug where kubeadm_certificate_key was not defined if control plane nodes were not in correct order (#11875, @Xartos)
• Fix a bug where custom TCP/UDP ports were not exposed by the ingress-nginx-controller container and service. (#11850, @commx)
• Fix broken calico Typha template when using both calico_ipam_host_local and typha_secure (#11917, @c-romeo)
• Fix broken dhclient hooks when using resolvconf (#11946, @kyrbrbik)
• Fix control plane pods deletion with proper shell quoting (#11943, @iptizer)
• Fix coredns deployment with coredns_pod_disruption_budget: true or enable_nodelocaldns_secondary (#11952, @RaulButuc)
• Fix hubble-ui deployment to not renders tls volume when the cilium_hubble_tls_generate option not configured. (#12143, @atobaum)
• Fix scale.yml problems with cached IP facts (#12020, @0ekk)
• Fix: Using the ./manage-offline-container-images.sh register command does not create a new container but registers the image in the existing container registry. (#11964, @DearJey)
• Fix: arm64 checksums for youki and kata-containers (#12173, @ErikJiang)
• Fix: missing 'v' prefix in offline image tags (#12086, @ErikJiang)
• Fix: prevent kubeadm to override coredns configuration/deployment on upgrade (#12028, @sathieu)
• Fixed an issue where the second and subsequent parameters in kubelet_cpu_manager_policy_options were ignored due to incorrect indentation. (#12123, @HoKim98)
• Fixed kube-vip to use kube-vip/kube-vip-iptables image instead of kube-vip/kube-vip when lb_fwdmethod or kube_vip_lb_fwdmethod is set to masquerade (#12145, @aviral-agarwal)
• Install symlinks parroting as other control plane nodes etcd certificates (and key) on all control plane nodes, to make kubeadm works (#12181, @VannTen)
• Kubelet-csr-approver moves to regular application installation (#12141, @tico88612)
• New Boolean default variable leave_etc_backup_files: true, set to false for uncluttered /etc directory on target nodes. (#11937, @bbaassssiiee)
• [calico] Fix kubecontrollersconfigurations list permission (#12035, @darkobas2)

Other (Cleanup or Flake)

• Binary checksums are no longer overridable from inventories or host facts (#12234, @VannTen)
• Calico-node pods no longer have a cpu limit by default (#11914, @VannTen)
• Enhance safety and validation mechanisms in the node removal process (#12085, @farshadasadpour)
• Heketi playbook (contrib) is removed. (#12091, @VannTen)
• Kubectl bash completion and alias available for Suse operation systems family (#11860, @noama-nv)
• Kubespray-defaults role is renamed to kubespray_defaults (#12202, @VannTen)
• Remove contrib/kvm-setup and contrib/mitogen. (#12093, @VannTen)
• Rename role bootstrap-os to bootstrap_os (#12203, @VannTen)
• Update KUBESPRAY_VERSION for v2.27.0 (#11854, @yankay)
• Update containerd.options key name (#12170, @flpanbin)
• Upgrade CI for openSuse 15.6 (#12074, @yankay)

Component versions

• kubernetes 1.32.5
• etcd 3.5.16
• docker 28.0
• containerd 2.0.5
• cri-o 1.32.0
• cni-plugins 1.4.1
• calico 3.29.3
• cilium 1.17.3
• flannel 0.22.0
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.1.0
• weave 2.8.7
• kube-vip 0.8.0
• cert-man...

v2.27.0

Urgent Upgrade Notes

No, really, you MUST read this before you upgrade

• Action required
  Change kubeadm_patches format to use an array of inline patch instead of patch files.
  See the example for new format. (#11521, @VannTen)
• Action required
  Removes the generation of static tokens for every node in the cluster when kube_token_auth: true (#11567, @VannTen)
• Action required
  The kubelet_node_{config_extra_args,custom_flags} are removed. Use kubelet_{config_extra_args,custom_flags} in <your_inventory>/group_vars/kube_node.yml.
  The {kube,system}_master_{cpu,memory,ephemeral-storage,pid} are removed. Use the {kube,system}_{cpu,memory,ephemeral-storage,pid} variables in <your_inventory>/group_vars/kube_control_plane.yml. kubelet_custom_flags` can no longer be a string, an array is required. (#10643, @VannTen)
• Action required
  k8s_cluster group is now automatically defined, it can be removed from your inventory if you're not using it for group_vars (#11559, @VannTen)
• Action required
  kubeadm_ignore_preflight_errors is introduced to ignore specific preflight checks from kubeadm. The previous was effectively all, so some errors might surface during upgrade, in which cases, users should add the ones they choose to ignore to that variable. (#11710, @VannTen)

Container-Managers

• Containerd upgrade to 1.7.24 (#11833, @tico88612)
  Add hashes containerd 1.7.23 (#11642, @robertvolkmann)
  Add hashes containerd 1.7.22 default (#11554, @foobaar)
• Runc upgrade to v1.2.3 (#11833, @tico88612)
  Runc add version v1.1.14 (#11574, @janosbabik)

API Change

• If you use CRI-O and want to keep runc as your container default runtime when you upgrade cluster, you must set runc_enable: true and crio_default_runtime: "runc".
  Make CRI-O's default runtime configurable
  CRI-O v1.31 default runtime change to crun
  Crun upgrade to 1.17
  Skopeo upgrade to v1.16.1 (#11601, @tico88612)

Feature

• Make Kubernetes v1.31.4 default
  Add hashes for Kubernetes 1.31.4, 1.30.8 and 1.29.12 (#11828, @tico88612)
  Add hashes for Kubernetes 1.31.3, 1.30.7 and 1.29.11 (#11737, @tico88612)
  Add hashes for Kubernetes 1.31.2, 1.30.6 and 1.29.10 (#11662, @robertvolkmann)
  Add hashes for Kubernetes 1.31.1 and 1.31.0 (#11533, @philipsabri)
  Add hashes for kubernetes 1.29.8, 1.29.9, 1.30.5 (#11581, @DirkTheDaring)
• Add CI for openeuler 24.03
  Add CI Image for openeuler 24.03, 22.03 (#11689, @yankay)
• Add ResourceQuota AdmissionController plugin Configuration (#11814, @chadswen)
• Add a new CRI-O crio_root variable (#11692, @toliger)
• Add external Oracle cloud infrastructure cloud controller manager (#11378, @tico88612)
• Add optional support for Host Firewall and PolicyAuditMode features in Cilium (#11230, @ledroide)
• Add support Fedora 39/40 (#11573, @tico88612)
• Add support to use existing fips with terraform OpenStack (#11558, @anders-elastisys)
• Add the support of network isolation configuration in Multus. (#11605, @Sispheor)
• Added support for using ntpsec (#11665, @davidumea)
• Adds ingress_nginx_service_annotations variable to allow setting annotations for ingress-nginx controller service (#11544, @ThisIsQasim)
• Adds nodelocaldns_additional_configs variable (#11657, @0x4c6565)
• Allow disabling cilium hubble-ui using cilium_enable_hubble_ui variable (#10939, @pedro-peter)
• Allow to skip network configuration by setting kube_network_plugin value to none (#11844, @ant31)
• Configuration can now be supplied to ImagePolicyWebhook and PodNodeSelector admission plugins (#11471, @VannTen)
• Feat(calico): add support for numAllowedLocalASNumbers on bgppeers per node definition (#11570, @mirwan)
• Feat: Kubeadm config API support v1beta4 (#11674, @tico88612)
• Iproute is installed before gathering facts (needed for getting ansible_default_ipv4) (#11816, @0ekk)
• Partial Support of Cilium v1.16+ - kube-proxy replacement var changes
  Add optional support for configuring BGP Control Plane, IP Load Balancer Pools , Legacy BGP Peer Config v1 and BGP Config v2 features in Cilium (#11620, @logicsys)
• [cilium] Make cilium 1.15.9 default (#11593, @foobaar)
• Make cri-dockerd log level configurable (#11646, @mirwan)
• Remove support Fedora 37/38 (#11600, @tico88612)
• Reset operation: remove /var/log/containers and disable service auto-boot, make sure that multi-user.target.wants is deleted. (#11501, @leeonfu)
• Support Configuring EncryptionAlgorithm in Kubeadm v1beta4 (#11757, @ErikJiang)
• Update crictl to version v1.31.1 for Kubernetes 1.31
  Update crictl to version v1.30.1 for Kubernetes 1.30 (#11661, @robertvolkmann)
• Update multus to v4.1.0 (#11434, @ThisIsQasim)
• Upgrade CoreDNS version to v1.11.3 (#11653, @tico88612)
• Upgrade OpenStack Cloud Controller Manager to v1.31.1 (#11738, @tico88612)
• Upgrade pause container to 3.10 (#11695, @tico88612)
• [calico] Update default calico to v3.29.1 (#11798, @mzaian)
• [cert-manager] upgrade to v1.15.3 (#11668, @tico88612)
• [cri-o] Switch binaries to libexecdir
  Update youki version to 0.4.1 to fix ci. (#11584, @yankay)
• [etcd] Default version to 3.5.16 for 1.28, 1.29, 1.30, 1.31 (#11572, @janosbabik)
• [helm] Upgrade to v3.16.4, add 3.16.x checksum (#11832, @tico88612)
• [ingress-nginx] upgrade controller to version 1.12.0 (#11846, @mzaian)
• [need notice] update containerd max_container_log_line_size default value to 16384 (#11585, @KubeKyrie)
• [nerdctl] Default version to 1.7.7 (#11575, @janosbabik)

Documentation

• No longer support in-tree cloud provider, please delete or write external to the cloud_provider variable. (#11633, @tico88612)
• Remove inventory_builder scripts and contrib/dind (#11748, @VannTen)
• Update dns-stack.md reference in docs/ansible/vars.md (#11745, @emmanuel-ferdman)

Failing Test

• Fix CI: elastx cleanup security groups (#11411, @yankay)
• Update rockylinux9 base image to 9.5 (#11744, @yankay)

Bug or Regression

• Action required
  Running kubespray with --limit without cached facts is no longer supported. Improves the scaling for large clusters. (#11598, @VannTen)
• Always copy cert generation script to first etcd to pick up fixes on existing clusters (#11612, @VannTen)
• Fix Cilium agent permission can't read loadbalancerippools and secrets (#11466, @foobaar)
• Fix calico dual stack installation when using ip and ip6. (#11770, @VannTen)
• Fix collection usage for calico and other configuration depending on .sh and .conf files in Kubespray (#11707, @VannTen)
• Fix format of kubeadm-config v1beta4 (#11709, @VannTen)
• Fix kube-vip container securityContext (#11647, @KubeKyrie)
• Fix openEuler system packages installation (#11688, @VannTen)
• Fix pretty-printing (in kubectl) of nodelocaldns and coredns configmap when using dns_upstream_forward_extra_opts with an empty value option. (#11694, @VannTen)
• Fix spurious failure with 'localhost' when using scale.yml --limit <some nodes> (#11817, @VannTen)
• Fix task naming in bootstrap-os (#11714, @ErikJiang)
• Fix terraform.py on python >=3.12 (#11773, @enrico9034)
• Fix the check for cached data when using --limit (#11693, @VannTen)
• Fix the usage of --limit when using legacy groups (#11577, @VannTen)
• Fix usage of admission plugins configuration. (#11779, @VannTen)
• Fix using the default network manager in reset.yml (#11678, @KubeKyrie)
• Fix: cannot stop & remove all cri containers via remove_node.yml (#11631, @tico88612)
• Fixed: VSphere CSI and CPI drivers and are now retrieved from registry.k8s.io instead of gcr.io, as they have been deleted from the latter. Only a few recent versions are available in the new repository; if you have pinned vsphere_csi_controller, vsphere_csi_driver_image_tag or vsphere_syncer_image_tag to a version older than v3.1.2, please check if that version is available from the new repository. The same goes for external_vsphere_cloud_controller_image_tag which can no longer be latest, and should align with the running version of Kubernetes. It now defaults to v1.31.0. (#11564, @luringens)
• HA etcd cluster keeps quorum during upgrades. (#11677, @VannTen)
• Kubeadm images (kube-controller-manager,kube-scheduler,kube-apiserver,kube-proxy) are properly downloaded, including when using the download cache. (#11741, @VannTen)
• Make sure kubespray-defaults can be executed successfully by executing bootstrap-os first (#11441, @huangkevin404)
• Make upcloud csi_driver use the correct pull secret (#11597, @VannTen)
• Modifies Helm parameters wait and atomic to be set to false when using kube_network_plugin=cni to prevent deployment issues with kubelet-csr-approver. (#11704, @M-JavadHeydarpour)
• Remove invalid extraArgs entry and update template file reference (#11703, @agravgaard)
• Update calico-nopde template and remove flexvol-driver initContainer (#11634, @KubeKyrie)
• Use correct version for community.general collection (#11724, @VannTen)

Other (Cleanup or Flake)

• Cleanup older terminology, replace "master" with "control plane" (#11394, @bogd)
• Drop support for Kubernetes 1.28.x minimum version now is 1.29.x
  Drop support for CRI-O 1.28.x minimum version now is 1.29.x (#11609, @yankay)
• Fix roles/download/tasks/download_file.yml task name typo (#11684, @dmncmn)
• Optimize CA cert hash calculation with community.crypto (#11758, @ErikJiang)
• Remove pip install . support and rpm spec file (#11760, @VannTen)
• Replace deprecated unarchive.copy with unarchive.remote_src (#11207, @Payback159)
• Update KUBESPRAY_VERSION for v2.26.0 (#11511, @yankay)
• containerd_use_config_path is removed as kubespray now always use containerd config_path configuration. (#11755, @VannTen)

v2.25.1

Changes by Kind

Deprecation / Removal

• Deprecating support for Centos7 and Weave ; they are not tested anymore (#11428, @k8s-infra-cherrypick-robot)
• Remove Debian 10 support. (#11429, @k8s-infra-cherrypick-robot)ra-cherrypick-robot)

Feature

• Default to Kubernetes v1.29.10 (#11666, @enneitex)
• Add hashes for Kubernetes v1.29.9 (#11599, @robertvolkmann), Add hashes for Kubernetes v1.29.7 (#11408, @mzaian), Add hashes for Kubernetes v1.29.6 (#11355, @tico88612)
• Default to etcd v3.5.16 (#11576, @janosbabik)
• [ingress-nginx] upgrade controller to version 1.11.2 (#11464, @k8s-infra-cherrypick-robot)
• Allow to run kubespray with an empty kube_node group, to provision only the control plane (#11249, @k8s-infra-cherrypick-robot)
• CentOS 7 yum repo baseurl update (#11365, @k8s-infra-cherrypick-robot)
• Check CentOS-Base.repo exists for CentOS 7 (#11438, @k8s-infra-cherrypick-robot)
• User has a possibility to fix nodePort of ingress-nginx service with property in addons.yaml (#11339, @k8s-infra-cherrypick-robot)

Applications

• [helm] Upgrade to v3.14.4 (#11599, @robertvolkmann)

Network

• [calico] Update default calico to v3.27.4
  [calico] Fix high cpu load due to XDP program in iptables (#11476, @mzaian)

Container-Managers

• [containerd] Default to v1.7.22
  [nerdctl] Upgrade to 1.7.7
  [runc] Upgrade to v1.1.14 (#11576, @janosbabik)

Bug or Regression

• Always copy cert generation script to first etcd to pick up fixes on existing clusters (#11616, @k8s-infra-cherrypick-robot)
• Ingress-nginx-controller admission service is automatically created when ingress_nginx_webhook_enabled: true (#11332, @k8s-infra-cherrypick-robot)
• Fix: cannot stop & remove all cri containers via remove_node.yml (#11638, @k8s-infra-cherrypick-robot)

v2.24.3

Changes by Kind

API Change

• Default to kubernetes v1.28.14
  Default to etcd v3.5.16
  Default to containerd v1.7.22
  Default to cri-o v1.28.10
  Default to nerdctl 1.7.7
  Default to runc v1.1.14 (#11516, @VannTen)

Feature

• User has a possibility to fix nodePort of ingress-nginx service with property in addons.yaml (#11361, @k8s-infra-cherrypick-robot)
• Check CentOS-Base.repo exists for CentOS 7 (#11546, @tico88612)

Other (Cleanup or Flake)

• Update KUBESPRAY_VERSION in galaxy.yml to v2.24.3 and Update Readme.md to v2.24.1 (#11385, @yankay)

v2.26.0

Deprecation / Removal

• Deprecating support for Centos7; they are not tested anymore (#11344, @ant31)
• Remove Debian 10 support. (#11347, @tico88612)
• Remove the kubeadm_version which is always equal to kube_version (#11473, @VannTen)
• Drop support for Kubernetes 1.27.x minimum version now is 1.28.x (#11221, @mzaian)
• if you were previously only setting serializeImagePulls: false to have unlimited parallel pulls, you will need to set kubelet_max_parallel_images_pulls to a suitable value instead (#11094, @tu1h)

Feature / Major Changes

• Make kubernetes v1.30.4 default (#11455, @kokyhm)
• Add hashes for Kubernetes v1.30.3 default (#11391, @tico88612), Add hashes for Kubernetes v1.30.2 default (#11343, @tmurakam), Add hashes for Kubernetes 1.30.0, 1.30.1 and 1.30.2 (#11261, @tmurakam), Add hashes for kubernetes 1.29.7, 1.28.[11-12] (#11407, @mzaian)
• Add option ubuntu_kernel_unattended_upgrades_disabled to control unattended-upgrades for Linux kernel and all packages start with linux- on Ubuntu (#11296, @tu1h)
• Added option to configure dependencies for kubelet.service (#11297, @ledroide)
• Adds the possibility to add extra arguments to the various containers in the cinder-csi plugin.(#11169, @Payback159)
• Allow to run kubespray with an empty kube_node group, to provision only the control plane (#11248, @VannTen)
• CentOS 7 yum repo baseurl update (#11360, @tico88612)
• Check CentOS-Base.repo exists for CentOS 7 (#11402, @tu1h)
• Check if peers is defined when peering with routers (#11259, @ehsan310)
• OpenStack Cloud Controller Manager upgrade to 1.30.0 (#11358, @tico88612)
• Rename systemd module to systemd_service (#11396, @tu1h)
• User has the ability to configure calico-kube-controllers log level (#11335, @mirwan)
• User has the ability to configure local_volume_provisioner log level (#11336, @mirwan)
• User has the ability to configure netchecker components log levels (#11334, @mirwan)
• You can now disable installing OS dependencies using system's package manager by skipping system-packages tag. (#10872, @hedayat)
• kubelet_max_parallel_image_pulls represents the maximum number of image pulls in parallel (#11094, @tu1h)
• Update reset task to support Tencent OS (reset_restart_network_service_name) (#11459, @KubeKyrie)
• Add conditional checking on ubuntu kernel unattended_upgrades disabling (#11479, @tu1h)

Applications

• Bump Cinder CSI Plugin to v1.30.0 (#11374, @tico88612)
• Bump upcloud csi driver to v1.1.0 in order to enable csi volume snapshots. (#11303, @Elias-elastisys)
• User has a possibility to fix nodePort of ingress-nginx service with property in addons.yaml (#11310, @mochizuki875)
• Update kube-vip to v0.8.0 (#11156, @jisnardo)
• [cert-manager] upgrade to v1.14.7 (#11341, @tico88612)
• [cert-manager] add support v1.13.6 (#11279, @tico88612)
• [ingress-nginx] upgrade controller to version 1.11.2 (#11463, @mzaian)
• [helm] Upgrade to v3.15.4, add 3.15.x, and drop 3.13.x (#11486, @yankay)
• Add support for LB in UpCloud private zone (#11260, @davidumea)
• Bump UpCloud terraform module to v5.6.0
  UpCloud servers specify server groups to be apart of, eliminates manual rescheduling. (#11311, @robinAwallace)
• Update node-feature-discovery to v0.16.4 (#11250, @mzaian)
• Allow for configuring etcd progress notify interval and default set to 5s (#11499, @liuxu623)
• Support Gateway API CRDs install (#11376, @tico88612)
• Increase ansible timeout to 300 (#11354, @rptaylor)

Network

• [calico] Change calico default version to v3.28.1, add v3.28.0 and checksum , Update calico apiserver deployment to use new readiness probe (#11234, @ehsan310)
• [calico] add calico support v3.27.4 to fix high cpu load due to XDP program in iptables (#11476, @ehsan310)
• Add cilium_hubble_event_buffer_capacity & cilium_hubble_event_queue_size vars (#10943, @pedro-peter)
• [network] bump cni version to v1.4.0 (#10698, @cyclinder)
• Change weave CNI to community version and upgrade to the latest version (2.8.7) (#11228, @tico88612)
• [kube-ovn] update to v1.12.21 (#11445, @oilbeater)

Container-Managers

• [containerd] Make containerd 1.7.21 default (#11478, @yankay)
• [containerd] added debug config variables (#11080, @spnngl)
• [containerd] fixes wrong templating for tracing config (#11372, @ugur99)
  [runc] Upgrade to v1.1.13 (#11413, @mzaian)
• Update docker cli version 26.1.2 (#11291, @ErikJiang)

Documentation

• Update RELEASE.md for upgrading k8s (#11321, @yankay)

Bug or Regression

• Delete /etc/NetworkManager/conf.d/dns.conf on reset. (#11440, @HoKim98)
• Fix Hetzner kubernetes group names (#11232, @jmaccabee13)
• Fix: skip multus when not defined (#10934, @darkobas2)
• Ingress-nginx-controller admission service is automatically created when ingress_nginx_webhook_enabled: true (#11309, @mochizuki875)
• Provide missing advertise-address flag to kube-apiserver (#11387, @derselbst)
• Update reset task to support Kylin OS (reset_restart_network_service_name) (#11406, @KubeKyrie)
• Updated indentation in cni-kube-ovn.yml.j2 (L658) (#11357, @sanshah1211)
• Fix CI with fail docker pull in gitlab runner by change DOCKER_HOST (#11315, @yankay)
• Fix etcd not starting up when using a custom access address (#11388, @derselbst)
• Fix the Auto Bump PR is blocked by the label do-not-merge/release-note-label-needed by adding dependabot release-note-none label. (#11256, @yankay)
• Fix kube_reserved so it only controls kubeReservedCgroup . (#11367, @rptaylor)
• Disables reconfiguring the cluster during upgrade (remove --config option from kubeadm upgrade apply) (#11352, @tmurakam)
• Fix error in boostrap-os when git does not handle symlinks (#11508, @VannTen)
• Fix static kube-apiserver advertise address based on first control plane (#11457, @Seljuke)
• Fix incorrect member matching when removing etcd nodes (#11488, @ErikJiang)
• Fix double pop of access_ip (#11435, @rptaylor)
• Fix use super-admin.conf for kube-vip on first master when it exists to support initial k8s v1.29+ installation with kube-vip enabled (#11422, @Seljuke)

Other (Cleanup or Flake)

• Contrib playbooks are no longer included in the ansible kubespray collection (#11239, @VannTen)
• Reduced required python packages in requirements.txt (#11199, @itayporezky)
• Fix openstack cleanup by change the delete security_group order (#11299, @yankay)
• RHEL 7, Centos 7 and derivatives are no longer supported. (#11246, @VannTen)
• Use TasksMask=infinity on ostree systems for docker systemd service (#11493, @VannTen)

Supported Components

• Core
  • kubernetes v1.30.4
  • etcd v3.5.12
  • docker v26.1
  • containerd v1.7.21
  • cri-o v1.30.3 (experimental: see CRI-O Note. Only on fedora, ubuntu and centos based OS)
• Network Plugin
  • cni-plugins v1.2.0
  • calico v3.28.1
  • cilium v1.15.4
  • flannel v0.22.0
  • kube-ovn v1.12.21
  • kube-router v2.0.0
  • multus v3.8
  • weave v2.8.7
  • kube-vip v0.8.0
• Application
  • cert-manager v1.14.7
  • coredns v1.11.1
  • ingress-nginx v1.11.2
  • krew v0.4.4
  • argocd v2.11.0
  • helm v3.15.4
  • metallb v0.13.9
  • registry v2.8.1
• Storage Plugin
  • cephfs-provisioner v2.1.0-k8s1.11
  • rbd-provisioner v2.1.1-k8s1.11
  • aws-ebs-csi-plugin v0.5.0
  • azure-csi-plugin v1.10.0
  • cinder-csi-plugin v1.30.0
  • gcp-pd-csi-plugin v1.9.2
  • local-path-provisioner v0.0.24
  • local-volume-provisioner v2.5.0
  • node-feature-discovery v0.16.4

Known issues

• Upgrade of clusters with external etcd can be problematic (in particular long lived clusters, as this is not reproducible on cluster created by v2.25.1) ; see #11500 (comment) and the previous discussion for details and a workaround

Notes

• Deprecating support for Centos7
• The Ansible version has been upgrade to 9.8.0
• Change weave CNI to community version https://github.com/weaveworks/weave

Maintainers

Great respect for joining maintainers 🎉

• Adding myself (VannTen) as approver #11483 @VannTen
• Add tico88612 as reviewer #11453 @tico88612
• owners: move ant31 from emeritus_approvers to approvers #11247 @ant31

v2.24.2

Changes by Kind

Feature

• Make kubernetes v1.28.10 default (#11269, @mzaian)
• Revert 'Support CoreDNS use host network & config CoreDNS port' (#10617, @liuxu623)
• User has a possibility to modify Service type with "ingress_nginx_service_type" property in addons. (#11330, @mochizuki875)

Bug or Regression

• Ingress-nginx-controller admission service is automatically created when ingress_nginx_webhook_enabled: true (#11331, @mochizuki875)
• Fix CentOS 7 yum repo baseurl update (#11364, @tico88612 )

Other (Cleanup or Flake)

• Remove the archived debian apt repository when installing docker-engine (#11215, @VannTen )
• Update KUBESPRAY_VERSION in galaxy.yml v2.24.1 (#10961, @yankay)

v2.25.0

Deprecation / Removal

• Remove support for Kubernetes 1.26.x (move min version to 1.27.x) (#10817, @KubeKyrie)
• Remove documentation for removed in-tree openstack provider (#10889, @LarssonOliver)

Feature / Major Changes

• A check is introduced to fail the playbook if cgroups are not enabled on the node (#11165, @franznemeth)
• Add Calico v3.27.3 and make it default (#11141, @pomland-94)
• Add extra_vars support to vagrant setup (#10932, @VannTen)
• Add kube-vip LeaderElection variables vip_leaseduration, vip_renewdeadline, vip_retryperiod options for kube-vip (#11021, @KubeKyrie)
• Add new option remove_anonymous_access to prevent granting RBAC permissions to anonymous users. (#11016, @nicolas-goudry)
• Add scheduler plugins support (scheduler_plugins_enabled enable or disable the installation scheduler plugins / scheduler_plugins_enabled_plugins describe the enabled plugins / scheduler_plugins_diabled_plugins describe the disabled plugins / scheduler_plugins_plugin_config set the custom config for enabled plugins) (#10747, @tu1h)
• Added a config option to filter ntp interfaces (#11066, @Pavan-Gunda)
• Adding egress IPv6 for node-local-dns queries (k8s_allowed_egress_ipv6_ips) (#10396, @raviranjanelastisys)
• Bump docker version for kylin linux (#11203, @ErikJiang)
• Bump docker version for openeuler linux (#11206, @ErikJiang)
• Update almalinux-8 base image to 8.9 (#10918, @VannTen)
• Bumping checksums and various versions (#10999, @MrFreezeex)
• Containerd: allow to configure fallback server (#10988, @sathieu)
• Docker upgrade from 24.0 to 26.1 (#11198, @tico88612)
• Download hash script: auto discover versions (#10849, @VannTen)
• Enable configuring mountOptions, reclaimPolicy and volumeBindingMode for cinder-csi StorageClasses. (#10450, @Payback159)
• Make containerd v1.7.15 default (#11083, @Payback159)
• Make kubernetes v1.28.6 default (#10810, @mzaian)
• Make kubernetes v1.29.1 default
  Remove SecCompDefault feature gate from hardening configuration for kubernetes 1.29 (#10820, @tmurakam)
• Make kubernetes v1.29.2 default (#10919, @mzaian)
• Make kubernetes v1.29.3 default (#11035, @mzaian)
• Make kubernetes v1.29.4 default (#11108, @mzaian)
• Make kubernetes v1.29.5 default (#11196, @mzaian)
• Metallb: added metallb_namespace variable to parameterize namespace (#11136, @oik741)
• OpenStack Cloud Controller Manager upgrade to 1.28.2 (#11174, @tico88612)
• Opensuse deployment is now tested in CI. (#11159, @VannTen)
• Add selinux-ng repo in Amazon Linux to install container-selinux (#11182, @yankay)
• Add CI Image for Ubuntu 24.04 (#11167, @yankay)
• Allows .vagrant folder location to be configured (#10718, @kri5)
• Prevent nodelocaldns to be OOM-killed (#11056, @sathieu)
• Support Node Feature Discovery (#10861, @yankay)
• Support Ubuntu 24.04 (#11132, @tico88612)
• Support following k8s version selection pause image (#10756, @my-git9)
• The variable old_dns_domains (list) can be used for backward compatibility when changing dns_domain (#10630, @VannTen)
• Update external huawei cloud controller to 0.26.6 (#10824, @dabeck)
• Update external huawei cloud controller to 0.26.8 (#11172, @dabeck)
• Update kube-vip to v0.8.0 (#11156, @jisnardo)
• Update metrics server to v0.7.0 (#10856, @mzaian)
• Updated ingress controller version to 1.9.6 (#10868, @kundan2707)
• User has a possibility to modify Service type with "ingress_nginx_service_type" property in addons. (#10925, @chrxmvtik)
• [Terraform-openstack] Added possibility to build an octavia loadbalancer for the Kubernetes Api. (#10924, @jaszil)
• [containerd] added distributed tracing config variables for containerd (containerd_tracing_enabled,containerd_tracing_endpoint,containerd_tracing_protocol, containerd_tracing_sampling_ratio,containerd_tracing_service_name ); it is disabled by default. (#11103, @ugur99)
• [etcd] Default version to 3.5.12 for k8s 1.27 , 1.28 , 1.29 (#11036, @mzaian)
• Minimum ansible-core version is now 2.16.4 (#10984, @VannTen)
• Remove the archived debian apt repository when installing docker-engine (#11088, @yankay)
• Change dependbot interval to weekly (#11189, @yankay)
• Allow specifying CPU Manager Policy options through kubelet_cpu_manager_policy_options (#11023, @derselbst)
• [kube-apiserver] added distributed tracing config variables for kube-apiserver (kube_apiserver_tracing,kube_apiserver_tracing_endpoint,kube_apiserver_tracing_sampling_rate_per_million); it is disabled by default.
  [kubelet] added distributed tracing config variables for kubelet (kubelet_tracing,kubelet_tracing_endpoint,kubelet_tracing_sampling_rate_per_million); it is disabled by default. (#10795, @ugur99)

Applications

• [argocd] update argocd to v2.11.0 (#11193, @mzaian)
• [helm] Upgrade to v3.14.2 (#10967, @cleman95)
• Bump coredns version to 1.11.1 (#10719, @batazor)
• Support CoreDNS use host network & config CoreDNS port (#10617, @liuxu623)
• Fix secondary coredns missing var (#10821, @VannTen)
• Revert "support CoreDNS use host network and config dns port (#10617)" (#11185, @VannTen)
• dns_mode: coredns_dual is now tested in CI. (#10903, @VannTen)

Network

• Adds support for cilium v1.15
  • Adds support for cilium_l2announcements to replace metallb with cilium l2 announcements, defaults to false
  • Adds support for cilium_loadbalancer_mode to switch bpf-lb-mode between snat, dsr or hybrid, default to snat (#11106, @deveshk0)
• Adds the option to install calico 3.27.3 (#11059, @danielfrg)
• [calico] Update default calico to v3.27.2 (#10960, @mzaian)

Container-Managers

• crictl stop container grace period, cri_stop_containers_grace_period: 0 (#10651, @krembu)
• Update the docker default version to 24.0 (#10873, @yankay)
• [Containerd] Enable by default discard_unpacked_layers to save some space (see containerd/containerd#6295) (#10905, @VannTen)
• [Nerdctl] Upgrade to version 1.7.4 (#10968, @cleman95)
• [containerd] Make containerd 1.7.13 default
  [runc] Upgrade to v1.1.12 (#10862, @KubeKyrie)
• [containerd] Make containerd 1.7.16 default (#11142, @mzaian)

API Change

• Make proxy protocol in Upcloud LB configurable (#10971, @davidumea)

Design

• Merge stop and remove systemd service task in reset/tasks/main.yml (#10902, @kimsehwan96)

Documentation

• Add documentation for configuring nat outgoing ipv6 (#10866, @anders-elastisys)
• Add new OpenStack Cloud for terraform (#10910, @DragomirAlin)
• BREAKING CHANGE: This script is introduced to facilitate living documentation and its administration. This leads to a restructuring in the documentation at https://kubespray.io/#/ to simplify the automatic creation of links, as the structure in the sidebar changes. (#11128, @Payback159)
• Change a task name Ensure kube-bench parameters are set into Ensure kubelet expected parameters are set in roles/kubernetes/preinstall/tasks/0080-system-configurations.yml for a clearer understanding of its operation (#11171, @kimsehwan96)
• Do not disable SELinux surreptitiously (#10920, @rptaylor)
• Doc clarification: skipping patches releases is OK (#10850, @VannTen)
• Docs: vagrant-libvirt is tested in CI (#10847, @VannTen)
• Explicit private/public nature of *ip vars (#10904, @VannTen)
• Fix typo in vagrant.md (#10836, @kundan2707)
• Fix typo mistake in roles/kubernetes/control-plane/tasks/define-first-kube-control.yml (#10835, @kimsehwan96)
• Fixed typos in inventory/sample/group_vars/k8s_cluster (#10911, @arahmangulov)
• Kubespray used as a collection will have the correct collection version (#10727, @VannTen)
• Make large-deployments.md link to downloads.md (#10840, @spantaleev)
• Removed not needed graduated feature gates. (#10448, @Smidra)
• Update upgrades.md with serial=1 for rolling updates (#10837, @titansmc)
• Variable cilium_ipsec_key must be base64 encoded (#10781, @ledroide)

Bug or Regression

• Added an optional variable (cni_bin_owner) to allow the user to set a different owner for /opt/cni/bin/ and it's contents. (#10929, @Rickkwa)
• Change the position of the containerd_extra_args parameter to enhance its universality. (#11013, @qcu266)
• Configure crio container runtime to use kube reserved cgroup (#11028, @pedro-peter)
• Don't overwrite changes to openstack allowed_address_pairs #10760 (#10760, @rptaylor)
• Download cache directory permissions are no longer reset recursively (#10900, @VannTen)
• Fix ClusterRole for Calico >=v1.26.x with Calico API Server installed (#11089, @RaSerge)
• Fix ansible parameter ssh_args in ansible.cfg file not work (#10981, @joy717)
• Fix boostrap for Amazon Linux (#11139, @VannTen)
• Fix crio registries config file when using slashes in the registry path (#11030, @pedro-peter)
• Fix file loss during download (#10779, @ErikJiang)
• Fix kubespray-defaults: Check for boostrap-os FQCN (#11073, @KubeKyrie)
• Fix local path provisioner image repo in sample inventory. (#11180, @tico88612)
• Fix logical error when checking for boostrap-os (#10867, @VannTen)
• Fix lsattr command error when kubelet has symbolic link (#11074, @KubeKyrie)
• Fix network manage service of Debian 12 (#11058, @KubeKyrie)
• Fix nginx controller leader election RBAC (#10913, @VannTen)
• Fix python regex matching problem when finding docker packages (#11075, @KubeKyrie)
• Fix waiting for MetalLB controller (#10858, @flxbwr)
• Fix(kubernetes): taint nodes on cluster upgrade (#10705, @maxime1907)
• Fix: config hostname as string type in kubeadmConfig rendering (#10997, @ErikJiang)
• Fixes running recover-control-plane.yml with offline broken etcd nodes. (#10660, @yuha0)
• Revert OCCM standard dnsPolicy to ClusterFirst to fix #10914 which was introduced with #10618 and make dnsPolicy configurable to furthermore support #10618 (#11168, @Payback159)
• Force update helm repo if ...

v2.24.1

Changes by Kind

Feature

• Make kubernetes v1.28.6 default (#10810, @mzaian)

Bug or Regression

• Add configuration to create cilium CNI plugin file when cilium>=1.14.0 (#10945, @cleman95 )
• Fix logical error when checking for boostrap-os (#10953, @VannTen)
• Make containerd 1.7.13 default
  Make runc 1.1.12 default
  Patch GHSA-xr7r-f8xq-vfvv (#10877, @VannTen)

Other (Cleanup or Flake)

• Bump galaxy version before release (#10890, @VannTen)

---

The release intend to address GHSA-xr7r-f8xq-vfvv

v2.22.2

Changes by Kind

Network

• [calico] Use calico_pool_blocksize from cluster when existing (#10516, @VannTen)

API Change

• Make kubernetes 1.26.11 default (#10704, @VannTen)

Feature

• Add hashes for kubernetes version 1.26.6, 1.26.7, 1.26.8 & 1.26.9 (#10444, @bozzo)
• Don't let find search filesystem mounts in docker build run step (#10131, @tomodachi)
• Make kubernetes 1.26.13 the default version (#10823, @VannTen)

Failing Test

• Bump vagrant version 2.3.7 (#10789, @yankay)

Bug or Regression

• Fix hardcoded pod infra version (#10805, @ErikJiang)
• Make containerd 1.7.13 default
  Make runc 1.1.12 default
  Patch GHSA-xr7r-f8xq-vfvv (#10878, @VannTen)
• [Multus] Fix loop_control template error when item is None (#10347, @nicolas-goudry)

---

The release intend to address GHSA-xr7r-f8xq-vfvv

v2.23.3

Changes by Kind

Feature

• Update kubernetes default version to 1.27.10 (#10876, @VannTen)

Bug or Regression

• Fix hardcoded pod infra version (#10806, @ErikJiang)
• Make containerd 1.7.13 default
  Make runc 1.1.12 default
  Make kubernetes 1.27.10 default
  Patch GHSA-xr7r-f8xq-vfvv (#10876, @VannTen)

Other (Cleanup or Flake)

• Update KUBESPRAY_VERSION in galaxy.yml and Readme for v2.23.2 (#10801, @yankay)

---

The release intend to address GHSA-xr7r-f8xq-vfvv