Releases: kubermatic/kubeone
v1.4.3
Changes by Kind
Bug or Regression
- Add missing VolumeAttachments permissions to machine-controller (#2032, @kubermatic-bot)
- Provide registry configuration to kubeadm when pre-pulling images (#2028, @kron4eg)
Checksums
SHA256 checksums can be found in the kubeone_1.4.3_checksums.txt
file.
v1.4.2
Attention Needed
This patch releases updates etcd to v3.5.3 which includes a fix for the data inconsistency issues reported earlier (https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQc/m/rSgNOzV2BwAJ). To upgrade etcd for an existing cluster, you need to force upgrade the cluster as described here. If you're running Kubernetes 1.22 or newer, we strongly recommend upgrading etcd as soon as possible.
Changes by Kind
Feature
- Domain is not required when using application credentials (#1938, @ahmedwaleedmalik)
Bug or Regression
- Bump flannel image to v0.15.1 (#1993, @ahmedwaleedmalik)
- Deploy etcd v3.5.3 for clusters running Kubernetes 1.22 or newer. etcd v3.5.3 includes a fix for [the data inconsistency issues announced by the etcd maintainers](https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQc/m/rSgNOzV2BwAJ. To upgrade etcd) for an existing cluster, you need to force upgrade the cluster as described here (#1953)
- Fixes containerd upgrade on deb based distros (#1935)
- Show "Ensure MachineDeployments" as an action to be taken only when provisioning a cluster for the first time (#1931)
- Update machine-controller to v1.43.2 (#2001, @kron4eg)
- Fixes an issue where the machine-controller would not wait for the volumeAttachments deletion before deleting the node
- Fixes an issue where masked services on Flatcar are not properly stopped when provisioning a Flatcar node
Checksums
SHA256 checksums can be found in the kubeone_1.4.2_checksums.txt
file.
v1.3.5
Attention Needed
This patch releases updates etcd to v3.5.3 which includes a fix for the data inconsistency issues reported earlier (https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQc/m/rSgNOzV2BwAJ). To upgrade etcd for an existing cluster, you need to force upgrade the cluster as described here. If you're running Kubernetes 1.22 or newer, we strongly recommend upgrading etcd as soon as possible.
Changed
- Upgrade machine-controller to v1.37.3 (#1984)
- This fixes an issue where the machine-controller would not wait for the volumeAttachments deletion before deleting the node.
- Deploy etcd v3.5.3 for clusters running Kubernetes 1.22 or newer. etcd v3.5.3 includes a fix for [the data inconsistency issues announced by the etcd maintainers](https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQc/m/rSgNOzV2BwAJ. To upgrade etcd) for an existing cluster, you need to force upgrade the cluster as described here (#1953)
Checksums
SHA256 checksums can be found in the kubeone_1.3.5_checksums.txt
file.
v1.3.4
Attention Needed
This patch release enables the etcd corruption checks on every etcd member that is running etcd 3.5 (which applies to all Kubernetes 1.22+ clusters). This change is a recommendation from the etcd maintainers due to issues in etcd 3.5 that can cause data consistency issues. The changes in this patch release will prevent corrupted etcd members from joining or staying in the etcd ring.
Changed
- Enable the etcd integrity checks (on startup and every 4 hours) for Kubernetes 1.22+ clusters. See the official etcd announcement for more details. (#1928)
- Validate Kubernetes version against supported versions constraints. The minimum supported version is 1.19, and the maximum supported version is 1.22 (#1817)
- Fix AMI filter in Terraform configs for AWS to always use
x86_64
images (#1692)
Checksums
SHA256 checksums can be found in the kubeone_1.3.4_checksums.txt
file.
v1.4.1
Attention Needed
This patch release enables the etcd corruption checks on every etcd member that is running etcd 3.5 (which applies to all Kubernetes 1.22+ clusters). This change is a recommendation from the etcd maintainers due to issues in etcd 3.5 that can cause data consistency issues. The changes in this patch release will prevent corrupted etcd members from joining or staying in the etcd ring.
Changes by Kind
Bug or Regression
- Regenerate container runtime configurations based on kubeone.yaml during control-plane upgrades on Flatcar Linux nodes, not only on the initial installation. (#1918)
- Approve pending CSRs when upgrading control plane and static worker nodes (#1888)
- Enable the etcd integrity checks (on startup and every 4 hours) for Kubernetes 1.22+ clusters. See the official etcd announcement for more details. (#1909)
- Fix CSR approving issue for existing nodes with already approved and GCed CSRs (#1897)
- Fix missing snapshot CRDs for Openstack CSI (#1913)
- Ensure old machine-controller MutatingWebhookConfiguration is deleted (#1913)
- Fix overwriteRegistry not overwriting the Kubernetes control plane images (#1885)
- Mount /usr/share/ca-certificates to the OpenStack CCM pod to fix the OpenStack CCM pod CrashLooping on Flatcar Linux (#1905)
- Fix the GoBetween script failing to install the zip package on Flatcar Linux (#1905)
- Expand path to SSH private key file (#1859)
- Fix an issue with
kubeone config migrate
failing to migrate configs with thecontainerRuntime
block (#1861)
Checksums
SHA256 checksums can be found in the kubeone_1.4.1_checksums.txt
file.
v1.4.0
KubeOne v1.4.0
Today, we are pleased to announce that KubeOne 1.4 is now generally available. With this release, we introduce our new KubeOneCluster API version with many new features that simplify configuration management. Additionally, we have added support for Kubernetes 1.23 and Cilium CNI and facilitated CCM/CSI migration, among other features. KubeOne 1.4 also provides alpha-level support for Nutanix.
Major Highlights
We recommend checking out the Upgrading from KubeOne 1.3 to 1.4 tutorial, as well as, the changelog for more information about upgrading and the latest features and improvements.
Attention Needed
- KubeOne 1.4.0-beta.0 introduces the new KubeOneCluster v1beta2 API
- The existing KubeOneCluster v1beta1 manifests can be migrated by using the
kubeone config migrate
command - The
kubeone config print
command now uses the new v1beta2 API - The existing KubeOneCluster v1beta1 API is considered deprecated and will be removed in KubeOne 1.6+
- Highlights:
- The API group has been changed from
kubeone.io
tokubeone.k8c.io
- The AssetConfiguration API has been removed from the v1beta2 API. The AssetConfiguration API can still be used with the v1beta1 API, but we highly recommend migrating away because the v1beta1 API is deprecated
- The PodPresets feature has been removed from the v1beta2 API because Kubernetes removed support for PodPresets in Kubernetes 1.20
- Packet (
packet
) cloud provider has been rebranded to Equinix Metal (equinixmetal
). The existing Packet cluster will work withequinixmetal
cloud provider, however, manual migration steps are required if you want to use new Terraform configs for Equinix Metal - A new ContainerRuntime API has been added to the v1beta2 API in order to support configuring mirror registries
- The API group has been changed from
- The existing KubeOneCluster v1beta1 manifests can be migrated by using the
kubeone install
andkubeone upgrade
commands are considered deprecated in favor ofkubeone apply
install
andupgrade
commands will be removed in KubeOne 1.6+- We highly encourage switching to
kubeone apply
. Theapply
command has the same semantics and works in the same way asinstall
/upgrade
, with some additional checks to ensure each requested operation is safe for the cluster
- Unconditionally deploy AWS, AzureDisk, AzureFile, and vSphere CSI drivers if the Kubernetes version is 1.23 or newer (#1831)
- Those providers have the CSI migration enabled by default in Kubernetes 1.23, so the CSI driver will be used for all volumes operations
- Unconditionally deploy DigitalOcean, Hetzner, Nutanix, and OpenStack Cinder CSI drivers (#1831)
- OpenStack has the CSI migration enabled by default since Kubernetes 1.18, so the CSI driver will be used for all operations
- CentOS 8 has reached End-Of-Life (EOL) on January 31st, 2022. It will no longer receive any updates (including security updates). Support for CentOS 8 in KubeOne is deprecated and will be removed in a future release. We strongly recommend migrating to another operating system or RHEL/CentOS distribution as soon as possible.
Breaking changes / Action Required
- The default AMI for CentOS in Terraform configs for AWS has been changed to Rocky Linux. If you use the new Terraform configs with an existing cluster, make sure to bind the AMI as described in the production recommendations document (#1809)
- The
cloud-provider-credentials
Secret is removed by KubeOne because KubeOne does not use it any longer. If you have any workloads NOT created by KubeOne that use this Secret, please migrate before upgrading KubeOne. Instead, KubeOne now createskubeone-machine-controller-credentials
andkubeone-ccm-credentials
Secrets used by machine-controller and external CCM - Support for Amazon EKS-D clusters has been removed starting from this release
- GCP: Default operating system for control plane instances is now Ubuntu 20.04 (#1576)
- Make sure to bind
control_plane_image_family
to the image you're currently using or Terraform might recreate all your control plane instances
- Make sure to bind
- Azure: Default VM type is changed to
Standard_F2
(#1528)- Make sure to bind
control_plane_vm_size
andworker_vm_size
to the VM size you're currently using or Terraform might recreate all your instances
- Make sure to bind
Known Issues
- It's not possible to run kube-proxy in IPVS mode on Kubernetes 1.23 clusters using Canal/Calico CNI. Trying to upgrade existing 1.22 clusters using IPVS to 1.23 will result in a validation error from KubeOne
Checksums
SHA256 checksums can be found in the kubeone_1.4.0_checksums.txt
file.
v1.4.0-rc.1
Attention Needed
- Unconditionally deploy AWS, AzureDisk, AzureFile, and vSphere CSI drivers if the Kubernetes version is 1.23 or newer (#1831)
- Those providers have the CSI migration enabled by default in Kubernetes 1.23, so the CSI driver will be used for all volumes operations
- Unconditionally deploy DigitalOcean, Hetzner, Nutanix, and OpenStack Cinder CSI drivers (#1831)
- OpenStack has the CSI migration enabled by default since Kubernetes 1.18, so the CSI driver will be used for all operations
- [BREAKING] The default AMI for CentOS in Terraform configs for AWS has been changed to Rocky Linux. If you use the new Terraform configs with an existing cluster, make sure to bind the AMI as described in the production recommendations document (#1809)
Added
- Include darwin/arm64 and linux/arm64 builds in release artifacts (#1821)
- Allow providing operating system via the API (#1809)
Changed
General
- Increase the minimum Kubernetes version to 1.20 (#1818)
- Validate the Kubernetes version against supported versions constraints (#1808)
- Allow Docker as a container runtime up to Kubernetes v1.24 (previously up to v1.22) (#1826)
- Unconditionally deploy AWS, AzureDisk, AzureFile, and vSphere CSI drivers if the Kubernetes version is 1.23 or newer (#1831)
- Those providers have the CSI migration enabled by default in Kubernetes 1.23, so the CSI driver will be used for all volumes operations
- Unconditionally deploy DigitalOcean, Hetzner, Nutanix, and OpenStack Cinder CSI drivers (#1831)
- OpenStack has the CSI migration enabled by default since Kubernetes 1.18, so the CSI driver will be used for all operations
Fixed
- Restore missing addons deploy after containerd migration (#1824)
- Select correct CSR to approve (#1813)
Terraform Configs
- [BREAKING] The default AMI for CentOS in Terraform configs for AWS has been changed to Rocky Linux. If you use the new Terraform configs with an existing cluster, make sure to bind the AMI as described in the production recommendations document (#1809)
- Add the
control_plane_vm_count
variable to the AWS configs used to control the number of control plane nodes (defaults to 3) (#1810) - Update the Terraform provider for OpenStack to version 1.47.0 (#1816)
- Set Ubuntu 20.04 as the default image for OpenStack (#1816)
- Add example Terraform configs for Flatcar on vSphere (#1838)
Updated
Checksums
SHA256 checksums can be found in the kubeone_1.4.0-rc.1_checksums.txt
file.
v1.4.0-rc.0
Attention Needed
- CentOS 8 has reached End-Of-Life (EOL) on January 31st, 2022. It will no longer receive any updates (including security updates). Support for CentOS 8 in KubeOne is deprecated and will be removed in a future release. We strongly recommend migrating to another operating system or CentOS distribution as soon as possible.
Added
- Add experimental/alpha-level support for Kubermatic Operating System Manager (OSM) (#1748)
- Add ability to change the container log maximum size (defaults to 100Mi) (#1644)
- Add ability to change the container log maximum files (defaults to 5) (#1759)
- Add the DigitalOcean CSI driver. The CSI driver is deployed automatically if
.cloudProvider.external
is enabled (#1754) - Add the default StorageClass and VolumeSnapshotClass for the DigitalOcean CSI driver. The StorageClass and VolumeSnapshotClass can be deployed by enabling the default-storage-class embedded addon (#1754)
- Generate and approve CSRs for control plane and static workers nodes. Enable the server TLS bootstrap for control plane and static worker nodes (#1750, #1758)
- Source
.cloudProvider.csiConfig
from the credentials file if present (#1739) - Fetch containerd auth config from the credentials file if present (#1745)
Changed
Fixed
- Change baseurl to
vault.centos.org
for CentOS 8 (#1767) - Fix Docker to containerd migration on non-Flatcar operating systems (#1743)
- Fix propagation of proxy config to machines and Kubernetes components (#1746)
Addons
- Replace Hubble static certificate with CronJob generation (#1752)
- Make template function
required
available to addons manifest templates (#1737) - Ensure unattended-upgrades in dpkg is active (#1756)
Terraform Configs
- Create a placement group for control plane nodes in Terraform configs for Hetzner (#1762)
Updated
- Update Canal CNI to v3.22.0 (#1797)
- Update Cilium to v1.11.1 (#1752)
- Update Calico VXLAN addon to v3.22.0 (#1797)
- Update images in order to support Kubernetes 1.23 (#1751, #1753)
- Update AWS External Cloud Controller Manager (CCM) to v1.23.0-alpha.0 for Kubernetes 1.23 clusters
- Update Azure External Cloud Controller Manager (CCM) to v1.23.2 for Kubernetes 1.23 clusters
- Update AWS EBS CSI driver to v1.5.0
- Update AzureFile CSI driver to v1.9.0
- Update AzureDisk CSI driver to v1.10.0
- Update OpenStack External Cloud Controller Manager (CCM) to v1.23.0 for Kubernetes 1.23 clusters
- Update the DigitalOcean External Cloud Controller Manager (CCM) to v0.1.36
- Update the Hetzner External Cloud Controller Manager (CCM) to v1.12.1
- Update machine-controller to v1.42.2 (#1748)
Checksums
SHA256 checksums can be found in the kubeone_1.4.0-rc.0_checksums.txt
file.
v1.4.0-beta.1
Attention Needed
- [BREAKING] The
cloud-provider-credentials
Secret is removed by KubeOne because KubeOne does not use it any longer. If you have any workloads NOT created by KubeOne that use this Secret, please migrate before upgrading KubeOne. Instead, KubeOne now createskubeone-machine-controller-credentials
andkubeone-ccm-credentials
Secrets used by machine-controller and external CCM (#1717, #1718)
Added
- Add experimental/alpha support for Nutanix (#1723, #1725, #1733)
- Support for Nutanix is experimental, so implementation and relevant addons might be changed until it doesn't graduate to beta/stable
- Add the Nutanix CSI driver addon. The addon is deployed manually, on-demand, by enabling the
csi-nutanix
embedded addon (see the PR description for more details and examples) (#1733, #1734) - Add the default StorageClass for the Nutanix CSI driver. The StorageClass can be deployed by enabling the
default-storage-class
embedded addon (see the PR description for more details and examples) (#1733) - Add the Registry Credentials configuration to the RegistryConfiguration API (#1724)
- Add support for different credentials for machine-controller and CCM. Environment variables can be prefixed with
MC_
for machine-controller credentials andCCM_
for CCM credentials (#1717)
Changed
General
- [BREAKING] The
cloud-provider-credentials
Secret is removed by KubeOne because KubeOne does not use it any longer. If you have any workloads NOT created by KubeOne that use this Secret, please migrate before upgrading KubeOne. Instead, KubeOne now createskubeone-machine-controller-credentials
andkubeone-ccm-credentials
Secrets used by machine-controller and external CCM (#1717, #1718)
Fixed
- Fix a bug with the addons applier applying all files when the addons path is not provided (#1733)
Addons
- Fix control plane tolerations in Azure CCM and CSI addons (
node-role.kubernetes.io/master
doesn't have a value) (#1733) - Add node affinity to the cluster-autoscaler addon (#1716)
Terraform Configs
- Remove
centos
choice from the GCE Terraform example configs as it's unsupported (#1712)
Updated
- Update machine-controller to v1.42.0 (#1733)
Checksums
SHA256 checksums can be found in the kubeone_1.4.0-beta.1_checksums.txt
file.
v1.4.0-beta.0
Attention Needed
- KubeOne 1.4.0-beta.0 introduces the new KubeOneCluster v1beta2 API
- The new v1beta2 API is still under-development and might be changed before the KubeOne 1.4.0 release
- We recommend and highly encourage testing the new API, but considering that the API might be changed before the final release, we don't recommend migrating production clusters to the new API yet
- The migration for existing KubeOneCluster manifests is not yet available
- The
kubeone config print
command now uses the new v1beta2 API - The existing KubeOneCluster v1beta1 API is considered as deprecated and will be removed in KubeOne 1.6+
- Highlights:
- The API group has been changed from
kubeone.io
tokubeone.k8c.io
- The AssetConfiguration API has been removed from the v1beta2 API. The AssetConfiguration API can still be used with the v1beta1 API, but we highly recommend migrating away because the v1beta1 API is deprecated
- The PodPresets feature has been removed from the v1beta2 API because Kubernetes removed support for PodPresets in Kubernetes 1.20
- Packet (
packet
) cloud provider has been rebranded to Equinix Metal (equinixmetal
). The existing Packet cluster will work withequinixmetal
cloud provider, however, manual migration steps are required if you want to use new Terraform configs for Equinix Metal - A new ContainerRuntime API has been added to the v1beta2 API in order to support configuring mirror registries. This API is still work-in-progress and will mostly like be extended before the final release
- The API group has been changed from
kubeone install
andkubeone upgrade
commands are considered as deprecated in favor ofkubeone apply
install
andupgrade
commands will be removed in KubeOne 1.6+- We highly encourage switching to
kubeone apply
. Theapply
command has the same semantics and works in the same way asinstall
/upgrade
, with some additional checks to ensure each requested operation is safe for the cluster
- Support for Amazon EKS-D clusters has been removed starting from this release
Known Issues
- It's not possible to run kube-proxy in IPVS mode on Kubernetes 1.23 clusters using Canal/Calico CNI. Trying to upgrade existing 1.22 clusters using IPVS to 1.23 will result in a validation error from KubeOne
- More information about this issue can be found in the following Calico ticket: projectcalico/calico#5011
Added
API
- Add the KubeOneCluster v1beta2 API and change the API group to
kubeone.k8c.io
(#1649)- Make
kubeone config print
command use the newkubeone.k8c.io/v1beta2
API (#1651) - Add the new ContainerRuntime API with support for mirror registries (#1674)
- Addons directory path (
.addons.path
) is not required when using only embedded addons (#1668) - Addons directory path (
.addons.path
) is not defaulted to./addons
any longer (#1668) - Add the KubeletConfig API used to configure
systemReserved
,kubeReserved
, andevictionHard
Kubelet options (#1698) - Remove the PodPresets feature (#1662)
- Remove the AssetConfiguration API (#1699)
- Rebrand Packet (
packet
) to Equinix Metal (equinixmetal
) and support migrating existing Packet clusters to Equinix Metal
clusters (#1663)
- Make
Features
- Add support for Kubernetes 1.23 (#1678)
- Add
kubeone addons list
command used to list available and enabled addons (#1642) - Add support for OpenStack Application Credentials (#1666)
- Add a new
--kubernetes-version
flag to thekubeone config images
command (#1671)- This flag is used to filter images for a particular Kubernetes version. The flag cannot be used along with the KubeOneCluster manifest (
--manifest
flag)
- This flag is used to filter images for a particular Kubernetes version. The flag cannot be used along with the KubeOneCluster manifest (
- Addon parameters can be resolved into environment variable contents if the
env:
prefix is set in the parameter value (#1691)
Changed
General
- Improve installation scripts used to install container runtime (#1664)
Fixed
- Fix issues when disabling nm-cloud-setup on RHEL (#1706)
- cri-tools is now installed automatically as a dependency of kubeadm on Amazon Linux 2. This fixes provisioning issues on Amazon Linux 2 with newer Kubernetes versions. (#1701)
- Fix the image loader script to support KubeOne 1.3+ and Kubernetes 1.22+ (#1671)
- The
kubeone config images
command now shows images for the latest Kubernetes version (instead of for the oldest) (#1671) - Allow pods with the seccomp profile defined to get scheduled if the PodSecurityPolicy (PSP) feature is enabled (#1686)
Addons
- Update the cluster-autoscaler addon to match the upstream manifest (#1713)
Terraform Configs
- Automatically determine GCE zone for the initial MachineDeployment (#1703)
- Fix AMI filter in Terraform configs for AWS to always use
x86_64
images (#1692)
Updated
- Update Cilium CNI addon to v1.11.0 (#1681)
- Update vSphere CSI driver addon to v2.4.0. This change introduces Kubernetes 1.22 support for vSphere clusters (#1675)
- Update Go to 1.17.5 (#1689)
Removed
- Remove support for Amazon EKS-D clusters (#1699)
Checksums
SHA256 checksums can be found in the kubeone_1.4.0-beta.0_checksums.txt
file.