diff --git a/content/en/blog/_posts/2023-12-14-disabling-in-tree-cloud-provider-goes-beta.md b/content/en/blog/_posts/2023-12-14-disabling-in-tree-cloud-provider-goes-beta.md new file mode 100644 index 0000000000000..638e29139017c --- /dev/null +++ b/content/en/blog/_posts/2023-12-14-disabling-in-tree-cloud-provider-goes-beta.md @@ -0,0 +1,217 @@ +--- +layout: blog +title: "Kubernetes 1.29: Cloud Provider Integrations Are Now Separate Components" +date: 2023-12-14T09:30:00-08:00 +slug: cloud-provider-integration-changes +--- + +**Authors:** Michael McCune (Red Hat), Andrew Sy Kim (Google) + +For Kubernetes v1.29, you need to use additional components to integrate your +Kubernetes cluster with a cloud infrastructure provider. By default, Kubernetes +v1.29 disallows components configured with legacy compiled-in cloud provider +integrations. + +In 2018, the [Kubernetes community agreed to form the Cloud Provider Special +Interest Group (SIG)][oldblog], with a mission to externalize all cloud provider +integrations and remove all the existing in-tree cloud provider integrations. +In January 2019, the Kubernetes community approved the initial draft of +[KEP-2395: Removing In-Tree Cloud Provider Code][kep2395]. This KEP defines a +process by which we can remove cloud provider specific code from the core +Kubernetes source tree. From the KEP: + +> Motiviation [sic] behind this effort is to allow cloud providers to develop and +> make releases independent from the core Kubernetes release cycle. The +> de-coupling of cloud provider code allows for separation of concern between +> "Kubernetes core" and the cloud providers within the ecosystem. In addition, +> this ensures all cloud providers in the ecosystem are integrating with +> Kubernetes in a consistent and extendable way. + +After many years of development and collaboration across many contributors, +the default behavior for legacy cloud provider integrations is changing. +This means that users will need to confirm their Kubernetes configurations, +and in some cases run external cloud controller managers. These changes are +taking effect in Kubernetes version 1.29; read on to learn if you are affected +and what changes you will need to make. + +These updated default settings affect a large proportion of Kubernetes users, +and **will require changes** for users who were previously using the in-tree +provider integrations. The legacy integrations offered compatibility with +Azure, AWS, GCE, OpenStack, and vSphere; however for AWS and OpenStack the +compiled-in integrations were removed in Kubernetes versions 1.26 and 1.27, +respectively. + +## What has changed? + +At the most basic level, two [feature gates][fg] are changing their default +value from false to true. Those feature gates, `DisableCloudProviders` and +`DisableKubeletCloudCredentialProviders`, control the way that the +[kube-apiserver][kapi], [kube-controller-manager][kcm], and [kubelet][kubelet] +invoke the cloud provider related code that is included in those components. +When these feature gates are true (the default), the only recognized value for +the `--cloud-provider` command line argument is `external`. + +Let's see what the [official Kubernetes documentation][fg] says about these +feature gates: + +> `DisableCloudProviders`: Disables any functionality in `kube-apiserver`, +> `kube-controller-manager` and `kubelet` related to the `--cloud-provider` +> component flag. + +> `DisableKubeletCloudCredentialProviders`: Disable the in-tree functionality +> in kubelet to authenticate to a cloud provider container registry for image +> pull credentials. + +The next stage beyond beta will be full removal; for that release onwards, you +won't be able to override those feature gates back to false. + +## What do you need to do? + +If you are upgrading from Kubernetes 1.28+ and are not on Azure, GCE, or +vSphere then there are no changes you will need to make. If +you **are** on Azure, GCE, or vSphere, or you are upgrading from a version +older than 1.28, then read on. + +Historically, Kubernetes has included code for a set of cloud providers that +included AWS, Azure, GCE, OpenStack, and vSphere. Since the inception of +[KEP-2395][kep2395] the community has been moving towards removal of that +cloud provider code. The OpenStack provider code was removed in version 1.26, +and the AWS provider code was removed in version 1.27. This means that users +who are upgrading from one of the affected cloud providers and versions will +need to modify their deployments. + +### Upgrading on Azure, GCE, or vSphere + +There are two options for upgrading in this configuration: migrate to external +cloud controller managers, or continue using the in-tree provider code. +Although migrating to external cloud controller managers is recommended, +there are scenarios where continuing with the current behavior is desired. +Please choose the best option for your needs. + +#### Migrate to external cloud controller managers + +Migrating to use external cloud controller managers is the recommended upgrade +path, when possible in your situation. To do this you will need to +enable the `--cloud-provider=external` command line flag for the +`kube-apiserver`, `kube-controller-manager`, and `kubelet` components. In +addition you will need to deploy a cloud controller manager for your provider. + +Installing and running cloud controller managers is a larger topic than this +post can address; if you would like more information on this process please +read the documentation for [Cloud Controller Manager Administration][ccmadmin] +and [Migrate Replicated Control Plane To Use Cloud Controller Manager][ccmha]. +See [below](#cloud-provider-integrations) for links to specific cloud provider +implementations. + +#### Continue using in-tree provider code + +If you wish to continue using Kubernetes with the in-tree cloud provider code, +you will need to modify the command line parameters for `kube-apiserver`, +`kube-controller-manager`, and `kubelet` to disable the feature gates for +`DisableCloudProviders` and `DisableKubeletCloudCredentialProviders`. To do +this, add the following command line flag to the arguments for the previously +listed commands: + +``` +--feature-gates=DisableCloudProviders=false,DisableKubeletCloudCredentialProviders=false +``` + +_Please note that if you have other feature gate modifications on the command +line, they will need to include these 2 feature gates._ + +**Note**: These feature gates will be locked to `true` in an upcoming +release. Setting these feature gates to `false` should be used as a last +resort. It is highly recommended to migrate to an external cloud controller +manager as the in-tree providers are planned for removal as early as Kubernetes +version 1.31. + +### Upgrading on other providers + +For providers other than Azure, GCE, or vSphere, good news, the external cloud +controller manager should already be in use. You can confirm this by inspecting +the `--cloud-provider` flag for the kubelets in your cluster, they will have +the value `external` if using external providers. The code for AWS and OpenStack +providers was removed from Kubernetes before version 1.27 was released. +Other providers beyond the AWS, Azure, GCE, OpenStack, and vSphere were never +included in Kubernetes and as such they began their life as external cloud +controller managers. + +### Upgrading from older Kubernetes versions + +If you are upgrading from a Kubernetes release older than 1.26, and you are on +AWS, Azure, GCE, OpenStack, or vSphere then you will need to enable the +`--cloud-provider=external` flag, and follow the advice for installing and +running a cloud controller manager for your provider. + +Please read the documentation for +[Cloud Controller Manager Administration][ccmadmin] and +[Migrate Replicated Control Plane To Use Cloud Controller Manager][ccmha]. See +below for links to specific cloud provider implementations. + +## Where to find a cloud controller manager? + +At its core, this announcement is about the cloud provider integrations that +were previously included in Kubernetes. As these components move out of the +core Kubernetes code and into their own repositories, it is important to note +a few things: + +First, SIG Cloud Provider offers a reference framework for developers who +wish to create cloud controller managers for any provider. See the +[cloud-provider repository][cloud-provider] for more information about how +these controllers work and how to get started creating your own. + +Second, there are many cloud controller managers available for Kubernetes. +This post is addressing the provider integrations that have been historically +included with Kubernetes but are now in the process of being removed. If you +need a cloud controller manager for your provider and do not see it listed here, +please reach out to the cloud provider you are integrating with or the +[Kubernetes SIG Cloud Provider community][sig] for help and advice. It is +worth noting that while most cloud controller managers are open source today, +this may not always be the case. Users should always contact their cloud +provider to learn if there are preferred solutions to utilize on their +infrastructure. + +### Cloud provider integrations provided by the Kubernetes project {#cloud-provider-integrations} + +* AWS - https://github.com/kubernetes/cloud-provider-aws +* Azure - https://github.com/kubernetes-sigs/cloud-provider-azure +* GCE - https://github.com/kubernetes/cloud-provider-gcp +* OpenStack - https://github.com/kubernetes/cloud-provider-openstack +* vSphere - https://github.com/kubernetes/cloud-provider-vsphere + +If you are looking for an automated approach to installing cloud controller +managers in your clusters, the [kOps][kops] project provides a convenient +solution for managing production-ready clusters. + +## Want to learn more? + +Cloud providers and cloud controller managers serve a core function in +Kubernetes. Cloud providers are often the substrate upon which Kubernetes is +operated, and the cloud controller managers supply the essential lifeline +between Kubernetes clusters and their physical infrastructure. + +This post covers one aspect of how the Kubernetes community interacts with +the world of cloud infrastructure providers. If you are curious about this +topic and want to learn more, the Cloud Provider Special Interest Group (SIG) +is the place to go. SIG Cloud Provider hosts bi-weekly meetings to discuss all +manner of topics related to cloud providers and cloud controller managers in +Kubernetes. + +### SIG Cloud Provider + +* Regular SIG Meeting: [Wednesdays at 9:00 PT (Pacific Time)](https://zoom.us/j/508079177?pwd=ZmEvMksxdTFTc0N1eXFLRm91QUlyUT09) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=9:00&tz=PT%20%28Pacific%20Time%29). +* [Kubernetes slack][kslack] channel `#sig-cloud-provider` +* [SIG Community page][sig] + +[kep2395]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-cloud-provider/2395-removing-in-tree-cloud-providers +[fg]: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/ +[kubelet]: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ +[kcm]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/ +[kapi]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ +[ccmadmin]: https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ +[ccmha]: https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/ +[kslack]: https://kubernetes.slack.com +[sig]: https://github.com/kubernetes/community/tree/master/sig-cloud-provider +[cloud-provider]: https://github.com/kubernetes/cloud-provider +[oldblog]: https://kubernetes.io/blog/2019/04/17/the-future-of-cloud-providers-in-kubernetes/ +[kops]: https://github.com/kubernetes/kops