Skip to content

Latest commit

 

History

History

2395-removing-in-tree-cloud-providers

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
kep.yaml
kep.yaml
 
 

README.md

KEP-2395: Removing In-Tree Cloud Provider Code

Release Signoff Checklist

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
    • e2e Tests for all Beta API Operations (endpoints)
    • (R) Ensure GA e2e tests meet requirements for Conformance Tests
    • (R) Minimum Two Week Window for GA e2e tests to prove flake free
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • (R) Production readiness review approved
  • "Implementation History" section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Terms

  • CCM: Cloud Controller Manager - The controller manager responsible for running cloud provider dependent logic, such as the service and route controllers.
  • KCM: Kubernetes Controller Manager - The controller manager responsible for running generic Kubernetes logic, such as job and node_lifecycle controllers.
  • KAS: Kubernetes API Server - The core api server responsible for handling all API requests for the Kubernetes control plane. This includes things like namespace, node, pod and job resources.
  • K8s/K8s: The core kubernetes github repository.
  • K8s/cloud-provider: Any or all of the repos for each cloud provider. Examples include cloud-provider-gcp, cloud-provider-aws and cloud-provider-azure. We have created these repos for each of the in-tree cloud providers. This document assumes in various places that the cloud providers will place the relevant code in these repos. Whether this is a long-term solution to which additional cloud providers will be added, or an incremental step toward moving out of the Kubernetes org is out of scope of this document, and merits discussion in a broader forum and input from SIG-Architecture and Steering Committee.
  • K8s SIGs/library: Any SIG owned repository.
  • Staging: Staging: Separate repositories which are currently visible under the K8s/K8s repo, which contain code considered to be safe to be vendored outside of the K8s/K8s repo and which should eventually be fully separated from the K8s/K8s repo. Contents of Staging are prevented from depending on code in K8s/K8s which are not in Staging. Controlled by publishing kubernetes-rules-configmap
  • In-tree: code that lives in the core Kubernetes repository k8s.io/kubernetes.
  • Out-of-Tree: code that lives in an external repository outside of k8s.io/kubernetes.

Summary

This is a proposal outlining steps to remove "in-tree" cloud provider code from the k8s.io/kubernetes repo while being as least disruptive to end users and other Kubernetes developers as possible.

Motivation

Motiviation behind this effort is to allow cloud providers to develop and make releases independent from the core Kubernetes release cycle. The de-coupling of cloud provider code allows for separation of concern between "Kubernetes core" and the cloud providers within the ecosystem. In addition, this ensures all cloud providers in the ecosystem are integrating with Kubernetes in a consistent and extendable way.

Having all cloud providers developed/released in their own external repos/modules will result in the following benefits:

  • The core pieces of Kubernetes (kubelet, kube-apiserver, kube-controller-manager, etc) will no longer depend on cloud provider specific APIs (and their dependencies). This results in smaller binaries and lowers the chance of security vulnerabilities via external dependencies.
  • Each cloud provider is free to release features/bug fixes at their own schedule rather than relying on the core Kubernetes release cycle.

Goals

  • Remove all cloud provider specific code from the k8s.io/kubernetes repository with minimal disruption to end users and developers.

Non-Goals

  • Testing/validating out-of-tree provider code for all cloud providers, this should be done by each provider.

Proposal

In order to remove cloud provider code from k8s.io/kubernetes. A 4 phase approach will be taken.

  1. Move all code in k8s.io/kubernetes/pkg/cloudprovider/providers/<provider> to k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers/<provider>/. This requires removing all internal dependencies in each cloud provider to k8s.io/kubernetes.
  2. Begin to build/release the CCM from external repos (k8s.io/cloud-provider-<provider>) with the option to import the legacy providers from k8s.io/legacy-cloud-providers/<provider>. This allows the cloud-controller-manager to opt into legacy behavior in-tree (for compatibility reasons) or build new implementations of the provider. Development for cloud providers in-tree is still done in k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers/<provider> during this phase.
  3. Migrate all code in k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers and shift main development to k8s.io/cloud-provider-<provider>. External cloud provider repos can optionally still import k8s.io/legacy-cloud-providers but it will no longer be imported from core components in k8s.io/kubernetes. Changes to k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers are no longer accepted unless they are critical or security-related.
  4. Disable in-tree providers and set the DisableCloudProviders and DisableKubeletCloudCredentialProvider feature gates to true by default. This will enable external CCM behavior by default in Kubernetes.

Risks and Mitigations

  • Kubernetes users will need to add CCM deployments to their clusters. Previously, users were able to enable the cloud controller loops of the kubernetes-controller-manager through command line flags. With the change to external CCMs users will be responsible for managing their own CCM deployments.
  • Security for the core Kubernetes cloud provider interface will continue to reviewed by the Kuberenetes SIG Security community.
  • Security for the external CCMs will be reviewed by the project communities which own the specific CCM implementation, with supplemental reviews done by the SIG Security community.
  • UX for the core Kubernetes cloud provider interface will continue to be reviewed by the Kubernetes SIG Cloud Provider, SIG Architecture, ans SIG API Machinery communities.
  • UX for the external CCMs will be reviewed by the project communities which own the specific CCM implementation.

Design Details

Phase 1 - Moving Cloud Provider Code to Staging

In Phase 1, all cloud provider code in k8s.io/kubernetes/pkg/cloudprovider/providers/<provider> will be moved to k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers/<provider>. Reasons why we "stage" cloud providers as the first phase are:

  • The staged legacy provider repos can be imported from the out-of-tree provider if they choose to opt into the in-tree cloud provider implementation. This allows for a smoother transition between in-tree and out-of-tree providers in cases where there are version incompatibilites between the two.
  • Staging the cloud providers indicates to the community that they are slated for removal in the future.

The biggest challenge of this phase is to remove dependences to k8s.io/kubernetes in all the providers. This is a requirement of staging a repository and a best practice for consuming external dependencies. All other repos "staged" (client-go, apimachinery, api, etc) in Kubernetes follow the same pattern. The full list of internal dependencies that need to be removed can be found in issue 69585.

Phase 2 - Building CCM from Provider Repos

In Phase 2, cloud providers will be expected to build the cloud controller manager from their respective provider repos (k8s.io/cloud-provider-<provider>). Providers can choose to vendor in their legacy provider in k8s.io/legacy-cloud-providers/<provider>, build implementations from scratch or both. Development in-tree is still done in the staging directories under the k8s.io/kubernetes repo.

The kube-controller-manager will still import the cloud provider implementations in staging. The package location of the provider implementations will change because each staged directory will be "vendored" in from their respective staging directory. The only change in core components is how the cloud providers are imported and the behavior of each cloud provider should not change.

Phase 3 - Migrating Provider Code to Provider Repos

In Phase 3, feature development is no longer accepted in k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers/<provider> and development of each cloud provider should be done in their respective external repos. Only bug and security fixes are accepted in-tree during this phase. It's important that by this phase, both in-tree and out-of-tree cloud providers are tested and production ready. Ideally most Kubernetes clusters in production should be using the out-of-tree provider before in-tree support is removed. A plan to migrate existing clusters from using the kube-controller-manager to the cloud-controller-manager is currently being developed. More details soon.

External cloud providers can optionally still import providers from k8s.io/legacy-cloud-providers but no core components in k8s.io/kubernetes will import the legacy provider and the respective staging directory will be removed along with all its dependencies.

Phase 4 - Disabling In-Tree Providers

In Phase 4, two feature gates will be introduced to gradually disable and remove in-tree cloud providers:

  1. DisableCloudProviders - this feature gate will disable any functionality in kube-apiserver, kube-controller-manager and kubelet related to the --cloud-provider component flag.
  2. DisableKubeletCloudCredentialProvider - this feature gate will disable in-tree functionality in the kubelet to authenticate to the AWS, Azure and GCP container registries for image pull credentials.

Both of these features gates only impacts functionality tied to the --cloud-provider flag, specifically in-tree volume plugins are not covered. Users should refer to CSI migration efforts for these.

For alpha, the feature gates will be used for testing purposes. When enabled, tests will ensure that clusters with in-tree cloud providers disabled behaves as expected. This is targeted for v1.21 and will be disabled by default.

For beta, the feature gates will be on by default, meaning core components will disallow use of in-tree cloud providers. This will act as a warning for users to migrate to external components. Users may choose to continue using the in-tree provider by explicitly disabling the feature gates. Beta is targeted for v1.29 with the caveat that a majority of our CI signal jobs across providers should have converted to use CCM by then.

For GA, the feature gate will be enabled by default and locked. Users at this point MUST migrate to external components and use of the in-tree cloud providers will be disallowed. GA is targeted for v1.31. One release after GA, the in-tree cloud providers can be safely removed.

NOTE: the removal of the code will depend on when we can remove the in-tree storage plugins, so the actual removal may end up in a later release.

Staging Directory

There are several sections of code which need to be shared between the K8s/K8s repo and the K8s/Cloud-provider repos. The plan for doing that sharing is to move the relevant code into the Staging directory as that is where we share code today. The current Staging repo has the following packages in it.

  • Api
  • Apiextensions-apiserver
  • Apimachinery
  • Apiserver
  • Client-go
  • Code-generator
  • Kube-aggregator
  • Metrics
  • Sample-apiserver
  • Sample-Controller

With the additions needed in the short term to make this work; the Staging area would now need to look as follows.

  • Api
  • Apiextensions-apiserver
  • Apimachinery
  • Apiserver
  • Client-go
  • legacy-cloud-providers
  • Code-generator
  • Kube-aggregator
  • Metrics
  • Sample-apiserver
  • Sample-Controller

Cloud Provider Instances

Currently in K8s/K8s the cloud providers are actually included in the providers.go file which then includes each of the in-tree cloud providers. In the short term, we would leave that file where it is and adjust it to point at the new homes under Staging. For the K8s/cloud-provider repo, would have the following CCM wrapper file. (Essentially a modified copy of cmd/cloud-controller-manager/controller-manager.go) The wrapper for each cloud provider would import just their vendored cloud-provider implementation rather than providers.go file.

k8s/k8s: pkg/cloudprovider/providers/providers.go


import (
  // Prior to cloud providers having been moved to Staging
  _ "k8s.io/kubernetes/pkg/cloudprovider/providers/aws"
  _ "k8s.io/kubernetes/pkg/cloudprovider/providers/azure"
  _ "k8s.io/kubernetes/pkg/cloudprovider/providers/cloudstack"
  _ "k8s.io/kubernetes/pkg/cloudprovider/providers/gce"
  _ "k8s.io/kubernetes/pkg/cloudprovider/providers/openstack"
  _ "k8s.io/kubernetes/pkg/cloudprovider/providers/ovirt"
  _ "k8s.io/kubernetes/pkg/cloudprovider/providers/photon"
  _ "k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere"
)

k8s/cloud-provider-gcp: pkg/cloudprovider/providers/providers.go


import (
  // Cloud providers
  _ "k8s.io/legacy-cloud-providers/aws"
  _ "k8s.io/legacy-cloud-providers/azure"
  _ "k8s.io/legacy-cloud-providers/gce"
  _ "k8s.io/legacy-cloud-providers/openstack"
  _ "k8s.io/legacy-cloud-providers/vsphere"
)

Test Plan

This change will leverage the tests available in the kubernetes/kuberenetes repository that exercise cloud controller behavior. The largest change to testing for this KEP is the default enablement of external CCMs. Test workflows have been updated to exercise the DisableCloudProviders and DisableKubeletCloudCredentialProvider feature gates while also engaging the --cloud-provider=external, and related, command line flags to kubelet, kube-apiserver, and kube-controller-manager.

In all other respects, the expected behavior of the cloud controller managers is not changing, and as such the original tests for in-tree cloud controllers continue to be relevant and necessary.

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing update

The behavior of the cloud controllers is not changing with respect to cluster functioning, as such the prerequisite for this KEP is that all the current cloud controller related tests are passing. When the external CCMs are enabled, these tests should continue to pass.

As described in the non-goals section, testing of individual cloud provider CCMs is not in scope for this KEP. Each provider is expected to own the tests related to their specific cloud platform.

Unit tests

This KEP describes a process for extracting and migrating current in-tree code to external repositories. The notion of unit style testing takes on a different meaning in this perspective as the focus for the unit testing will move to those external repositories.

SIG Cloud Provider maintains a framework for creating CCMs which contains unit testing for the core controller loops of a CCM implementation. See:

Cloud providers who create external CCMs will be responsible for providing unit testing within their own repositories.

Integration tests

Integration testing on this change is complicated by the fact that any testing with concrete CCM implementations requires cloud infrastructure from the same provider. While it would be possible to create a mock provider for these type of scenarios, the SIG considers this testing to be a low value when compared to the e2e test which are running on concrete providers.

For the reasons stated above, this KEP focuses in e2e testing over integration.

e2e tests

This KEP will leverage the existing e2e test suite for the majority of testing. Given that this KEP proposes no behavioral changes to the functioning the of the cloud controllers, the existing tests will provide a valuable signal to ensure that nothing has been broken during the migration from in-tree to external CCM.

The following pull request is the first in a series (documented in the request) which enable external CCMs by default for all GCE/GCP testing from the Kubernetes repository:

The following test grids show the e2e tests running with external CCMs on GCP and AWS cloud providers respectively:

Graduation Criteria

Alpha

  • Feature gates added for DisableCloudProviders and DisableKubeletCloudCredentialProviders
  • Working out-of-tree ccms for existing in-tree providers
  • Unit and e2e tests to exercise the new out-of-tree CCMs

Beta

  • Disable in-tree cloud providers, feature gates enabled by default.
  • Most, if not all, testing in Kubernetes is using external CCMs. Exceptions are made for cases where no CCM is required.
  • Multiple providers are being e2e tested against the latest Kubernetes libraries on master. This is to prevent regressions when pinning to known versions of external CCMs.
  • Promotion of the migration progress and process through documentation, announcements on mailing lists, and delivered conference presentations.

GA

  • Two releases have passed at beta status without incident or regression.

Deprecation

  • Removal of all in-tree code no earlier than one release after GA release.
  • Removal of option to set specific cloud providers through --cloud-provider, these options should be removed at in-tree providers are removed.

Upgrade / Downgrade Strategy

The strategy for upgrading a Kubernetes cluster to use external CCMs can be briefly described as updating the commnand line flags for the kubelet, kube-apiserver, and kube-controller-manager, and then deploying the external CCM pods into the cluster. More detailed information about the command line flags and CCM operation can be found in the Kubernetes documentation for Cloud Controller Manager Administration and Migrate Replicated Control Plane To Use Cloud Controller Manager.

Similarly the strategy for downgrading a Kubernetes cluster to not use external CCMs is a reverse of the previous description: change the command line flags back to their original values, and then remove the external CCM deployments from the cluster.

The strategy for upgrading and downgrading the external CCMs is not in scope for this KEP as each cloud provider community will be responsible for the maintenance of their CCM. Similarly, documenting the strategy for operating specific cloud provider CCM implementations will be the responsibility of those provider communities.

Cloud Provider Specific Guidance

The user is now responsible for running the external CCMs, this should be done during cluster installation. Each provider may have different guidance for operating their CCM and as such provider-specific documentation should be consulted.

Some former in-tree cloud providers have created documentation to guide users in operating the external CCMs for those platforms. Please see the following:

An example of deploying an external CCM on GCE can be seen in the start-cloud-controller-manager script function from the Kubernetes repository:

General Guidance

There are a few general notes to observe when migrating to external CCMs. These notes are collected from Kubernetes issue 120411 - Collect tips for external cloud provider migration.

  • kube-apiserver
    • Ensure that the DisableCloudProviders feature gate is true, whether by default or explicitly.
  • kubelet
    • Ensure that the DisableKubeletCloudCredentialProviders feature gate is true, whether by default or explicitly.
    • Set the --image-credential-provider-config and --image-credential-provider-bin-dir flags as appropriate for the cloud provider, see the kubelet Synopsis documentation for more information.
  • kube-controller-manager

Version Skew Strategy

The external CCMs follow the same version skew policy as the kube-controller-manager. Providers, and communities, that publish their own CCM are responsible for updating those CCMs to follow the version and skew policy.

Copied from the Kubernetes Version Skew Policy documentation:

kube-controller-manager, kube-scheduler, and cloud-controller-manager must not be newer than the kube-apiserver instances they communicate with. They are expected to match the kube-apiserver minor version, but may be up to one minor version older (to allow live upgrades).

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?
  • Feature gate (also fill in values in kep.yaml)
    • Feature gate name: DisableCloudProviders
    • Components depending on the feature gate: kubelet, kube-apiserver, kube-controller-manager
  • Feature gate (also fill in values in kep.yaml)
    • Feature gate name: DisableKubeletCloudCredentialProvider
    • Components depending on the feature gate: kubelet
Does enabling the feature change any default behavior?

Yes, enabling this feature will disable all capabilities enabled when --cloud-provider is set in core components. Users need to ensure they have migrated to out-of-tree components prior to enabling this feature gate. If appropriate extensions (CCM, credential provider, apiserver-network-proxy, etc) are in use, cloud provider capabilities should remain the same at the very least.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes, the feature can be disabled once it is enabled. If disabled, users must ensure that the CCM is no longer running in the cluster. Credential provider plugins and the apiserver network proxy do not have to be stopped on rollback.

What happens if we reenable the feature if it was previously rolled back?

All capabilities from in-tree cloud providers will be re-disabled.

Are there any tests for feature enablement/disablement?

Yes, there are a large number of feature tests including:

For enablement/disablement tests, we are relying on manual tests that were done downstream by at least two large cloud providers: GCP and AWS.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

The primary method of rollout failure is due to configuration issues. When migrating to use external CCMs there are several changes which must be made to ensure proper functioning. When encountering a failure, users should review the migration documentation to ensure that all command line flags, RBAC requirements, taint tolerations, and manifests are following the best practices.

It is possible that a failed rollout could affect running workloads. Normal functioning of external CCMs involves activities around labeling new node objects, removing node objects, and configuring loadbalancer type services. If a rollout of external CCMs failed, users might see failures of workloads to schedule, workloads might be evicted, and service-based traffic could be interrupted.

For more information about rollouts, upgrades, and planning, please see the Kubernetes documentation:

What specific metrics should inform a rollback?

There are a few observable behaviors that inform a rollback. If users start to see nodes failing to register, load balancer calls failing or not happening, deleted nodes not being removed, or components failing to start due to missing taint tolerations, these could all be possible signs that a CCM rollout has failed.

In addition there are several metrics exposed by the cloud-provider framework that is maintained by SIG Cloud Provider. If the external CCM is using the cloud-provider framework, the following metrics could inform about failing behavior, users should be observant of these metric values increasing over a short period of time, or after upgrade:

  • cloud_provider_taint_removal_delay_seconds
  • initial_node_sync_delay_seconds
  • nodesync_error_total
  • nodesync_latency_seconds
  • update_loadbalancer_host_latency_seconds
Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

Yes, the upgrade and downgrade paths have been tested manually on AWS and GCE by those provider owners. We do not have artifacts of those tests.

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
  • The provider options (e.g. aws, azure, etc), with the exception of external, for the --cloud-provider flag of kubelet, kube-apiserver, and kube-controller-manager are deprecated with the beta release.
  • The in-tree cloud controller loops for kubelet, kube-apiserver, and kube-controller-manager are deprecated with the beta release.

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?
  • On cloud providers that utilize an image credential provider, this feature is in use when container images are being pulled using those credentials.
  • Workloads that explicitly use load balancer type services will depend on external CCMs on providers that support the load balancer service controller.
  • Workloads that depend on well-known zone and region labels on nodes are also considered to be using this feature transitively.
How can someone using this feature know that it is working for their instance?
  • Other (treat as last resort)
    • Details: The Uninitialized taint should be removed from Node objects.
What are the reasonable SLOs (Service Level Objectives) for the enhancement?
  • 99% of cloud_provider_taint_removal_delay_seconds metric counts are less than 60 seconds.
  • 99% of initial_node_sync_delay_seconds metric counts are less than 60 seconds.
  • On providers that utilize load balancer type services:
    • 99% of update_loadbalancer_host_latency_seconds metric counts are less than 60 seconds.
    • 99% of nodesync_latency_seconds metric counts are less than 60 seconds.
    • Rate of increase for nodesync_error_total is less than 1 per 15 minutes.
  • The existing SLOs for functionalities the kube-controller-manager should be used, this change is supposed to be no-op from the end-user perspective.
What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
  • Metrics
    • Metric name: cloud_provider_taint_removal_delay_seconds
      • Components exposing the metric: CCM
    • Metric name: initial_node_sync_delay_seconds
      • Components exposing the metric: CCM
    • Metric name: update_loadbalancer_host_latency_seconds
      • Components exposing the metric: CCM
    • Metric name: nodesync_error_total
      • Components exposing the metric: CCM
    • Metric name: nodesync_latency_seconds
      • Components exposing the metric: CCM
Are there any missing metrics that would be useful to have to improve observability of this feature?

A metric to indicate failed connections with the cloud provider API endpoints could be helpful in determining when a CCM is not able to communicate with the underlying infrastructure.

Dependencies

Does this feature depend on any specific services running in the cluster?

CCM must be able to connect to cloud provider services and endpoints, this will depend on the specifics of each provider

Scalability

Will enabling / using this feature result in any new API calls?

Possibly. This change focuses on a migration of code from core Kubernetes components into external user-managed components. During the alpha state of this feature, the code from in-tree implementations was migrated into external out-of-tree repositories. At that point in time, the feature code was the same between in-tree and out-of-tree implementations. As the out-of-tree CCMs have now become independent projects, it is out of scope for this KEP to provide continuous updating of those project statuses with respect to API calls.

Will enabling / using this feature result in introducing new API types?

No.

Will enabling / using this feature result in any new calls to the cloud provider?

Possibly. As this change focuses on a migration of previously in-tree code to externally developed CCMs, it is out of scope to provide predictive guidance about how those project teams will add or remove calls to the cloud provider. In the alpha state of this feature, the in-tree code was migrated into external out-of-tree repositories. At that point in time, no new cloud provider calls were added.

Will enabling / using this feature result in increasing size or count of the existing API objects?

No.

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

No.

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?

No. In fact, it should reduce resource usage.

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

No, it should not. But, as with any change that requires running additional pods it is possible that the new CCMs will place load on cluster resources. The load from the new CCMs should not be significantly greater than the previous in-tree implementations.

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?
  • CCMs will not be able to update node and service objects.
  • In general, the expectations for the kube-controller-manager with respect to an unavailable API server and/or etcd will apply to this feature as well.
What are other known failure modes?
  • CCM not communicating with provider infrastructure.
    • Detection: Nodes not having the Uninitialized taint removed autoamtically. The nodesync_error_total metric shows an increasing rate.
    • Mitigation: Check configuration of cloud credentials and CCM to ensure that the proper values, RBAC, and quotas are granted with the provider.
    • Diagnostics: Logs should be checked for failed calls to the infrastructure provider (these will looks different on each provider). The default log level should be sufficient to show these failures.
    • Testing: Check cloud credentials manually with provider. Additionally, watch nodes for removal of the Uninitialized taint and the application of Ready state.
What steps should be taken if SLOs are not being met to determine the problem?
  • Checking the CCM logs for failures and errors.
  • Review the RBAC requirements for the CCMs and related Kubernetes components.
  • Confirm cloud provider configurations, settings, and quotas.
  • Review requests and limits for CCMs to ensure proper resourcing.
  • If bug or other failure, rollback to in-tree providers and open an issue with the appropriate cloud provider.

Implementation History

  • 2019-01-28 - Summary, Motivation, and Proposal sections merged.
  • 2021-02-09 - Add phase 4 to Proposal section with addition of DisableCloudProviders and DisableKubeletCloudCredentialProvider feature gates.
  • 2021-02-09 - Add production readiness review for alpha stage.
  • 2021-08-04 - First Kubernetes release (v1.22) with DisableCloudProviders and DisableKubeletCloudCredentialProviders feature gates.
  • 2023-09-02 - Enable external CCMs by default for k/k CI.
  • 2023-05-07 - All the in-tree cloud providers have been removed.

Drawbacks

A drawback of this proposal is the complexity involved with doing this type of change in the Kubernetes community. It will necessarily require an increased level of turbulence for developers and users as they learn about the changes to the kubelet, kube-apiserver, and kube-controller-manager processes, and the changes necessary to operate the external CCMs. Users will need to adapt their practices and automation to include the changes to command line flags and manifests to enable external cloud providers.

Alternatives

Staging Alternatives

Git Filter-Branch

One possible alternative is to make use of a Git Filter Branch to extract a sub-directory into a virtual repo. The repo needs to be sync'd in an ongoing basis with K8s/K8s as we want one source of truth until K8s/K8s does not pull in the code. This has issues such as not giving K8s/K8s developers any indications of what the dependencies various K8s/Cloud-providers have. Without that information it becomes very easy to accidentally break various cloud providers and time you change dependencies in the K8s/K8s repo. With staging the dependency line is simple and automatically enforced. Things in Staging are not allowed to depend on things outside of Staging. If you want to add such a dependency you need to add the dependent code to Staging. The act of doing this means that code should get synced and solve the problem. In addition the usage of a second different library and repo movement mechanism will make things more difficult for everyone.

“Trying to share code through the git filter will not provide this protection. In addition it means that we now have two sharing code mechanisms which increases complexity on the community and build tooling. As such I think it is better to continue to use the Staging mechanisms. ”

Build Location Alternatives

Build K8s/K8s from within K8s/Cloud-provider

The idea here is to avoid having to add a new build target to K8s/K8s. The K8s/Cloud-provider could have their own custom targets for building things like KAS without other cloud-providers implementations linked in. It would also allow other customizations of the standard binaries to be created. While a powerful tool, this mechanism seems to encourage customization of these core binaries and as such to be discouraged. Providing the appropriate generic binaries cuts down on the need to duplicate build logic for these core components and allow each optimization of build. Download prebuilt images at a version and then just build the appropriate addons.

Build K8s/Cloud-provider within K8s/K8s

The idea here would be to treat the various K8s/Cloud-provider repos as libraries. You would specify a build flavor and we would pull in the relevant code based on what you specified. This would put tight restrictions on how the K8s/Cloud-provider repos would work as they would need to be consumed by the K8s/K8s build system. This seems less extensible and removes the nice loose coupling which the other systems have. It also makes it difficult for the cloud providers to control their release cadence.

Config Alternatives

Use component config to determine where controllers run

Currently KCM and CCM have their configuration passed in as command line flags. If their configuration were obtained from a configuration server (component config) then we could have a single source of truth about where each controller should be run. This both solves the HA migration issue and other concerns about making sure that a controller only runs in 1 controller manager. Rather than having the controllers as on or off, controllers would now be configured to state where they should run, KCM, CCM, Nowhere, … If the KCM could handle this as a run-time change nothing would need to change. Otherwise it becomes a slight variant of the proposed solution. This is probably the correct long term solution. However for the timeline we are currently working with we should use the proposed solution.