New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support out-of-process and out-of-tree cloud providers #88

Open
errordeveloper opened this Issue Sep 12, 2016 · 103 comments

Comments

@errordeveloper
Member

errordeveloper commented Sep 12, 2016

Feature Description:
Support out-of-tree and out-of-process cloud providers, a.k.a pluggable cloud providers.

Feature Progress:
In order to complete this feature, cloud provider dependencies need to be moved out the the following Kubernetes binaries, then docs and tests need to be added. The Links to the right hand side of the binary denote the PRs that lead to the completion of the sub-feature

  1. Kube-controller-manager -
  1. Kubelet
  1. Docs
  1. Tests
e2e Tests - Incomplete

The cloud-specific functionality of the above features needs to be moved into a new binary called cloud-controller-manager that support a plugin architecture.

Primary Contact: @wlan0

Responsible SIG: @k8s-mirror-cluster-lifecycle-feature-re

Design Proposal Link: kubernetes/community#128

Reviewers:
@luxas
@roberthbailey
@thockin

Approver:
@thockin

Feature Target:
Alpha: 1.7
Beta: 1.8
Stable: 1.10


Here's an updated status report for this feature, please let me know if anything needs clarification:

Beta (starting v1.11)

  • The common interface used by cloud providers has been well tested and support will not be dropped, though implementation details may change. Any methods that are deprecated should follow the Kubernetes Deprecation Policy.
  • The cloud controller manager has been tested by various cloud providers and is considered safe to use for out-of-tree providers. Features to be deprecated that are part of the cloud controller manager (controllers, component flags, etc) will follow the Kubernetes Deprecation Policy.
  • The cloud controller manager does not run in any cluster by default. It must be explicitly turned on and added like any other control plane component. Instructions for setup may slightly vary per cloud provider. More details here.

Reasoning for Graduation

There were a few things on our TODO list that we wanted to get done before graduating to beta such as collecting E2E tests from all providers & improving out-of-tree storage. However, many of these initiatives require collaboration from external parties that was delaying progress on this effort. In addition, there was uncertainty since we do not develop some of the components we rely on, a good example is whether CSI would be able to meet demands for out-of-tree storage that was on par with in-tree storage support. Though in hindsight we have more confidence in CSI, prior to its beta release it was unclear if it would meet our requirements. With this context in mind, we had decided to graduate to beta because:

  • blocking out-of-tree cloud providers from going beta meant that less in-tree providers will adopt this feature.
  • some goals (like E2E tests from cloud providers) requires a significant amount of collaboration and may unnecessarily block progress for many releases.
  • features that are lacking from the cloud controller manager (mainly storage) would be handled by future projects from other SIGs (e.g. CSI by SIG Storage).

Goals for GA (targetted for v1.13/v1.14)

  • Frequently collect E2E tests results from all in-tree & out-of-tree cloud providers kubernetes/community#2224
  • Cloud Provider Documentation includes:
    • “Getting Started” documentation - outlines the necessary steps required to stand up a Kubernetes cluster.
    • Documentation outlining all cloud provider features such as LoadBalancers, Volumes, etc. There should be docs providing a high-level overview and docs that dig into sufficient details on how each feature works under the hood.
    • Docs should also be centralized in an automated fashion where documentation from all cloud providers are placed into a central location (ideally https://kubernetes.io/docs/home/).
  • A well-documented plan exists for how to migrate a cluster from using in-tree cloud provider to out-of-tree cloud provider, this only applies to AWS, Azure, GCP, OpenStack, and VMWare.
  • All current cloud providers have implemented an out-of-tree solution, deprecation of in-tree code is preferred but not a requirement.
@colemickens

This comment has been minimized.

colemickens commented Sep 12, 2016

Benefits:

  • Easier configuration for providers like Azure that require a "cloud config" flag on kubelet/kcm. This file could instead by made a Secret (or ConfigMap + Secret). Makes bootstrapping easier and would eliminate the need for kubeadm to have special functionality for handling the cloudprovider flags.
  • Selectively enablement. Some people want to run their own overlay network, but still want auto-provisioned L4 load balancers. There's no way to do that today.
  • Moves more things out of core Kubernetes repo/project, and enables faster turn-around for shipping new cloudproviders or iterating/testing changes.

Just a note, kubelet uses cloudprovider too, in addition to KCM.

@errordeveloper

This comment has been minimized.

Member

errordeveloper commented Sep 14, 2016

cc @kubernetes/sig-cluster-lifecycle @kubernetes/sig-network @kubernetes/sig-storage @kubernetes/sig-aws @kubernetes/sig-openstack

@thockin

This comment has been minimized.

Member

thockin commented Sep 15, 2016

I endorse this idea in general. I think the built-in cloud provider logic has served its purpose and its time to modularize. I think there are a number of facets to this that we have to work out including but not limited to:

  • CloudProvider and all the APIs therein
  • Volume drivers and provisioner support
  • Cluster turnup support

I think it would be worthwhile to start building a doc that details these and explores options for ejecting each one. I don't think there's anything here that hasn't been considered at SOME point. Once we get that written down, we can craft a roadmap...

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Sep 20, 2016

Sounds useful. @errordeveloper, do we have any mailing lists or GitHub discussions with this question what we can refer to?

@errordeveloper

This comment has been minimized.

Member

errordeveloper commented Sep 20, 2016

@idvoretskyi not yet, this is probably very much on the radar of @kubernetes/sig-cluster-lifecycle.

@thockin

This comment has been minimized.

Member

thockin commented Sep 21, 2016

I don't know that anyone is working on speccing this. It touches on a few
SIGs, but it is not exactly any of them.

On Tue, Sep 20, 2016 at 5:35 AM, Ilya Dmitrichenko <notifications@github.com

wrote:

@idvoretskyi https://github.com/idvoretskyi not yet, this is probably
very much on the radar of @kubernetes/sig-cluster-lifecycle
https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVJJcJXb3gnueygHKf_uNVH7GkQ2nks5qr9MLgaJpZM4J6QkU
.

@errordeveloper

This comment has been minimized.

Member

errordeveloper commented Sep 21, 2016

@thockin you are right. May be we should form sig-cloud?

@thockin

This comment has been minimized.

Member

thockin commented Sep 21, 2016

so. many. sigs. I don't think we need a SIG for this. I doubt if it is
going to garner much resistance. There are just a lot of details to hammer
out. Being on the radar for lifecycle is fine. The hardest part here is
balancing the desire for modularity with the need for simplicity. That's
what I want to see explored :)

On Wed, Sep 21, 2016 at 12:07 AM, Ilya Dmitrichenko <
notifications@github.com> wrote:

@thockin https://github.com/thockin you are right. May be we should
form sig-cloud?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVA1M1spBOe4hF4Wsgy-YQsvppxUeks5qsNeygaJpZM4J6QkU
.

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Sep 22, 2016

@errordeveloper no need in yet another SIG (SIG-Cloud sounds like an abstract and umbrella one). I agree with @thockin - the primary SIG has to be @kubernetes/sig-cluster-lifecycle; while on behalf of @kubernetes/sig-openstack I'm going to track this item.

Hope other cloud-SIG's will be involved in the process as well.

@errordeveloper

This comment has been minimized.

Member

errordeveloper commented Sep 28, 2016

@justinsb and I have discussed this on Slack, and looks like we may be able to get closer to getting similar user-facing value by exposing flags via component config. It also turns out --configure-cloud-routes is already there. It doesn't look like this should involve moving code as such.

@bboreham

This comment has been minimized.

bboreham commented Sep 28, 2016

I think there is additional value to moving code out to add-ons: it will enable further cloud providers to be added without enlarging the core of Kubernetes.

Example: kubernetes/kubernetes#32419

@errordeveloper

This comment has been minimized.

Member

errordeveloper commented Sep 28, 2016

Ah, but it also looks like someone is working on this:
kubernetes/kubernetes#32419 (comment).

On Wed, 28 Sep 2016, 18:19 Bryan Boreham, notifications@github.com wrote:

I think there is additional value to moving code out to add-ons: it will
enable further cloud providers to be added without enlarging the core of
Kubernetes.

Example: kubernetes/kubernetes#32419
kubernetes/kubernetes#32419


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPWS7hzMWW-U6R-KrehxlYilUUXAIqnks5quqGkgaJpZM4J6QkU
.

@thockin

This comment has been minimized.

Member

thockin commented Sep 28, 2016

That does not read as someone working on it, to me. This is a big problem
with a lot of facets, and it needs a capital-O Owner.

On Wed, Sep 28, 2016 at 10:27 AM, Ilya Dmitrichenko <
notifications@github.com> wrote:

Ah, but it also looks like someone is working on this:
kubernetes/kubernetes#32419 (comment)
.

On Wed, 28 Sep 2016, 18:19 Bryan Boreham, notifications@github.com
wrote:

I think there is additional value to moving code out to add-ons: it will
enable further cloud providers to be added without enlarging the core of
Kubernetes.

Example: kubernetes/kubernetes#32419
kubernetes/kubernetes#32419


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#88 (comment)
,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAPWS7hzMWW-U6R-
KrehxlYilUUXAIqnks5quqGkgaJpZM4J6QkU>
.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVHiMNL34Eyg1T0vaRtIvQ065ssuTks5quqOKgaJpZM4J6QkU
.

@ibuildthecloud

This comment has been minimized.

ibuildthecloud commented Sep 29, 2016

@thockin In reference to kubernetes/kubernetes#32419, Rancher would be up for being a guinea pig for this. @wlan0 will be working on this and if the scope is massive we will see if we can pull in more resources. I want to see if understand the approach you were proposing in kubernetes/kubernetes#32419 and see if we are on the same page.

What we would do is implement the existing cloudprovider.Interface with a new cloud provider called "external". Ideally we wouldn't change the existing Interface, but if we hit some oddities it might make sense to modify it. This new external implementation will not delegate via a plugin model but instead through k8s resources and expect one to write controllers. Upfront it seems like we would need some new resources like CloudProviderLoadBalancer, Instance, Zone, Cluster, Route. A new cloud provider would need to be a controller that interacted with these resources.

That all seems pretty straight forward to me. Now the weird part is volume plugins. While it's not part of the CloudProvider interface, there seems to be a back channel relationship between volume plugins and cloud providers. To decouple those I'd have to spend a bit more time researching.

@thockin Is this the basic approach you were thinking?

@thockin

This comment has been minimized.

Member

thockin commented Sep 30, 2016

I replied to @wlan0, but for the record...

simpler.

My "external" suggestion was more about designating that we are not using a
built-in and any controller loops that use CloudProvider should be
disabled. "" may be just as viable.

Once the built-in controllers are nullified, we run a cloud-specific
controller manager. I propose that the starting point LITERALLY be a fork
of the kube-controller-manager code. But instead of linking in 8
CloudProviders and switching on a flag, just link one. Simplify and
streamline.

One possible result is a library pkg that accepts a type CloudProvider interface. In doing this, I am sure you will find things that need
restructuring or that are significantly harder this way, and that is when
we should discuss design.

I would suggest leaving volumes for last :)

On Thu, Sep 29, 2016 at 2:37 PM, Darren Shepherd notifications@github.com
wrote:

@thockin https://github.com/thockin In reference to
kubernetes/kubernetes#32419
kubernetes/kubernetes#32419, Rancher would be
up for being a guinea pig for this. @wlan0 https://github.com/wlan0
will be working on this and if the scope is massive we will see if we can
pull in more resources. I want to see if understand the approach you were
proposing in kubernetes/kubernetes#32419
kubernetes/kubernetes#32419 and see if we are
on the same page.

What we would do is implement the existing cloudprovider.Interface with a
new cloud provider called "external". Ideally we wouldn't change the
existing Interface, but if we hit some oddities it might make sense to
modify it. This new external implementation will not delegate via a plugin
model but instead through k8s resources and expect one to write
controllers. Upfront it seems like we would need some new resources like
CloudProviderLoadBalancer, Instance, Zone, Cluster, Route. A new cloud
provider would need to be a controller that interacted with these resources.

That all seems pretty straight forward to me. Now the weird part is volume
plugins. While it's not part of the CloudProvider interface, there seems to
be a back channel relationship between volume plugins and cloud providers.
To decouple those I'd have to spend a bit more time researching.

@thockin https://github.com/thockin Is this the basic approach you were
thinking?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVEdVlIPVVXuLrBE-NameYsUfMHULks5qvC98gaJpZM4J6QkU
.

@colemickens

This comment has been minimized.

colemickens commented Sep 30, 2016

Why is one linked in at all? What is the difference between --cloud-provider=external and simply not specifying the --cloud-provider at all? Then the Service/Routes are established with standalone addon controllers?

Or maybe we're on the same page? And then you're proposing a generic implementation of these standalone controllers (for now?) that can take the existing CloudProvider interface to preserve existing functionality?

@thockin

This comment has been minimized.

Member

thockin commented Sep 30, 2016

On Fri, Sep 30, 2016 at 12:16 AM, Cole Mickens notifications@github.com wrote:

Why is one linked in at all? What is the difference between --cloud-provider=external and simply not specifying the --cloud-provider at all? Then the Service/Routes are established with standalone addon controllers?

Without inspecting, I don't know if "" disables the controllers today,
so I didn't want to break compat during the transition. That's all.
If "" works, that is simpler.

Or maybe we're on the same page? And then you're proposing a generic implementation of these standalone controllers (for now?) that can take the existing CloudProvider interface to preserve existing functionality?

I think same page. As a starting point, we would decompose the single
{kube-controller-manager (KCM) + 8 CloudProviders} into 8 * {KCM + 1
CloudProvider}. At that point, each cloud-controller could diverge if
they want to, or we could keep maintaining the cloud controller
manager as a library.

@alena1108

This comment has been minimized.

alena1108 commented Sep 30, 2016

So the controller manager that embeds certain control loops. Some of these loops are cloud provider specific:

  • nodeController
  • volumeController
  • routeController
  • serviceController

but most are provider agnostic:

  • replicationController
  • endpointController
  • resourcequotacontroller
  • namespacecontroller
  • deploymentController
    etc

I wonder if it would make sense to split the controller into 2 parts: base-controller (k8s code base). And provider-specific-controller (external repo, deployed by the user by choice). This way it would be more similar to the ingress controller path with the only slight difference: controller loops should be maintained as a library as all the providers will share them. Only implementation - attach/detachDisk/etc - will be provider specific. To make it backwards compatible, we can disble initializing cloud provider specific controllers in the current controller-manager code if the provider is passed as empty on the kubernetes start.

Or may be I'm just stating what you've already meant by "keep maintaining the cloud controller
manager as a library" @thockin

@thockin

This comment has been minimized.

Member

thockin commented Oct 1, 2016

I think we are saying the same thing. kube-controller-manager will still
exist after this, but it will eventually get rid of all the cloud-specific
stuff. All the cloud-stuff will move to per-cloud controller binaries.

I would leave volumes for VERY LAST :)

On Fri, Sep 30, 2016 at 4:10 PM, Alena Prokharchyk <notifications@github.com

wrote:

So the controller manager that embeds certain control loops. Some of these
loops are cloud provider specific:

  • nodeController
  • volumeController
  • routeController
  • serviceController

but most are provider agnostic:

  • replicationController
  • endpointController
  • resourcequotacontroller
  • namespacecontroller
  • deploymentController etc

I wonder if it would make sense to split the controller into 2 parts:
base-controller (k8s code base). And provider-specific-controller (external
repo, deployed by the user by choice). This way it would be more similar to
the ingress controller path with the only slight difference: controller
loops should be maintained as a library as all the providers will share
them. Only implementation - attach/detachDisk/etc - will be provider
specific. To make it backwards compatible, we can disble initializing cloud
provider specific controllers in the current controller-manager code if the
provider is passed as empty on the kubernetes start.

Or may be I'm just stating what you've already meant by "keep maintaining
the cloud controller
manager as a library" @thockin https://github.com/thockin


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVEIPxqm6_wbc8Ss0QZuONiwM2IrNks5qvZbWgaJpZM4J6QkU
.

wlan0 added a commit to wlan0/kubernetes that referenced this issue Oct 6, 2016

start breaking up controller manager into two pieces
Addresses: kubernetes/enhancements#88

This commit starts breaking the controller manager into two pieces, namely,

1. cloudprovider dependent piece
2. coudprovider agnostic piece

the controller manager has the following control loops -

 - nodeController
 - volumeController
 - routeController
 - serviceController
 - replicationController
 - endpointController
 - resourcequotacontroller
 - namespacecontroller
 - deploymentController etc..

among the above controller loops,

 - nodeController
 - volumeController
 - routeController
 - serviceController

are cloud provider dependent. As kubernetes has evolved tremendously, it has become difficult
for different cloudproviders (currently 8), to make changes and iterate quickly. Moreover, the
cloudproviders are constrained by the kubernetes build/release lifecycle. This commit is the first
step in moving towards a kubernetes code base where cloud providers specific code will move out of
the core repository, and will be maintained by the cloud providers themselves.

I have added a new cloud provider called "external", which signals the controller-manager that
cloud provider specific loops are being run by another controller. I have added these changes in such
a way that the existing cloud providers are not affected. This change is completely backwards compatible, and
does not require any changes to the way kubernetes is run today.

Finally, along with the controller-manager, the kubelet also has cloud-provider specific code, and that will
be addressed in a different commit/issue.

wlan0 added a commit to wlan0/kubernetes that referenced this issue Oct 13, 2016

start breaking up controller manager into two pieces
Addresses: kubernetes/enhancements#88

This commit starts breaking the controller manager into two pieces, namely,

1. cloudprovider dependent piece
2. coudprovider agnostic piece

the controller manager has the following control loops -

 - nodeController
 - volumeController
 - routeController
 - serviceController
 - replicationController
 - endpointController
 - resourcequotacontroller
 - namespacecontroller
 - deploymentController etc..

among the above controller loops,

 - nodeController
 - volumeController
 - routeController
 - serviceController

are cloud provider dependent. As kubernetes has evolved tremendously, it has become difficult
for different cloudproviders (currently 8), to make changes and iterate quickly. Moreover, the
cloudproviders are constrained by the kubernetes build/release lifecycle. This commit is the first
step in moving towards a kubernetes code base where cloud providers specific code will move out of
the core repository, and will be maintained by the cloud providers themselves.

I have added a new cloud provider called "external", which signals the controller-manager that
cloud provider specific loops are being run by another controller. I have added these changes in such
a way that the existing cloud providers are not affected. This change is completely backwards compatible, and
does not require any changes to the way kubernetes is run today.

Finally, along with the controller-manager, the kubelet also has cloud-provider specific code, and that will
be addressed in a different commit/issue.
@dims

This comment has been minimized.

Member

dims commented Oct 31, 2016

Hi @thockin Is this the one you mentioned to me in the hall way conversation in Barcelona? Just making sure!

@luxas

This comment has been minimized.

Member

luxas commented Oct 31, 2016

Is this planned for v1.6 or what's the plan?
@wlan @ibuildthecloud @alena1108

@thockin

This comment has been minimized.

Member

thockin commented Oct 31, 2016

@dims yeah, this is the one.

On Mon, Oct 31, 2016 at 8:17 PM, Davanum Srinivas notifications@github.com
wrote:

Hi @thockin https://github.com/thockin Is this the one you mentioned to
me in the hall way conversation in Barcelona? Just making sure!


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVAYCPv8n1JV8TNOunXBr0z8KR-SBks5q5j7ggaJpZM4J6QkU
.

@idvoretskyi idvoretskyi added this to the next-milestone milestone Nov 3, 2016

@karataliu

This comment has been minimized.

karataliu commented Apr 26, 2018

To provide a few questions from my observation:

  1. Release
  • How are the separated cloud controller manager be released, do we propose single process or each should be on their on?
  • Who will be responsible for tagging the commit and uploading release packages? Does each sig own their release process?
  1. Doc location
  1. Migration doc
  • For users switching to standalone cloud controller provider mode, they would find the cloudprovider based volume plugins won't work.
    There should be instruction in turning on external-cloud-volume-plugin, adding provider-id, and so on.
  1. E2E baseline
  • When running e2e tests against cloud provider, which Kubernets release to pick: the latest version or latest tagged version on the same major releasing branch?
    Note there are two kubernetes versions to choose:
  1. The main release, providing hyperkube, which is to be deployed to cluster
  2. The test base, providing e2e test cases, which is to be run locally
    This won't be a problem for testing k8s main repo, since it would always pick current build.
  • In the long term , when out of tree providers become stable (removed from main), will kubernetes main repo still run e2e tests with specific cloud providers? If so which cloud provider verison will it pick?
@andrewsykim

This comment has been minimized.

Member

andrewsykim commented Apr 26, 2018

How are the separated cloud controller manager be released, do we propose single process or each should be on their on?

It would be up to each provider to maintain their own schedule. Ideally it will be in sync with kubernetes releases but not a strict requirement.

Who will be responsible for tagging the commit and uploading release packages? Does each sig own their release process?

Each provider should have a set of OWNERS that will be responsible for that. For in-tree providers that transition into out-of-tree providers will likely adopt the current SIG OWNERs.

For users switching to standalone cloud controller provider mode, they would find the cloudprovider based volume plugins won't work. There should be instruction in turning on external-cloud-volume-plugin, adding provider-id, and so on.

Agreed, as we go from beta -> GA, docs will become increasingly important, however, we have a tested mechanism in place to provide external-cloud-volume-plugin as you mentioned which we think is enough to move forward for now.

When running e2e tests against cloud provider, which Kubernets release to pick: the latest version or latest tagged version on the same major releasing branch?
In the long term , when out of tree providers become stable (removed from main), will kubernetes main repo still run e2e tests with specific cloud providers? If so which cloud provider verison will it pick?

Yes we're still hashing this out, but we don't think it will change the API/interface we use for this feature, but definitely something that will be worked on for GA release. We do have a process in place for E2E testing cloud providers, openstack being early adopters there.

Happy to answer any questions in further details either here or in the weekly WG meetings. Overall we think there's a lot of work left for this feature but we think the feature set/API we provide to make this work has been well tested and is functional enough for a beta release. The biggest problems we'll face going forward is pushing SIGs/providers in-tree to adopt this but we don't think adoption from in-tree providers is a strict requirement for beta (but most likely for GA).

@mistyhacks

This comment has been minimized.

mistyhacks commented May 24, 2018

@wlan0 @andrewsykim please fill out the appropriate line item of the
1.11 feature tracking spreadsheet
and open a placeholder docs PR against the
release-1.11 branch
by 5/25/2018 (tomorrow as I write this) if new docs or docs changes are
needed and a relevant PR has not yet been opened.

@andrewsykim

This comment has been minimized.

Member

andrewsykim commented May 25, 2018

Preliminary GA milestones

  • E2E tests reported to test grid by all providers
  • Automated Docs update - working with docs team to automatically fetch docs from provider repos and into kubernetes/website
  • Production ready migration plan for in-tree providers to migrate to out-of-tree successfully
  • Have a better story for integrating external clouds with the TLS bootstrapping feature - right now there's a dependency loop since TLS bootstrapping relies on the address types set by the cloud provider aware kubelet.
@zparnold

This comment has been minimized.

Member

zparnold commented May 30, 2018

Looks like we still need some docs to get this feature ready for release @wlan0 @andrewsykim
Could I please get some help with that? If there’s anything I can do to assist please let me know

At a minimum we're looking to have a placeholder PR on the kubernetes/website repo. The process is fairly straightforward: checkout release-1.11 branch, make a placeholder commit, push it to your fork, and raise a PR between it and the release-1.11 branch, with /hold status.

THANKS SO MUCH!!!!!

@mistyhacks

This comment has been minimized.

mistyhacks commented May 30, 2018

This one actually seems to have merged docs -- just the spreadsheet needs to be filled in. Can you do that, @zparnold ?

@zparnold

This comment has been minimized.

Member

zparnold commented May 30, 2018

@mistyhacks Sure can and sure will!

@justaugustus

This comment has been minimized.

Member

justaugustus commented Jun 4, 2018

@wlan0 @andrewsykim -- We're doing one more sweep of the 1.11 Features tracking spreadsheet.
Would you mind filling in any incomplete / blank fields for this feature's line item?

@andrewsykim

This comment has been minimized.

Member

andrewsykim commented Jun 19, 2018

FYI we're realizing that this features issue is lacking so we will have a more detailed update soon on what we're expecting out of beta (v1.11) and GA (tentatively targeted for v1.13)

@andrewsykim

This comment has been minimized.

Member

andrewsykim commented Jul 3, 2018

Here's an updated status report for this feature, please let me know if anything needs clarification:

Beta (starting v1.11)

  • The common interface used by cloud providers has been well tested and support will not be dropped, though implementation details may change. Any methods that are deprecated should follow the Kubernetes Deprecation Policy.
  • The cloud controller manager has been tested by various cloud providers and is considered safe to use for out-of-tree providers. Features to be deprecated that are part of the cloud controller manager (controllers, component flags, etc) will follow the Kubernetes Deprecation Policy.
  • The cloud controller manager does not run in any cluster by default. It must be explicitly turned on and added like any other control plane component. Instructions for setup may slightly vary per cloud provider. More details here.

Reasoning for Graduation

There were a few things on our TODO list that we wanted to get done before graduating to beta such as collecting E2E tests from all providers & improving out-of-tree storage. However, many of these initiatives require collaboration from external parties that was delaying progress on this effort. In addition, there was uncertainty since we do not develop some of the components we rely on, a good example is whether CSI would be able to meet demands for out-of-tree storage that was on par with in-tree storage support. Though in hindsight we have more confidence in CSI, prior to its beta release it was unclear if it would meet our requirements. With this context in mind, we had decided to graduate to beta because:

  • blocking out-of-tree cloud providers from going beta meant that less in-tree providers will adopt this feature.
  • some goals (like E2E tests from cloud providers) requires a significant amount of collaboration and may unnecessarily block progress for many releases.
  • features that are lacking from the cloud controller manager (mainly storage) would be handled by future projects from other SIGs (e.g. CSI by SIG Storage).

Goals for GA (targetted for v1.13/v1.14)

  • Frequently collect E2E tests results from all in-tree & out-of-tree cloud providers kubernetes/community#2224
  • Cloud Provider Documentation includes:
    • “Getting Started” documentation - outlines the necessary steps required to stand up a Kubernetes cluster.
    • Documentation outlining all cloud provider features such as LoadBalancers, Volumes, etc. There should be docs providing a high-level overview and docs that dig into sufficient details on how each feature works under the hood.
    • Docs should also be centralized in an automated fashion where documentation from all cloud providers are placed into a central location (ideally https://kubernetes.io/docs/home/).
  • A well-documented plan exists for how to migrate a cluster from using in-tree cloud provider to out-of-tree cloud provider, this only applies to AWS, Azure, GCP, OpenStack, and VMWare.
  • All current cloud providers have implemented an out-of-tree solution, deprecation of in-tree code is preferred but not a requirement.

@justaugustus would you mind updating the features description with our progress above?

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Jul 4, 2018

@andrewsykim

This comment has been minimized.

Member

andrewsykim commented Jul 25, 2018

/sig cloud-provider

@justaugustus

This comment has been minimized.

Member

justaugustus commented Jul 26, 2018

Thanks for keeping this up-to-date, @andrewsykim!

@justaugustus justaugustus removed this from the v1.11 milestone Jul 26, 2018

@luxas

This comment has been minimized.

Member

luxas commented Jul 30, 2018

Should we do something @justaugustus to make it tracked/yes?

@justaugustus

This comment has been minimized.

Member

justaugustus commented Jul 31, 2018

@luxas --
Just gathering from what @andrewsykim mentions above, this feature is remaining in Beta until at least 1.13. As the feature is not net new or graduating for the 1.12 release cycle, I've marked it as tracked/no.

Admittedly, that label makes it seems negative, which isn't the intention. I've opened #601 to better clarify all of the labels we use for Features Tracking.

We'll assess the tracked/* status each release cycle or if we get pinged with a change in status.

@kacole2

This comment has been minimized.

Contributor

kacole2 commented Oct 8, 2018

Following up from our conversation in the SIG meeting. This is going to remain in beta for 1.13 unless there are plans to GA the interface. Please correct me @wlan0 @andrewsykim if I'm mistaken.

@andrewsykim

This comment has been minimized.

Member

andrewsykim commented Oct 18, 2018

This feature will remain beta in 1.13. In Q4 we're planning to break this feature down a bit because the "out-of-tree" cloud provider model has a lot of different components and not all the pieces need to be feature gated. We're hoping to have final consensus on this at KubeCON Seattle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment