Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
kubelet: lookup node address for external provider if none is set #75229
What type of PR is this?
What this PR does / why we need it:
This should alleviate some of the common bootstrapping problems we see with external providers because the kubelet hosting the control plane does not have node addresses set.
Which issue(s) this PR fixes:
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
Mar 8, 2019
referenced this pull request
Mar 9, 2019
the only issue I can think of would be if the serving certificate approver process refuses to approve the self-detected addresses. in that case, the kubelet could be waiting for approval for the old SANs for a long time (
This was referenced
Apr 19, 2019
referenced this pull request
Jun 20, 2019
I've only tried it with
fwiw I think the bootstraping flow is well identified for the internal cloud provider case already, it just breaks for the external case because of it's assumptions around node addresses. Will let kubeadm maintainers comment on this one though.
We have a KEP but it does not address self-hosting since it's still considered an alpha feature and wasn't part of the initial work on external providers.
@andrewsykim given that there are conformant k8s distributions that run self-hosted, i think we need to capture a recommendation on how those distributions should proceed. i am happy to help rally resources to work through this topic if its not yet defined, but i think we need to account for it as part of the transition.
@derekwaynecarr that's good to know, thank you! I'm curious if conformance actually covers the bootstrapping of self-hosted clusters also. I assume it only covers post-bootstrap which this PR shouldn't change - either way happy to dig into this if needed
hi @andrewsykim ,
if by self-hosting we mean running the kubeadm created control-plane as DaemonSets (and not the popular internet meaning which is running static-pods), then sadly this feature is pretty much unsupported at this point in kubeadm. it remained in alpha for a long time due to a set of major caveats:
reading back, the conclusion was that self-hosting and external CPs will not work in kubeadm:
given my comment above, perhaps kubeadm is not the right tool to base evaluations of this change upon.
i might be lacking context on the external CP case.
so once the control plane is up the external CP can provide a node address at which point the control plane has to be restarted? does that mean that a new api server serving certificate has to be recreated to include a new advertise address?
kubeadm does not have e2e tests for self-hosting at this point. i remember seeing self-hosting tests in the suite, but i don't think these are part of conformance.
Apologies if this wasn't clear - yes I was only referring to kubeadm self-hosting where the control plane is bootstrapped into a DaemonSet. Static Pods with external cloud providers works and is adopted.
This PR was specifically open to address the kubeadm self-hosting case, but it also addresses general trouble shooting issues for the external CP case. More specifically, if a node fails to register with an external cloud provider, it's generally hard to trouble shoot pods running on cluster cause you can't fetch any logs without node addresses being set. With this change, node's have more functionality prior to registration with the cloud provider.
It's not any different from how you would run static pods previously aside from changing some flags (mainly --cloud-provider=external) which we document here https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller. And the external add-on for the cloud provider should be documented by the cloud provider.
[APPROVALNOTIFIER] This PR is APPROVED
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing