kubelet: lookup node address for external provider if none is set

[andrewsykim]

What type of PR is this?
/kind feature

What this PR does / why we need it:
This changes the node address lookup in the kubelet for external providers. If --cloud-provider=external and the kubelet has no node address set already, we attempt to fill its node address with what we can find at runtime (hostname + internal IP). In most cloud providers, whatever hostname/internal IP the kubelet sets at startup will get overwritten later.

This should alleviate some of the common bootstrapping problems we see with external providers because the kubelet hosting the control plane does not have node addresses set.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Attempt to set the kubelet's hostname & internal IP if `--cloud-provider=external` and no node addresses exists

[andrewsykim]

/sig node
/assign @mtaufen @mcrute @cheftako

[andrewsykim]

/sig cloudprovider
/area cloudprovider

[andrewsykim]

/priority important-longterm

[andrewsykim]

@liggitt do you see this being a problem for kubelet serving certificates with node addresses? (#65594)

[cheftako]

We were just in an externalCloudProvider if statement. Why not fold this into the end of that if statement?

[andrewsykim]

good catch

[andrewsykim]

fixed!

[liggitt]

@liggitt do you see this being a problem for kubelet serving certificates with node addresses?

the only issue I can think of would be if the serving certificate approver process refuses to approve the self-detected addresses. in that case, the kubelet could be waiting for approval for the old SANs for a long time (manager#rotateCerts() looks like it waits up to 15 minutes before giving up and re-requesting)

[andrewsykim]

@mtaufen can I get a review from you please?

[derekwaynecarr]

is there a pointer i can reference for how external cloud provider with self hosted control planes are recommended for bootstrapping?

[andrewsykim]

is there a pointer i can reference for how external cloud provider with self hosted control planes are recommended for bootstrapping?

I've only tried it with kubeadm, but as far as I can tell there isn't much on this because it doesn't work. The self-hosted components rely on the kubelet/node for the API server's advertise address which relies on the node object having an InternalIP or ExternalIP set. For internal cloud provider this works fine because it queries instance metadata for node addresses, for the external case it won't get node addresses until the external provider is running which it can't without the control plane up. cc @neolit123 @fabriziopandini

[andrewsykim]

If you know the API server advertise address ahead of time I think you can work around it but it kind of defeats the purpose of self-hosting if you need to manually resync that value rather than relying on what Kubernetes says is the node's address.

[derekwaynecarr]

/hold

we need to ensure that we have a clear bootstrapping flow identified before proceeding on this imo.

is there a kep that we can iterate or extend upon for this as part of overall external cloud provider transition?

[andrewsykim]

we need to ensure that we have a clear bootstrapping flow identified

fwiw I think the bootstraping flow is well identified for the internal cloud provider case already, it just breaks for the external case because of it's assumptions around node addresses. Will let kubeadm maintainers comment on this one though.

is there a kep that we can iterate or extend upon for this as part of overall external cloud provider transition?

We have a KEP but it does not address self-hosting since it's still considered an alpha feature and wasn't part of the initial work on external providers.

[derekwaynecarr]

@andrewsykim given that there are conformant k8s distributions that run self-hosted, i think we need to capture a recommendation on how those distributions should proceed. i am happy to help rally resources to work through this topic if its not yet defined, but i think we need to account for it as part of the transition.

[andrewsykim]

@derekwaynecarr that's good to know, thank you! I'm curious if conformance actually covers the bootstrapping of self-hosted clusters also. I assume it only covers post-bootstrap which this PR shouldn't change - either way happy to dig into this if needed

[neolit123]

hi @andrewsykim ,

This should alleviate some of the common bootstrapping problems we see with external providers because the kubelet hosting the control plane does not have node addresses set. kubeadm self-hosting is the first example that comes to mind.

if by self-hosting we mean running the kubeadm created control-plane as DaemonSets (and not the popular internet meaning which is running static-pods), then sadly this feature is pretty much unsupported at this point in kubeadm. it remained in alpha for a long time due to a set of major caveats:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/self-hosting/#caveats

reading back, the conclusion was that self-hosting and external CPs will not work in kubeadm:
ref: #60607 (comment)

I've only tried it with kubeadm, but as far as I can tell there isn't much on this because it doesn't work. The self-hosted components rely on the kubelet/node for the API server's advertise address which relies on the node object having an InternalIP or ExternalIP set.

given my comment above, perhaps kubeadm is not the right tool to base evaluations of this change upon.

For internal cloud provider this works fine because it queries instance metadata for node addresses, for the external case it won't get node addresses until the external provider is running which it can't without the control plane up

i might be lacking context on the external CP case.

so once the control plane is up the external CP can provide a node address at which point the control plane has to be restarted? does that mean that a new api server serving certificate has to be recreated to include a new advertise address?

I'm curious if conformance actually covers the bootstrapping of self-hosted clusters

kubeadm does not have e2e tests for self-hosting at this point. i remember seeing self-hosting tests in the suite, but i don't think these are part of conformance.

[derekwaynecarr]

i am particularly concerned about using static pods with external cloud controller manager. is there a concern with that approach?

[andrewsykim]

if by self-hosting we mean running the kubeadm created control-plane as DaemonSets (and not the popular internet meaning which is running static-pods), then sadly this feature is pretty much unsupported at this point in kubeadm.

Apologies if this wasn't clear - yes I was only referring to kubeadm self-hosting where the control plane is bootstrapped into a DaemonSet. Static Pods with external cloud providers works and is adopted.

[derekwaynecarr]

@andrewsykim thanks for clarifying. is there a document/kep that describes how a static pod approach is enabled?

[andrewsykim]

This PR was specifically open to address the kubeadm self-hosting case, but it also addresses general trouble shooting issues for the external CP case. More specifically, if a node fails to register with an external cloud provider, it's generally hard to trouble shoot pods running on cluster cause you can't fetch any logs without node addresses being set. With this change, node's have more functionality prior to registration with the cloud provider.

[andrewsykim]

@andrewsykim thanks for clarifying. is there a document/kep that describes how a static pod approach is enabled?

It's not any different from how you would run static pods previously aside from changing some flags (mainly --cloud-provider=external) which we document here https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller. And the external add-on for the cloud provider should be documented by the cloud provider.

[derekwaynecarr]

/hold cancel

apologies for initial confusion. this makes sense.

/approved
/lgtm

[derekwaynecarr]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewsykim, derekwaynecarr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/kubelet/OWNERS [derekwaynecarr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[mkjpryor-stfc]

Is this in a release yet?

[andrewsykim]

yes, this is included in v1.16