kubeadm init and kubeadm join auto-detects external and internal IPs incorrectly

[ghost]

Relatively new user. On a fresh bare metal installation (v1.17.0), I end up with this:

root@k8s-master:~# kubectl get nodes -o wide
NAME           STATUS   ROLES    AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-master     Ready    master   4h9m   v1.17.0   185.x.x.20   <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.4
k8s-worker-1   Ready    <none>   4h3m   v1.17.0   185.x.x.18   <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.4
k8s-worker-2   Ready    <none>   4h3m   v1.17.0   185.x.x.19   <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.4
k8s-worker-3   Ready    <none>   4h2m   v1.17.0   185.x.x.17   <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.4

The machines have a single public IP on net0, and a single internal IP on net1.

Expected behaviour: public IP ends up listed as the external IP, internal IP ends up listed as the internal IP, based on whether the range is in rfc1918, or based on which subnet the default route matches against.

Workaround: Can't find one - there doesn't seem to be a way to configure this? --node-ip seems insufficiently granular.

As a new user, I am unsure what the consequences of this is. Is it safe to continue with the external IPs listed as internal IPs? Is it safe to run without external IPs listed? What will happen? Not intuitive :-/ Perhaps I'm misunderstanding something fundamental here.

[neolit123]

hi,

i've googled this guide which covers the same problem:
https://medium.com/@kanrangsan/how-to-specify-internal-ip-for-kubernetes-worker-node-24790b2884fd

also this is a duplicate of:
kubernetes/kubernetes#58609 (comment)

both suggest --node-ip, which will set the "internal address" of the node.
the Node API objects (that you see when you do kubectl get no) are not managed by kubeadm, but by the kubelet probing a primary network interface for a preferred address for an Node IP.

in your case it finds 185.x.x.x and assigns that to the "InternalIP" address type of the Node objects.
if your network configuration includes a link-local address in the primary interface (e.g. "bogon" addresses 10.0.0.0/8, or 192.168.0.0/16..etc), then the kubelet will pick that.

As a new user, I am unsure what the consequences of this is. Is it safe to continue with the external IPs listed as internal IPs?

unless you have NAT exposing the machine ports externally this is a non-issue.
still best to specify --node-ip explicitly or update your network setup (e.g. swapping interface order).

i'm going to assume the workers joined the cluster on the external address, so this means the master machine allows traffic on the api-server port at least. you might want to do a security evaluation of your network setup on these machines before running the cluster in production.

googling for related guides should give plenty of results.

Is it safe to run without external IPs listed? What will happen? Not intuitive :-/ Perhaps I'm misunderstanding something fundamental here.

the Node "ExternalIP" address type is managed by cloud providers. i cannot find where this is documented, so it might be tribal knowledge and i do agree that the kubectl get no output is confusing. in a manual machine provisioning external-ip will remain "none".

hope this helps.

closing as not a kubeadm bug or FR.
/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

hi,

i've googled this guide which covers the same problem:
https://medium.com/@kanrangsan/how-to-specify-internal-ip-for-kubernetes-worker-node-24790b2884fd

also this is a duplicate of:
kubernetes/kubernetes#58609 (comment)

both suggest --node-ip, which will set the "internal address" of the node.
the Node API objects (that you see when you do kubectl get no) are not managed by kubeadm, but by the kubelet probing a primary network interface for a preferred address for an Node IP.

in your case it finds 185.x.x.x and assigns that to the "InternalIP" address type of the Node objects.
if your network configuration includes a link-local address in the primary interface (e.g. "bogon" addresses 10.0.0.0/8, or 192.168.0.0/16..etc), then the kubelet will pick that.

As a new user, I am unsure what the consequences of this is. Is it safe to continue with the external IPs listed as internal IPs?

unless you have NAT exposing the machine ports externally this is a non-issue.
still best to specify --node-ip explicitly or update your network setup.

given the workers joined the cluster, i'm going to assume your master machine allows traffic on the api-server port at least. you might want to do a security evaluation of your network setup before running this cluster in production.

googling for related guides should give plenty of results.

Is it safe to run without external IPs listed? What will happen? Not intuitive :-/ Perhaps I'm misunderstanding something fundamental here.

the Node "ExternalIP" address type is managed by cloud providers. i cannot find where this is documented, so it might be tribal knowledge and i do agree that the kubectl get no output is confusing. in a manual machine provisioning external-ip will remain "none".

hope this helps.

closing as not a kubeadm bug or FR.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ghost]

Hi neolit123,

Thanks for looking into this, I genuinely appreciate you getting back to me so quickly especially during the holidays and in such detail.

I appreciate that you must have to filter through a lot of crud on here with things that are not issues, and I fully appreciate that Github Issues is not a support forum. In fact, I had already googled this extensively, and had already worked around the issue by specifying --node-ip.

Here's why I still believe this is, if not a bug, at least simply "user unfriendly behaviour", worthy of consideration as a feature request or documentation improvement; if this issue belongs elsewhere I'm happy to file it elsewhere - please let me know the best place to put this. Time permitting I'd even be willing to contribute code or revised documentation, should such a contribution be welcomed.

The environment I'm using is not in any way special - it's a Joyent Triton Private Cloud environment, and I'm deploying the nodes using Terraform + cloud-config to bootstrap, with official Canonical Ubuntu Certified 18.04 images, using the official Kubernetes Getting Started documentation found here:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

The primary interface (net0) has a public 185.x.x.x address, and the secondary interface (net1) has a RFC1918 address (10.28.0.0/24). The default gateway is via net0 on the public IP.

This is not, I would assume, unusual - I have always placed the external IP on the first interface, and the internal on the secondary.

First, there is no mention in the requirements section that interface ordering is important and that the external IP must be on the secondary interface, by convention, for kubernetes to select the internal/external IPs correctly.

Second, there is no way to specify the node-ip to "kubeadm init", despite there being a way to, for example, specify --control-plane-endpoint, --pod-network-cidr or --service-cidr. This requires a post-init kludge, such as the kind I have now implemented in my cloud-config code:

sed -i "s/\"$/ --node-ip=$(hostname -i | xargs -n1 | grep ^10.)/" /var/lib/kubelet/kubeadm-flags.env

Third, after doing this, ExternalIP remains blank, and there's seemingly no way to inform kubelet as to what it actually is. Even if this is of no consequence, it may seem concerning to new users - since my nodes do have both internal and external IPs, does this mean I forfeit some feature or ability later? Should I google for a solution? If this is expected, perhaps it should be officially documented:

root@k8s-master:~# kubectl get nodes -o wide
NAME           STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-master     Ready    master   16h   v1.17.0   10.28.0.14    <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.4
k8s-worker-1   Ready    <none>   16h   v1.17.0   10.28.0.12    <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.4
k8s-worker-2   Ready    <none>   16h   v1.17.0   10.28.0.13    <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.4
k8s-worker-3   Ready    <none>   16h   v1.17.0   10.28.0.11    <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.4

It seems entirely reasonable, and hopefully I'm not the only one, that any product aiming for user-friendlyness should attempt, out of the box, to calculate sane defaults, based on sane logic.

It would also seem reasonable, that if on a public cloud InternalIP and ExternalIP are both populated with the correct addresses, that a private cloud installation should be the same. The fact I can't get ExternalIP to display the correct value, leaves me feeling short-changed and unfairly cheated.

This is currently a baptism of fire for new users, and seemingly unnecessary with some seemingly simple adjustments:

1. Update the getting started documentation; mention the convention of requiring the internal IP to be on the primary interface, or how kubelet handles this
2. Use logic instead of convention to auto-detect InternalIP vs ExternalIP, based on the IP's membership of the standard RFC1918 address spaces
3. Allow specifying --node-ip to "kubeadm init" to avoid post-init kludges
4. (Better) allow specifying --node-internal-ip (aliased to --node-ip) and --node-external-ip to kubeadm init and/or kubelet - perhaps useful for those using RFC1918 addresses on both their "public" and "internal" interfaces (for example this was the case with the Medium post you linked to - this didn't seem applicable to my environment).
5. If ExternalIP is irrelevant, perhaps "kubeadm get nodes -o wide" should display NodeIP instead of "InternalIP" and "ExternalIP"

I hope this is useful and I apologise if I am coming across as obtuse, it's not the intention. I imagine most new users simply figure out you use "--node-ip" and leave it at that, but this did add several hours of head scratching and frustration to an otherwise very pleasant setup experience, and I hope you can see that I am by no means an inexperienced user.

[ghost]

I re-created the master node with the interfaces swapped in the Terraform code (and removed the --node-ip), and it made no difference:

root@k8s-master:~# ifconfig | grep -A 8 ^net[0-9]
net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.28.0.18  netmask 255.255.255.0  broadcast 10.28.0.255
        inet6 fe80::92b8:d0ff:fe44:e454  prefixlen 64  scopeid 0x20<link>
        ether 90:b8:d0:44:e4:54  txqueuelen 1000  (Ethernet)
        RX packets 12  bytes 840 (840.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 936 (936.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

net1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 185.x.x.24  netmask 255.255.255.0  broadcast 185.x.x.255
        inet6 fe80::92b8:d0ff:fed2:b849  prefixlen 64  scopeid 0x20<link>
        ether 90:b8:d0:d2:b8:49  txqueuelen 1000  (Ethernet)
        RX packets 312292  bytes 428466958 (428.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14517  bytes 1242727 (1.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@k8s-master:~# cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        net0:
            addresses:
            - 10.28.0.18/24
            match:
                macaddress: 90:b8:d0:44:e4:54
            mtu: 1500
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
                search: []
            set-name: net0
        net1:
            addresses:
            - 185.x.x.24/24
            gateway4: 185.x.x.254
            match:
                macaddress: 90:b8:d0:d2:b8:49
            mtu: 1500
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
                search: []
            set-name: net1

We still have an external IP on the internal IP, for unknown reasons:

root@k8s-master:~# kubectl get nodes -o wide
NAME         STATUS   ROLES    AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-master   Ready    master   9m27s   v1.17.0   185.x.x.24   <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.4

We do have net1 before net0 with the ip addr output, for unknown reasons:

root@k8s-master:~# ip addr | grep -A 5 net[0-9]:
2: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 90:b8:d0:d2:b8:49 brd ff:ff:ff:ff:ff:ff
    inet 185.x.x.24/24 brd 185.194.89.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::92b8:d0ff:fed2:b849/64 scope link
       valid_lft forever preferred_lft forever
3: net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 90:b8:d0:44:e4:54 brd ff:ff:ff:ff:ff:ff
    inet 10.28.0.18/24 brd 10.28.0.255 scope global net0
       valid_lft forever preferred_lft forever
    inet6 fe80::92b8:d0ff:fe44:e454/64 scope link
       valid_lft forever preferred_lft forever

I suppose "by convention" doesn't work in all environments.

[neolit123]

I appreciate that you must have to filter through a lot of crud on here with things that are not issues, and I fully appreciate that Github Issues is not a support forum. In fact, I had already googled this extensively, and had already worked around the issue by specifying --node-ip.

kubeadm as a user facing deployer often times has to filter bugs and documentation issues in other k8s components. in this case - kubelet and kubectl.

Here's why I still believe this is, if not a bug, at least simply "user unfriendly behaviour", worthy of consideration as a feature request or documentation improvement; if this issue belongs elsewhere I'm happy to file it elsewhere - please let me know the best place to put this. Time permitting I'd even be willing to contribute code or revised documentation, should such a contribution be welcomed.

the appropriate issue tracker for kubelet and kubectl is kubernetes/kubernetes.

but you are outlining separate issues:

1. unclear why kubelet picks an external IP by default instead of a bogon address (tag with /sig node).
2. misleading INTERNAL-IP column in kubectl output, unclear what EXTERNAL-IP is in the context of Node objects (tag with /sig cli).

Second, there is no way to specify the node-ip to "kubeadm init", despite there being a way to, for example, specify --control-plane-endpoint, --pod-network-cidr or --service-cidr. This requires a post-init kludge, such as the kind I have now implemented in my cloud-config code:

not on the CLI at least. we are not adding more flags to kubeadm at this point.
the preferred way to configure kubeadm is using its configuration file:
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2

search for kubeletExtraArgs

kubeadm init --config some-file

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    node-ip: "SOME-IP"

kubeadm join --config some-file

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
nodeRegistration:
  kubeletExtraArgs:
    node-ip: "SOME-IP"

Third, after doing this, ExternalIP remains blank, and there's seemingly no way to inform kubelet as to what it actually is. Even if this is of no consequence, it may seem concerning to new users - since my nodes do have both internal and external IPs, does this mean I forfeit some feature or ability later? Should I google for a solution? If this is expected, perhaps it should be officially documented:

as i've mentioned EXTERNAL-IP in the case of Node objects is filled by cloud providers.
this is AFAIK not documented and it feels like the correct place for this would be the kubectl docs.

We do have net1 before net0 with the ip addr output, for unknown reasons:

seems that i was wrong and the logic for that is documented with source code comments:
https://github.com/kubernetes/kubernetes/blob/0599ca2bcfcae7d702f95284f3c2e2c2978c7772/pkg/kubelet/nodestatus/setters.go#L171-L198

https://github.com/kubernetes/kubernetes/blob/9488fbef64afd580d053347e57819817e51ac0f9/staging/src/k8s.io/apimachinery/pkg/util/net/interface.go#L330-L349

in any case this is something that has to be documented in the user facing kubelet docs - e.g. here:
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
https://github.com/kubernetes/kubernetes/blob/73b2c82b2864eeac5b81eb01e06c5726a11176cf/cmd/kubelet/app/options/options.go#L368

[ghost]

Thanks for following up Lubomir especially during the festive period, especially with such detail - it's much appreciated. I will try and find some time to follow up your suggestions regarding documentation improvements.

The InitConfiguration and JoinConfiguration work perfectly, which is a neat/clean solution.

However after reading the logic here it became apparent that there's a simpler workaround, which is that the local hostname should resolve to the internal IP address instead of the external one. On my nodes, it was resolving to the external one.

And indeed, adding that mapping in /etc/hosts prior to the kubeadm init/join command solved the issue. I achieved this with this command:

echo $(hostname -i | xargs -n1 | grep ^10.) $(hostname) >> /etc/hosts

I'm writing up a blog post about this and will append the link to this issue once done.

Thanks again!

[neolit123]

However after reading the logic here it became apparent that there's a simpler workaround, which is that the local hostname should resolve to the internal IP address instead of the external one. On my nodes, it was resolving to the external one

i was assuming that this will work for you.

i did not find the precedence documented in the k8s.io website, so if you are willing you can also contribute documentation changes. the website repository is kubernetes/website.

[ghost]

Thanks - I'll try and find some time to do so, I'm keen to contribute to the project in some form.

Here's a post about it to help new users hitting the same thing:

https://medium.com/@aleverycity/kubeadm-init-join-and-externalip-vs-internalip-519519ddff89

[MurzNN]

@alaslums Great thanks for so useful article - it resolved my slow cluster performance problem!

It's very not obvious that Kubernetes uses external IPs by default, instead of internals! So with default settings - all nodes start using external network interface for connectivity instead of LAN, and can got the external channel limits like "100 Mbit" limit from data centers.

As result, before rebuilding the cluster, I had very low performance of SNI because it worked over external network, not internal LAN!