Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ipv6 support issue for vsphere cloud privider #85886

Closed
hubv opened this issue Dec 4, 2019 · 7 comments · Fixed by #86001
Closed

ipv6 support issue for vsphere cloud privider #85886

hubv opened this issue Dec 4, 2019 · 7 comments · Fixed by #86001
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider.

Comments

@hubv
Copy link
Contributor

hubv commented Dec 4, 2019

What happened:
IP addresses losing in k8s node status in a pure ipv6 vsphere environment.

[root@henry-c1 ~]# kubectl get nodes -o wide
NAME        STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                      KERNEL-VERSION               CONTAINER-RUNTIME
henry-c1    Ready    <none>   6h9m   v1.15.4   <none>        <none>        Red Hat Enterprise Linux Server 7.6 (Maipo)   3.10.0-1062.1.2.el7.x86_64   docker://1.13.1
henry-c2    Ready    <none>   6h9m   v1.15.4   <none>        <none>        Red Hat Enterprise Linux Server 7.6 (Maipo)   3.10.0-1062.1.2.el7.x86_64   docker://1.13.1
henry-c3    Ready    <none>   6h9m   v1.15.4   <none>        <none>        Red Hat Enterprise Linux Server 7.6 (Maipo)   3.10.0-1062.1.2.el7.x86_64   docker://1.13.1
henry-we2   Ready    <none>   6h8m   v1.15.4   <none>        <none>        Red Hat Enterprise Linux Server 7.6 (Maipo)   3.10.0-1062.1.2.el7.x86_64   docker://1.13.1

all interfaces of nodes in this cluster are ipv6 only

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:b0:96:27 brd ff:ff:ff:ff:ff:ff
    inet6 2650:1:87:96::113/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb0:9627/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:b0:0d:22 brd ff:ff:ff:ff:ff:ff
    inet6 2002:da8:8003:801::113/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb0:d22/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:b0:83:bc brd ff:ff:ff:ff:ff:ff
    inet6 2003:da8:8003:801::113/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb0:83bc/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

The error logs in kubelet as the following:

[root@henry-c1 ~]# journalctl -u kubelet -r -o cat
 W1204 10:55:42.811053   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:55:32.797336   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:55:22.778379   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:55:12.758991   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:55:02.742213   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:54:52.723597   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:54:42.707867   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:54:32.690791   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:54:22.675851   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:54:12.656082   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:54:02.637246   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:53:52.619014   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:53:42.601095   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:53:32.583253   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113
W1204 10:53:22.565725   22987 kubelet_node_status.go:505] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: 2650:1:87:96::113

What you expected to happen:
It will be very appreciated if we can get ipv6 addresses from vsphere cloud provider as ipv4 env

How to reproduce it (as minimally and precisely as possible):
deploy k8s cluster in vsphere pure ipv6 envirenment
Anything else we need to know?:
From the code
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/legacy-cloud-providers/vsphere/vsphere.go#L682-L700

                for _, v := range vmMoList[0].Guest.Net {
                                if vs.cfg.Network.PublicNetwork == v.Network {
                                                for _, ip := range v.IpAddress {
                                                                if net.ParseIP(ip).To4() != nil {
                                                                                nodehelpers.AddToNodeAddresses(&addrs,
                                                                                                v1.NodeAddress{
                                                                                                                Type:    v1.NodeExternalIP,
                                                                                                                Address: ip,
                                                                                                }, v1.NodeAddress{
                                                                                                                Type:    v1.NodeInternalIP,
                                                                                                                Address: ip,
                                                                                                },
                                                                                )
                                                                }
                                                }
                                }
                }
                return addrs, nil

we can know that while the provider retrieves VM’s IP, It uses To4() function for all ip addresses from vsphere platform, this function can’t handle ipv6 addresses as mentioned in guild: https://golang.org/pkg/net/#IP.To4

Environment:

  • Kubernetes version (use kubectl version):
[root@henry-c1 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"clean", BuildDate:"2019-09-18T14:51:13Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"clean", BuildDate:"2019-09-18T14:41:55Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
    vsphere
  • OS (e.g: cat /etc/os-release):
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"
  • Kernel (e.g. uname -a):
[root@henry-c1 ~]# uname -a
Linux henry-c1 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 16 14:19:51 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
    ansible
  • Network plugin and version (if this is a network-related bug):
    calico v3.5.6
  • Others:
@hubv hubv added the kind/bug Categorizes issue or PR as related to a bug. label Dec 4, 2019
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Dec 4, 2019
@hubv
Copy link
Contributor Author

hubv commented Dec 4, 2019

If you agree with this I can fix it.

@k8s-ci-robot k8s-ci-robot added sig/contributor-experience Categorizes an issue or PR as relevant to SIG Contributor Experience. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 4, 2019
@k8s-ci-robot
Copy link
Contributor

@hubv: Reiterating the mentions to trigger a notification:
@kubernetes/sig-contributor-experience-bugs

In response to this:

@kubernetes/sig-contributor-experience-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@hubv
Copy link
Contributor Author

hubv commented Dec 4, 2019

@kubernetes/cloud-provider-bugs
/sig cloud-provider

@k8s-ci-robot k8s-ci-robot added the sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. label Dec 4, 2019
@idealhack
Copy link
Member

cc @kubernetes/cloud-provider-vsphere-admins

/remove-sig contributor-experience

@k8s-ci-robot k8s-ci-robot removed the sig/contributor-experience Categorizes an issue or PR as relevant to SIG Contributor Experience. label Dec 5, 2019
@aojea
Copy link
Member

aojea commented Dec 5, 2019

/cc

@hubv
Copy link
Contributor Author

hubv commented Dec 6, 2019

please help to review, I think I can fix the issue by PR#86001
@kubernetes/cloud-provider-bugs
/sig cloud-provider
@aojea
Thanks a lot

@hubv hubv reopened this Dec 6, 2019
@hubv
Copy link
Contributor Author

hubv commented Dec 6, 2019

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider.
Projects
None yet
4 participants