ExternalIPs always empty

[uablrek]

The ExternalIPs is always empty even when a service have externalIPs;

$ kubectl get svc
NAME          TYPE           CLUSTER-IP   EXTERNAL-IP        PORT(S)                         AGE
kubernetes    ClusterIP      12.0.0.1     <none>             443/TCP                         72s
mserver       LoadBalancer   12.0.63.60   10.0.0.1,1000::1   5001:31093/TCP,5003:30150/TCP   43s
mserver-udp   LoadBalancer   12.0.207.6   10.0.0.2,1000::2   5001:31084/UDP                  43s
$  print-state 
I0824 09:12:48.195326    2759 client.go:168] connecting to 127.0.0.1:12090
# ------------------------------------------------------------------------
# 2021-08-24 09:12:48.210194493 +0000 UTC m=+0.024634303
#
&{Namespace:"default"  Name:"kubernetes"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"12.0.0.1"}  ExternalIPs:{}}  Ports:{Name:"https"  Protocol:TCP  Port:443  TargetPort:6443} [IPs:{V4:"192.168.1.1"}]}
&{Namespace:"default"  Name:"mserver"  Type:"LoadBalancer"  IPs:{ClusterIPs:{V4:"12.0.63.60"}  ExternalIPs:{}}  Ports:{Name:"mconnect"  Protocol:TCP  Port:5001  NodePort:31093  TargetPort:5001}  Ports:{Name:"ctraffic"  Protocol:TCP  Port:5003  NodePort:30150  TargetPort:5003} [IPs:{V4:"11.0.2.2"}  Local:true IPs:{V4:"11.0.3.2"} IPs:{V4:"11.0.1.2"} IPs:{V4:"11.0.4.2"}]}
&{Namespace:"default"  Name:"mserver-udp"  Type:"LoadBalancer"  IPs:{ClusterIPs:{V4:"12.0.207.6"}  ExternalIPs:{}}  Ports:{Name:"mconnect-udp"  Protocol:UDP  Port:5001  NodePort:31084  TargetPort:5001} [IPs:{V4:"11.0.2.2"}  Local:true IPs:{V4:"11.0.3.2"} IPs:{V4:"11.0.1.2"} IPs:{V4:"11.0.4.2"}]}

So, regardless of backend kpng can't support external access.

[mcluseau]

I don't reproduce:

$ kubectl get svc -A
NAMESPACE     NAME         TYPE        CLUSTER-IP       EXTERNAL-IP                 PORT(S)         AGE
default       kubernetes   ClusterIP   192.168.48.1     <none>                      443/TCP         14m
default       nginx        ClusterIP   192.168.61.211   1.1.1.1,fd42:6b70:6e67::1   80/TCP          7m17s
kube-system   coredns      ClusterIP   192.168.48.2     <none>                      53/UDP,53/TCP   14m

$ dist/print-state 
I0827 14:15:46.901184  830085 client.go:168] connecting to 127.0.0.1:12090
# ------------------------------------------------------------------------
# 2021-08-27 14:15:46.902536054 +0200 CEST m=+0.002547050
#
&{Namespace:"default"  Name:"kubernetes"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"192.168.48.1"}  ExternalIPs:{}}  Ports:{Name:"https"  Protocol:TCP  Port:443  TargetPort:6443} [IPs:{V4:"192.168.10.11"} IPs:{V4:"192.168.10.13"} IPs:{V4:"192.168.10.12"}]}
&{Namespace:"default"  Name:"nginx"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"192.168.61.211"}  ExternalIPs:{V4:"1.1.1.1"  V6:"fd42:6b70:6e67::1"}}  Ports:{Protocol:TCP  Port:80  TargetPort:80} [IPs:{V4:"192.168.64.8"} IPs:{V4:"192.168.64.7"}]}
&{Namespace:"kube-system"  Name:"coredns"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"192.168.48.2"}  ExternalIPs:{}}  Ports:{Name:"dns"  Protocol:UDP  Port:53  TargetPort:53}  Ports:{Name:"dns-tcp"  Protocol:TCP  Port:53  TargetPort:53} []}
^CI0827 14:15:50.068824  830085 client.go:154] got signal interrupt, stopping

$ dist/kpng-nodelog 
I0827 14:21:51.024722  830886 client.go:168] connecting to 127.0.0.1:12090
< req nwrk at 2021-08-27 14:21:51.02509846 +0200 CEST m=+0.003229086
< recv at 2021-08-27 14:21:51.025717988 +0200 CEST m=+0.003848624
+ Set:ServicesSet  Path:"default/kubernetes" -> Namespace:"default"  Name:"kubernetes"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"192.168.48.1"}  ExternalIPs:{}}  Ports:{Name:"https"  Protocol:TCP  Port:443  TargetPort:6443}
+ Set:ServicesSet  Path:"default/nginx" -> Namespace:"default"  Name:"nginx"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"192.168.61.211"}  ExternalIPs:{V4:"1.1.1.1"  V6:"fd42:6b70:6e67::1"}}  Ports:{Protocol:TCP  Port:80  TargetPort:80}
+ Set:ServicesSet  Path:"kube-system/coredns" -> Namespace:"kube-system"  Name:"coredns"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"192.168.48.2"}  ExternalIPs:{}}  Ports:{Name:"dns"  Protocol:UDP  Port:53  TargetPort:53}  Ports:{Name:"dns-tcp"  Protocol:TCP  Port:53  TargetPort:53}
+ Set:EndpointsSet  Path:"default/kubernetes/23dfdcb0905ef710" -> IPs:{V4:"192.168.10.11"}
+ Set:EndpointsSet  Path:"default/kubernetes/6a431dea888d9999" -> IPs:{V4:"192.168.10.13"}
+ Set:EndpointsSet  Path:"default/kubernetes/f1658b843ad13c39" -> IPs:{V4:"192.168.10.12"}
+ Set:EndpointsSet  Path:"default/nginx/1b0af828d197f91d" -> IPs:{V4:"192.168.64.8"}
+ Set:EndpointsSet  Path:"default/nginx/1d4e3871514536fb" -> IPs:{V4:"192.168.64.7"}
> sync after 445.488µs

kpng api to-file gives this global state: global-state.yaml.gz

kubectl version gives:
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.9", GitCommit:"7a576bc3935a6b555e33346fd73ad77c925e9e4a", GitTreeState:"clean", BuildDate:"2021-07-15T20:56:38Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}

[mcluseau]

same with a LB:

$ kubectl get svc
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP                 PORT(S)        AGE
kubernetes   ClusterIP      192.168.48.1     <none>                      443/TCP        23m
nginx        ClusterIP      192.168.61.211   1.1.1.1,fd42:6b70:6e67::1   80/TCP         15m
nginx-lb     LoadBalancer   192.168.51.238   1.1.1.2,fd42:6b70:6e67::2   80:30314/TCP   9s

$ dist/print-state 
I0827 14:24:04.662248  831232 client.go:168] connecting to 127.0.0.1:12090
# ------------------------------------------------------------------------
# 2021-08-27 14:24:04.663607811 +0200 CEST m=+0.002311830
#
&{Namespace:"default"  Name:"kubernetes"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"192.168.48.1"}  ExternalIPs:{}}  Ports:{Name:"https"  Protocol:TCP  Port:443  TargetPort:6443} [IPs:{V4:"192.168.10.11"} IPs:{V4:"192.168.10.13"} IPs:{V4:"192.168.10.12"}]}
&{Namespace:"default"  Name:"nginx"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"192.168.61.211"}  ExternalIPs:{V4:"1.1.1.1"  V6:"fd42:6b70:6e67::1"}}  Ports:{Protocol:TCP  Port:80  TargetPort:80} [IPs:{V4:"192.168.64.8"} IPs:{V4:"192.168.64.7"}]}
&{Namespace:"default"  Name:"nginx-lb"  Type:"LoadBalancer"  IPs:{ClusterIPs:{V4:"192.168.51.238"}  ExternalIPs:{V4:"1.1.1.2"  V6:"fd42:6b70:6e67::2"}}  Ports:{Protocol:TCP  Port:80  NodePort:30314  TargetPort:80} [IPs:{V4:"192.168.64.8"} IPs:{V4:"192.168.64.7"}]}
&{Namespace:"kube-system"  Name:"coredns"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"192.168.48.2"}  ExternalIPs:{}}  Ports:{Name:"dns"  Protocol:UDP  Port:53  TargetPort:53}  Ports:{Name:"dns-tcp"  Protocol:TCP  Port:53  TargetPort:53} []}

[uablrek]

Strange... I wonder what the difference is in my cluster.

The external IPs are fond in svc.Status.LoadBalancer.Ingress[i] if set by a cloud-provider or the metallb controller (or assign-lb-ip that I use for test).

I made a grep and found "Ingress" only in backend/iptables.

kpng > grep -r Ingress .
./backends/iptables/service.go: for _, ing := range info.loadBalancerStatus.Ingress {
./backends/iptables/service.go: //TODO : CHECK Load Balancer Ingress IPs , service.Status.LoadBalancer.Ingress
./backends/iptables/service.go: // Obtain Load Balancer Ingress IPs
./backends/iptables/service.go: // for _, ing := range service.Status.LoadBalancer.Ingress {
./backends/iptables/service.go: //              info.loadBalancerStatus.Ingress = append(info.loadBalancerStatus.Ingress, v1.LoadBalancerIngress{IP: ip})

[uablrek]

This is a bug.

https://github.com/kubernetes-sigs/kpng/blob/b26c64af90b6b54642d0d3ae93250d005b81cb20/jobs/kube2store/service-event-handler.go#L42

Only svc.Spec.ExternalIPs are collected, but they are actually not recommended to be used and has been subject to many discussions. Basically external addresses shall not be specified by the used in the service manifest.

The external addresses are normally taken from service.Status.LoadBalancer.Ingress which is usually set by the cloud-provider or the metallb-controller, but not by the user.

For test I made a small test;

func lbIPs(svc *v1.Service) []string {
	ips := []string{}
	for _, i := range svc.Status.LoadBalancer.Ingress {
		ips = append(ips, i.IP)
	}
	return ips
}
// ....
			ExternalIPs: localnetv1.NewIPSet(lbIPs(svc)...),

The correct way would be to concat both svc.Spec.ExternalIPs and svc.Status.LoadBalancer.Ingress into ExternalIPs.

[jayunit100]

/close
and moved to #68

[k8s-ci-robot]

@jayunit100: Closing this issue.

In response to this:

/close
and moved to #68

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.