1.5.2 on GKE breaks multiple LoadBalancers, because iptables processes host ports before service ports

[miracle2k]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): 1.4.8 and 1.5.2

Environment:

• Cloud provider or hardware configuration: Google Container Engine

What happened:

My cluster has three services of type=LoadBalancer; all of them respond on the same port. One of them is the official nginx ingress controller. So I have three external ips that have port 80 routed to three different Kubernetes services, and their respective pods.

This worked fine in 1.4.8. After I upgraded my cluster to 1.5.2, I saw the following: A request to one of the three IPs would frequently be directed to the wrong service/pods. After I downgraded to 1.4.8, the correct behaviour was restored. The service selectors are all set up correctly, and didn't change.

I believe this is because of the following iptables rules that kube-proxy sets up:

-A PREROUTING -m comment --comment "kube hostport portals" -m addrtype --dst-type LOCAL -j KUBE-HOSTPORTS
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES

-A KUBE-HOSTPORTS -p tcp -m comment --comment "nginx-ingress-controller-dr5g6_default hostport 80" -m tcp --dport 80 -j KUBE-HP-GPK4AFVKWROWERLM
-A KUBE-HOSTPORTS -p tcp -m comment --comment "nginx-ingress-controller-dr5g6_default hostport 443" -m tcp --dport 443 -j KUBE-HP-G6DMAJMA6N7NEYP4

As you can see, the hostport rules are processed before the service port rules. If I manually delete these rules from the KUBE-HOSTPORTS chain, the problem goes away.

By my best guess, what happens, then, is this (assuming two LoadBalancer services A and B both serving the same port).

1. A connection for service B comes into the Google Cloud, and is sent to any one host.
2. For most hosts, this is fine; the iptables rules on that host will direct the connection to the right service.
3. Sometimes, the connection is routed to the host on which the pods for service A are running. On this host only the KUBE-HOSTPORTS chains contains a rule that redirects the port to service A; iptables never processes the service chain, which would correctly direct the connection to service B.

To verify this a bit more, I created two clusters, with 1.4.8 and 1.5.2, and startet the nginx-ingress-controller on both. The iptables rules generated by 1.5.2 are the ones above. For 1.4.8 it looks like this:

-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m comment --comment "kube hostport portals" -m addrtype --dst-type LOCAL -j KUBE-HOSTPORTS

It seems to me that here, the rules for the service portals are checked first, thus avoiding this problem.

Anything else we need to know:

Wild guess - could it be related to? 5389a74

[miracle2k]

While it seems the change described here really did happen in 1.5, the real problem was the fact that two of my services set NodePorts.