Request time out from container to other container using hostIP:hostPort on same host with portmap CNI chained

[splushii]

Bug report

General Information

• Cilium version (run cilium version)

Client: 1.6.4 8048d320a 2019-11-27T17:00:12+01:00 go version go1.12.13 linux/amd64
Daemon: 1.6.4 8048d320a 2019-11-27T17:00:12+01:00 go version go1.12.13 linux/amd64

• Kernel version (run uname -a)

Linux tender-drake 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Orchestration system version in use (e.g. kubectl version, Mesos, ...)

Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:09:21Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

• Link to relevant artifacts (policies, deployments scripts, ...)
• Upload a system dump (run curl -sLO releases.cilium.io/tools/cluster-diagnosis.zip && python cluster-diagnosis.zip sysdump and then attach the generated zip file)

( The sysdump was too big to attach. Please let me know if it's needed and I'll send or make it available somehow. )

How to reproduce the issue

1. Install Cilium 1.6.4 using Helm chart from https://github.com/cilium/cilium/archive/v1.6.4.tar.gz with the following values:

---
global:
  tag: v1.6.4
  cluster:
    name: "k8slab"
    id: "1"
  cni:
    install: true
    chainingMode: portmap
  ipv4:
    enabled: true
  ipv6:
    enabled: false
  prometheus:
    enabled: false

2. Deploy container using hostPort

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
          - containerPort: 80
            hostPort: 35000

3. Deploy some debug container connecting to the hostPort service

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ubuntu
spec:
  selector:
    matchLabels:
      app: ubuntu
  template:
    metadata:
      labels:
        app: ubuntu
    spec:
      containers:
      - name: ubuntu
        image: ubuntu
        command:
          - '/bin/bash'
          - '-c'
          - '--'
        args:
          - 'apt-get update -y; apt-get install -y netcat curl; while true; do sleep 30; done;'

4. Select a hostPort pod and debug pod on the same host (e.g. nginx-2cb8j and ubuntu-vf44z in the example below)

$ kubectl get po -owide
NAME           READY   STATUS    RESTARTS   AGE   IP             NODE            NOMINATED NODE   READINESS GATES
nginx-2cb8j    1/1     Running   0          24m   172.16.3.16    tender-drake    <none>           <none>
nginx-7b4wr    1/1     Running   0          24m   172.16.7.154   mutual-dodo     <none>           <none>
nginx-drtk8    1/1     Running   0          24m   172.16.8.63    full-evil       <none>           <none>
nginx-dzk7n    1/1     Running   0          24m   172.16.4.59    right-ferret    <none>           <none>
nginx-l89zr    1/1     Running   0          24m   172.16.6.17    loving-gopher   <none>           <none>
nginx-l9ntn    1/1     Running   0          24m   172.16.5.128   strong-caiman   <none>           <none>
ubuntu-bb44g   1/1     Running   0          20m   172.16.5.210   strong-caiman   <none>           <none>
ubuntu-hhpv7   1/1     Running   0          20m   172.16.7.223   mutual-dodo     <none>           <none>
ubuntu-n2p99   1/1     Running   0          20m   172.16.4.245   right-ferret    <none>           <none>
ubuntu-rlv5c   1/1     Running   0          20m   172.16.6.156   loving-gopher   <none>           <none>
ubuntu-tjr88   1/1     Running   0          20m   172.16.8.70    full-evil       <none>           <none>
ubuntu-vf44z   1/1     Running   0          20m   172.16.3.47    tender-drake    <none>           <none>

5. Try to contact hostPort pod via its hostIP
   Using the above example:

$ kubectl get pod nginx-2cb8j -ojsonpath='{.status.hostIP}'
10.24.214.230
$ kubectl exec -ti ubuntu-vf44z bash
root@ubuntu-vf44z:/# curl 10.24.214.230:35000 -v
* Rebuilt URL to: 10.24.214.230:35000/
*   Trying 10.24.214.230...
* TCP_NODELAY set
* connect to 10.24.214.230 port 35000 failed: Connection timed out
* Failed to connect to 10.24.214.230 port 35000: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to 10.24.214.230 port 35000: Connection timed out

The cilium monitor output is (where id 568 is the debug pod ubuntu-vf44z):

cilium monitor --related-to 568
Listening for events on 32 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
-> host from flow 0xab4466b0 identity 111636->1 state new ifindex cilium_net: 172.16.3.47:49290 -> 10.24.214.230:35000 tcp SYN
-> endpoint 568 flow 0x85c0bb7d identity 111706->111636 state new ifindex lxc2faa014a7904: 172.16.3.16:80 -> 172.16.3.47:49290 tcp SYN, ACK
-> host from flow 0x65945f77 identity 111636->1 state established ifindex cilium_net: 172.16.3.47:49290 -> 10.24.214.230:35000 tcp SYN
-> host from flow 0x405d06ef identity 111636->1 state established ifindex cilium_net: 172.16.3.47:49290 -> 10.24.214.230:35000 tcp SYN
-> host from flow 0xc974addb identity 111636->1 state established ifindex cilium_net: 172.16.3.47:49290 -> 10.24.214.230:35000 tcp SYN
-> host from flow 0x73876df9 identity 111636->1 state established ifindex cilium_net: 172.16.3.47:49290 -> 10.24.214.230:35000 tcp SYN

Contacting the hostPort pod using its ClusterIP works:

root@ubuntu-vf44z:/# curl 172.16.3.16:80 -I
HTTP/1.1 200 OK
...

and contacting a pod running on another host using the hostIP and hostPort works:

$ kubectl get pod nginx-7b4wr -ojsonpath='{.status.hostIP}'
10.24.214.232
root@ubuntu-vf44z:/# curl 10.24.214.232:35000 -I   
HTTP/1.1 200 OK
...

It seems the issue only arises when the source and destination pod is on the same host and the hostIP:hostPort is used.

[croissong]

Same issue here. With cilium v1.6.3 the same test case is working.

[TimEvens]

We are having the same problem, even with "Cilium 1.6.90 ca7f68526 2019-11-19T08:00:31-08:00 go version go1.13.4 linux/amd64".

Use-case: Consul Agent - PODs access consul via the hostIP that the POD is running on, eg. http://${HOST_IP}:8500/v1/...

portmap commit attempted to fix the problem for hair-pinning, but it looks to me that they only fixed it for the host itself and PODs that are using hostNetworking. PODs that use cluster networking are not correctly being masqueraded when a local pod to the host tries to access the hostIP:hostPort (e.g., DNAT).

In the case of consul-agent using port 8500, we are able to work around the issue by adding the below:

iptables -t nat -I CNI-HOSTPORT-DNAT -p tcp -m tcp --dport 8500 -j KUBE-MARK-MASQ

If portmap is configured with defaults, we can also use:

iptables -t nat -I CNI-HOSTPORT-DNAT -p tcp -m tcp --dport 8500 -j CNI-HOSTPORT-SETMARK

It seems to me that this is an issue with portmap, but maybe there is something Cilium can do...?? :)

[lkysow]

I've reproduced this on Digital Ocean using the Consul helm chart. Every pod in other nodes can access the hostIP:hostPort except for the pods on that node itself.

[jdef]

the workaround in the description above doesn't seem to be compatible with a CiliumNetworkPolicy that enforces ingress using fromEntities: [ host ]

EDIT

.. meaning that I need to remove such policy in order for the workaround to be effective

[joestringer]

One additional snippet of information to add to @jdef's observations above is that the source identity in that case was 2 (world).

[tgraf]

The commit which likely broke this is 4ae2ead. The commit is needed to make externalTrafficPolicy=local work.

Traffic pattern looks like this:

-> stack flow 0x9a6ba3e1 identity 34607->1 state new ifindex 0 orig-ip 0.0.0.0: 192.168.131.170:60406 -> 192.168.52.196:35000 tcp SYN
-> endpoint 3150 flow 0x9a6ba3e1 identity 34607->2352 state new ifindex 0 orig-ip 192.168.131.170: 192.168.131.170:60406 -> 192.168.144.37:80 tcp SYN

The forward-path looks correct. The identity is correctly preserved.

-> stack flow 0x6a800030 identity 2352->34607 state reply ifindex 0 orig-ip 0.0.0.0: 192.168.144.37:80 -> 192.168.131.170:60406 tcp SYN, ACK
-> endpoint 3549 flow 0x6a800030 identity 2->34607 state reply ifindex 0 orig-ip 192.168.52.196: 192.168.52.196:35000 -> 192.168.131.170:60406 tcp SYN, ACK

The reply path loses the identity because the traffic seems to gets masqueraded.

[tgraf]

Prior to commit 4ae2ead, Cilium implemented a relatively aggressive masquerading behavior of traffic from the local host. This simplified integration with components such as portmap and others which implement their own DNAT rules but often lack SNAT to force reply traffic back through the same path.

Starting with commit 4ae2ead, this was relaxed in order to allow for proper behavior of externalTrafficPolicy=local. The commit removed the following masquerade rule:

-                       // Masquerade all traffic from the host into the
-                       // local Cilium cluster range if the source is not
-                       // in the cluster range and DNAT has been
-                       // applied.  These conditions are met by traffic
-                       // redirected via hostports from non-cluster sources.
-                       // The SNAT to the cluster address is needed so that
-                       // the return traffic from a host proxy (when used) is
-                       // routed back via the cilium_host device also
-                       // when the source address is originally
-                       // outside of the cluster range.
-                       //
-                       // The following conditions must be met:
-                       // * Must be targeted to an IP that IS local
-                       // * May not originate from any IP inside of the cluster range
-                       // * Must have DNAT applied (k8s hostport, etc.)
-
-                       if err := runProg("iptables", append(
-                               m.waitArgs,
-                               "-t", "nat",
-                               "-A", ciliumPostNatChain,
-                               "!", "-s", node.GetIPv4ClusterRange().String(),
-                               "-o", localDeliveryInterface,
-                               "-m", "conntrack", "--ctstate", "DNAT",
-                               "-m", "comment", "--comment", "cilium hostport cluster masquerade",
-                               "-j", "SNAT", "--to-source", node.GetHostMasqueradeIPv4().String()), false); err != nil {
-                               return err
-                       }

With the removal of this masquerade rule, it is no longer guaranteed for reply traffic to be routed symmetrically on the reply path. It was expected that portmap would ensure for this, given that it depends on seeing the reply traffic. It seems that portmap assumes that the routing path is always symmetric.

The following PRs will fix this issues:

• bpf: Preserve source identity for hairpin via stack #10926
• datapath/iptables: Masquerade hairpin traffic that traversed the stack #10928