CI: K8sDatapathConfig Encapsulation Check vxlan connectivity with per-endpoint routes #13774
Assignees
Comments
nebril
added
area/CI
|
focused test run also fails in the same way on 1.14, so it's unlikely this is infra/ci related: https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Kernel-Focus/100/ |
|
It looks like this test sometimes also fails in pipelines other than k8s-all. It failed before in #14097 with the same error message. It also just failed in #14525. That last PR should only affect the IPSec code path so I'm fairly confident it's a flake. |
|
I've hit this in #14571 via https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.9/359/. |
pchaigno
changed the title
K8sDatapathConfig Encapsulation Check vxlan connectivity with per-endpoint routes fails on k8s-all
|
I managed to reproduce this locally with: NFS=1 NETNEXT=1 KUBEPROXY=0 ginkgo -v --focus "K8sDatapathConfig.*Check vxlan connectivity with per-endpoint routes" -- -cilium.provision=false -cilium.holdEnvironment=true -cilium.skipLogs -cilium.runQuarantinedAlthough I'm seeing a different failure: |
|
Reproduced the actual failure: The deployment of coredns seems to be failing due to: although coredns looks healthy: The IP of the resolver is correct: But the dig command is failing: Looking at coredns logs we can see that it is receiving the requests: So the responses are getting dropped for some reason. Nothing interesting from tcpdump running on the hostns: cilium monitor: |
|
Running the same And looking at tcpdump on k8s2 (where coredns is running) I can also see the response when I run dig from the k8s1 node: so the response is somehow getting lost while being tunneled from k8s2 to k8s1 edit: the response we are seeing is from the lxc device. If dump the traffic from the |
|
Restarted Cilium with k8s1: and on k8s2: |
|
Underlying problem: both nodes have the same IP for the so k8s2 is blackholing the traffic to k8s1. Possible explanation to the flakiness: the test fails only when coredns is scheduled on k8s2 |
|
Setting |
|
I validated that the following diff fixes the flake locally: Matching this iptables rule causes packets to bypass masquerading and leave with |
pchaigno
added a commit
that referenced
this issue
Feb 9, 2021
--- Analysis --- In tunneling mode, our CILIUM_POST_nat chain is currently as follows. 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat ! -o cilium_host -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN 3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT 4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o cilium_host -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 The second rule implements an early exit from the chain, as none of the subsequent rules match on output interfaces other than cilium_host. Once per-endpoint routes are enabled in addition to tunneling, the chain changes. The second and fifth rules now match on lxc+ as the output interface: 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat ! -o lxc+ -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN 3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT 4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 Commit c496e25 ("eni: Support masquerading") implemented that change, based on the fact that with per-endpoint routes, packets are routed directly to lxc devices without going through cilium_host. Nevertheless, the fourth rule still matches on cilium_host and therefore becomes noop. At the time c496e25 was implemented, this change was correct because the fourth rule is only present when tunneling is enabled and per-endpoint routes were not compatible with tunneling. Commit 3179a47 ("datapath: Support enable-endpoint-routes with encapsulation") however made those options compatible and the above chain possible. --- Fix --- Ideally, we would update the second rule when running with tunneling and per-endpoint routes, to be '! -o lxc+ ! -o cilium_host'. Iptables however doesn't support multiple output interface matchers. This commit implements a different fix and drops the second rule. Since subsequent SNATing rules already match on an output interface, the second rule is unnecessary. With tunneling and per-endpoint routes, the table now looks like: 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquerade" -j ACCEPT 3. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 4. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 --- Bug Impact --- This lack of masquerading can cause issues for example when trying to connect to a VIP with a remote backend from the hostns in our test VMs: 1. DNS request is made to VIP 10.96.0.10. 2. 10.0.2.15, the IP of enp0s3 (default route) is assigned as source IP. 3. kube-proxy translates VIP to backend IP on different node, e.g. 10.0.0.87. 3. Packet is sent to cilium_host as per the ip routes. 4. The packet is not masqueraded because it matches rule 2 in the bogus iptables chain (i.e., cilium_host != lxc+). 5. The packet arrives as 10.0.2.15 -> 10.0.0.87 on the second node. 6. Second node tries to answer to 10.0.2.15 unsuccessfully (all nodes have the same IP 10.0.2.15 for enp0s3; that IP isn't routable across nodes). This bug is described in #13774. Fixes: #13774 Co-authored-by: Gilberto Bertin <gilberto@isovalent.com> Signed-off-by: Paul Chaignon <paul@cilium.io>
pchaigno
added a commit
that referenced
this issue
Feb 9, 2021
--- Analysis --- In tunneling mode, our CILIUM_POST_nat chain is currently as follows. 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat ! -o cilium_host -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN 3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT 4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o cilium_host -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 The second rule implements an early exit from the chain, as none of the subsequent rules match on output interfaces other than cilium_host. Once per-endpoint routes are enabled in addition to tunneling, the chain changes. The second and fifth rules now match on lxc+ as the output interface: 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat ! -o lxc+ -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN 3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT 4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 Commit c496e25 ("eni: Support masquerading") implemented that change, based on the fact that with per-endpoint routes, packets are routed directly to lxc devices without going through cilium_host. Nevertheless, the fourth rule still matches on cilium_host and therefore becomes noop. At the time c496e25 was implemented, this change was correct because the fourth rule is only present when tunneling is enabled and per-endpoint routes were not compatible with tunneling. Commit 3179a47 ("datapath: Support enable-endpoint-routes with encapsulation") however made those options compatible and the above chain possible. --- Fix --- Ideally, we would update the second rule when running with tunneling and per-endpoint routes, to be '! -o lxc+ ! -o cilium_host'. Iptables however doesn't support multiple output interface matchers. This commit implements a different fix and drops the second rule. Since subsequent SNATing rules already match on an output interface, the second rule is unnecessary. With tunneling and per-endpoint routes, the table now looks like: 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquerade" -j ACCEPT 3. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 4. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 --- Bug Impact --- This lack of masquerading can cause issues for example when trying to connect to a VIP with a remote backend from the hostns in our test VMs: 1. DNS request is made to VIP 10.96.0.10. 2. 10.0.2.15, the IP of enp0s3 (default route) is assigned as source IP. 3. kube-proxy translates VIP to backend IP on different node, e.g. 10.0.0.87. 3. Packet is sent to cilium_host as per the ip routes. 4. The packet is not masqueraded because it matches rule 2 in the bogus iptables chain (i.e., cilium_host != lxc+). 5. The packet arrives as 10.0.2.15 -> 10.0.0.87 on the second node. 6. Second node tries to answer to 10.0.2.15 unsuccessfully (all nodes have the same IP 10.0.2.15 for enp0s3; that IP isn't routable across nodes). This bug is described in #13774. Fixes: #13774 Co-authored-by: Gilberto Bertin <gilberto@isovalent.com> Signed-off-by: Paul Chaignon <paul@cilium.io>
pchaigno
added a commit
that referenced
this issue
Feb 9, 2021
--- Analysis --- In tunneling mode, our CILIUM_POST_nat chain is currently as follows. 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat ! -o cilium_host -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN 3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT 4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o cilium_host -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 The second rule implements an early exit from the chain, as none of the subsequent rules match on output interfaces other than cilium_host. Once per-endpoint routes are enabled in addition to tunneling, the chain changes. The second and fifth rules now match on lxc+ as the output interface: 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat ! -o lxc+ -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN 3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT 4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 Commit c496e25 ("eni: Support masquerading") implemented that change, based on the fact that with per-endpoint routes, packets are routed directly to lxc devices without going through cilium_host. Nevertheless, the fourth rule still matches on cilium_host and therefore becomes noop. At the time c496e25 was implemented, this change was correct because the fourth rule is only present when tunneling is enabled and per-endpoint routes were not compatible with tunneling. Commit 3179a47 ("datapath: Support enable-endpoint-routes with encapsulation") however made those options compatible and the above chain possible. --- Fix --- Ideally, we would update the second rule when running with tunneling and per-endpoint routes, to be '! -o lxc+ ! -o cilium_host'. Iptables however doesn't support multiple output interface matchers. This commit implements a different fix and drops the second rule. Since subsequent SNATing rules already match on an output interface, the second rule is unnecessary. With tunneling and per-endpoint routes, the table now looks like: 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquerade" -j ACCEPT 3. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 4. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 --- Bug Impact --- This lack of masquerading can cause issues for example when trying to connect to a VIP with a remote backend from the hostns in our test VMs: 1. DNS request is made to VIP 10.96.0.10. 2. 10.0.2.15, the IP of enp0s3 (default route) is assigned as source IP. 3. kube-proxy translates VIP to backend IP on different node, e.g. 10.0.0.87. 3. Packet is sent to cilium_host as per the ip routes. 4. The packet is not masqueraded because it matches rule 2 in the bogus iptables chain (i.e., cilium_host != lxc+). 5. The packet arrives as 10.0.2.15 -> 10.0.0.87 on the second node. 6. Second node tries to answer to 10.0.2.15 unsuccessfully (all nodes have the same IP 10.0.2.15 for enp0s3; that IP isn't routable across nodes). This bug is described in #13774. Fixes: #13774 Co-authored-by: Gilberto Bertin <gilberto@isovalent.com> Signed-off-by: Paul Chaignon <paul@cilium.io>
nathanjsweet
pushed a commit
that referenced
this issue
Feb 10, 2021
--- Analysis --- In tunneling mode, our CILIUM_POST_nat chain is currently as follows. 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat ! -o cilium_host -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN 3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT 4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o cilium_host -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 The second rule implements an early exit from the chain, as none of the subsequent rules match on output interfaces other than cilium_host. Once per-endpoint routes are enabled in addition to tunneling, the chain changes. The second and fifth rules now match on lxc+ as the output interface: 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat ! -o lxc+ -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN 3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT 4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 Commit c496e25 ("eni: Support masquerading") implemented that change, based on the fact that with per-endpoint routes, packets are routed directly to lxc devices without going through cilium_host. Nevertheless, the fourth rule still matches on cilium_host and therefore becomes noop. At the time c496e25 was implemented, this change was correct because the fourth rule is only present when tunneling is enabled and per-endpoint routes were not compatible with tunneling. Commit 3179a47 ("datapath: Support enable-endpoint-routes with encapsulation") however made those options compatible and the above chain possible. --- Fix --- Ideally, we would update the second rule when running with tunneling and per-endpoint routes, to be '! -o lxc+ ! -o cilium_host'. Iptables however doesn't support multiple output interface matchers. This commit implements a different fix and drops the second rule. Since subsequent SNATing rules already match on an output interface, the second rule is unnecessary. With tunneling and per-endpoint routes, the table now looks like: 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquerade" -j ACCEPT 3. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 4. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 --- Bug Impact --- This lack of masquerading can cause issues for example when trying to connect to a VIP with a remote backend from the hostns in our test VMs: 1. DNS request is made to VIP 10.96.0.10. 2. 10.0.2.15, the IP of enp0s3 (default route) is assigned as source IP. 3. kube-proxy translates VIP to backend IP on different node, e.g. 10.0.0.87. 3. Packet is sent to cilium_host as per the ip routes. 4. The packet is not masqueraded because it matches rule 2 in the bogus iptables chain (i.e., cilium_host != lxc+). 5. The packet arrives as 10.0.2.15 -> 10.0.0.87 on the second node. 6. Second node tries to answer to 10.0.2.15 unsuccessfully (all nodes have the same IP 10.0.2.15 for enp0s3; that IP isn't routable across nodes). This bug is described in #13774. Fixes: #13774 Co-authored-by: Gilberto Bertin <gilberto@isovalent.com> Signed-off-by: Paul Chaignon <paul@cilium.io>
This was referenced Feb 12, 2021
lyveng
pushed a commit
to lyveng/cilium
that referenced
this issue
Mar 4, 2021
--- Analysis --- In tunneling mode, our CILIUM_POST_nat chain is currently as follows. 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat ! -o cilium_host -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN 3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT 4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o cilium_host -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 The second rule implements an early exit from the chain, as none of the subsequent rules match on output interfaces other than cilium_host. Once per-endpoint routes are enabled in addition to tunneling, the chain changes. The second and fifth rules now match on lxc+ as the output interface: 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat ! -o lxc+ -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN 3. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT 4. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 5. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 Commit c496e25 ("eni: Support masquerading") implemented that change, based on the fact that with per-endpoint routes, packets are routed directly to lxc devices without going through cilium_host. Nevertheless, the fourth rule still matches on cilium_host and therefore becomes noop. At the time c496e25 was implemented, this change was correct because the fourth rule is only present when tunneling is enabled and per-endpoint routes were not compatible with tunneling. Commit 3179a47 ("datapath: Support enable-endpoint-routes with encapsulation") however made those options compatible and the above chain possible. --- Fix --- Ideally, we would update the second rule when running with tunneling and per-endpoint routes, to be '! -o lxc+ ! -o cilium_host'. Iptables however doesn't support multiple output interface matchers. This commit implements a different fix and drops the second rule. Since subsequent SNATing rules already match on an output interface, the second rule is unnecessary. With tunneling and per-endpoint routes, the table now looks like: 1. -A CILIUM_POST_nat -s 10.0.1.0/24 ! -d 10.0.0.0/8 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE 2. -A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquerade" -j ACCEPT 3. -A CILIUM_POST_nat ! -s 10.0.1.6/32 ! -d 10.0.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.0.1.6 4. -A CILIUM_POST_nat -s 127.0.0.1/32 -o lxc+ -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.0.1.6 --- Bug Impact --- This lack of masquerading can cause issues for example when trying to connect to a VIP with a remote backend from the hostns in our test VMs: 1. DNS request is made to VIP 10.96.0.10. 2. 10.0.2.15, the IP of enp0s3 (default route) is assigned as source IP. 3. kube-proxy translates VIP to backend IP on different node, e.g. 10.0.0.87. 3. Packet is sent to cilium_host as per the ip routes. 4. The packet is not masqueraded because it matches rule 2 in the bogus iptables chain (i.e., cilium_host != lxc+). 5. The packet arrives as 10.0.2.15 -> 10.0.0.87 on the second node. 6. Second node tries to answer to 10.0.2.15 unsuccessfully (all nodes have the same IP 10.0.2.15 for enp0s3; that IP isn't routable across nodes). This bug is described in cilium#13774. Fixes: cilium#13774 Co-authored-by: Gilberto Bertin <gilberto@isovalent.com> Signed-off-by: Paul Chaignon <paul@cilium.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
every build on https://jenkins.cilium.io/job/cilium-master-K8s-all/ fails
The text was updated successfully, but these errors were encountered: