Skip to content

Commit

Permalink
ipsec: Don't match on source IP for XFRM OUT policies
Browse files Browse the repository at this point in the history
On IPAM modes with one pod CIDR per node, the XFRM OUT policies look like below:

    src 10.0.1.0/24 dst 10.0.0.0/24
        dir out priority 0 ptype main
        mark 0x66d11e00/0xffffff00
        tmpl src 10.0.1.13 dst 10.0.0.214
            proto esp spi 0x00000001 reqid 1 mode tunnel

When sending traffic from the hostns, however, it may not match the
source CIDR above. Traffic from the hostns may indeed leave the node
with the NodeInternalIP as the source IP (vs. CiliumInternalIP which
would match).

In such cases, we don't match the XFRM OUT policy and fall back to the
catch-all default-drop rule, ending up with XfrmOutPolBlock packet
drops.

Why wasn't this an issue before? It was. Traffic would simply go in
plain-text (which is okay given we never intended to encrypt hostns
traffic in the first place). What changes is that we now have a
catch-all default-drop XFRM OUT policy to avoid leaking plain-text
traffic. So it now results in XfrmOutPolBlock errors.

In commit 5fe2b2d ("bpf: Don't encrypt on path hostns -> remote pod")
we removed encryption for the path hostns -> remote pod. Unfortunately,
that doesn't mean the issue is completely gone. On a new Cilium install,
we won't see this issue of XfrmOutPolBlock drops for hostns traffic
anymore. But on existing clusters, we will still see those drops during
the upgrade, after the default-drop rule is installed but before hostns
traffic encryption is removed.

None of this is an issue on AKS and ENI IPAM modes because there, the
XFRM OUT policies look like:

src 0.0.0.0/0 dst 10.0.0.0/16
    dir out priority 0 ptype main
    mark 0x66d11e00/0xffffff00
    tmpl src 10.0.1.13 dst 10.0.0.214
        proto esp spi 0x00000001 reqid 1 mode tunnel

Thus, hostns -> remote pod traffic is matched regardless of the source
IP being selected and packets are not dropped by the default-drop rule.

We can therefore avoid the upgrade drops by changing the XFRM OUT
policies to never match on the source IPs, as on AKS and ENI IPAM modes.

Fixes: 7d44f37 ("ipsec: Catch-default default drop policy for encryption")
Signed-off-by: Paul Chaignon <paul.chaignon@gmail.com>
  • Loading branch information
pchaigno authored and julianwiedmann committed Jul 12, 2023
1 parent e5b2da5 commit ebd02f1
Showing 1 changed file with 4 additions and 6 deletions.
10 changes: 4 additions & 6 deletions pkg/datapath/linux/node.go
Original file line number Diff line number Diff line change
Expand Up @@ -1075,11 +1075,10 @@ func (n *linuxNodeHandler) enableIPsecIPv4(newNode *nodeTypes.Node, zeroMark boo
upsertIPsecLog(err, "out IPv4", wildcardCIDR, cidr, spi)
}
} else {
localCIDR := n.nodeAddressing.IPv4().AllocationCIDR().IPNet
remoteCIDR := newNode.IPv4AllocCIDR.IPNet
n.replaceNodeIPSecOutRoute(new4Net)
spi, err = ipsec.UpsertIPsecEndpoint(localCIDR, remoteCIDR, localIP, remoteIP, remoteNodeID, ipsec.IPSecDirOut, false)
upsertIPsecLog(err, "out IPv4", localCIDR, remoteCIDR, spi)
spi, err = ipsec.UpsertIPsecEndpoint(wildcardCIDR, remoteCIDR, localIP, remoteIP, remoteNodeID, ipsec.IPSecDirOut, false)
upsertIPsecLog(err, "out IPv4", wildcardCIDR, remoteCIDR, spi)
}
}
}
Expand Down Expand Up @@ -1149,11 +1148,10 @@ func (n *linuxNodeHandler) enableIPsecIPv6(newNode *nodeTypes.Node, zeroMark boo
upsertIPsecLog(err, "out IPv6", wildcardCIDR, cidr, spi)
}
} else {
localCIDR := n.nodeAddressing.IPv6().AllocationCIDR().IPNet
remoteCIDR := newNode.IPv6AllocCIDR.IPNet
n.replaceNodeIPSecOutRoute(new6Net)
spi, err := ipsec.UpsertIPsecEndpoint(localCIDR, remoteCIDR, localIP, remoteIP, remoteNodeID, ipsec.IPSecDirOut, false)
upsertIPsecLog(err, "out IPv6", localCIDR, remoteCIDR, spi)
spi, err := ipsec.UpsertIPsecEndpoint(wildcardCIDR, remoteCIDR, localIP, remoteIP, remoteNodeID, ipsec.IPSecDirOut, false)
upsertIPsecLog(err, "out IPv6", wildcardCIDR, remoteCIDR, spi)
}
}
}
Expand Down

0 comments on commit ebd02f1

Please sign in to comment.