BPF: Encrypted Overlay Support #31073

ldelossa · 2024-02-29T22:15:07Z

This pull request implements the data-path side of "Encrypted Overlay" support.

"Encrypted Overlay" allows Cilium to IPSec encrypt overlay traffic and deliver the encrypted overlay traffic to a remote node via IPSec tunnel.

A subsequent PR will follow which enables this in the IPSec control plane, creating the necessary policies and states to configure the tunnel source and destination addresses correctly.

This feature is useful when an application's traffic destined for a remote host cannot be routed into the XFRM netfilter hooks for various reasons.
In these cases we can first encapsulate the traffic in VXLAN, catch it at the egress interface, and recirculate the packet back into the system, at which point the traffic looks like unicast traffic that should be routed off host (and XFRM will be applied at this point).

It's important to note that the following sysctl configs must be applied to any interfaces which tunnels traffic toward a remote node for this to work

- net.ipv4.conf.default.rp_filter = 0
- net.ipv4.conf.default.accept_local = 1

bpf: introduce encrypted overlay datapath support

bpf/bpf_host.c

ldelossa · 2024-03-01T00:51:26Z

/test

brb

@cilium/wireguard changes LGTM.

pchaigno

Neat pull request! I really like the code comments 🙂

Couple comments below. IIUC, I think there's an issue with the SPI and security identity.

bpf/bpf_host.c

bpf/lib/encrypt.h

bpf/lib/trace.h

kaworu

Thanks for the PR @ldelossa, on the Hubble side:

Please add TraceReasonEncryptOverlay introduced by this PR in TraceReasonIsEncap. That way the corresponding Hubble flow won't have the IsReply field set. The test case should be similar to SRV6 encap, except that the Source should be hostEP instead (since this trace is emitted from bpf_host.c).
I'm not sure whether decodeTrafficDirection will correctly infer EGRESS for the corresponding flow. Can you run hubble observe -o json and see what the TraceReasonCtReply flow looks like? If shown as INGRESS, please add a comment in decodeTrafficDirection stating that it is currently broken for this particular trace reason 🙏

bpf/lib/trace.h

ldelossa · 2024-03-01T20:46:41Z

discussion with @joestringer lead to the following suggested changes:

Create and use a reserved security ID to signal that vxlan traffic should be encapsulated and not the encryption bits.
Suggested re-writing the VNI based on the source destination of the encapsulated packet before it leaves the host.

These suggestions are to mitigate any shenanigans that could occur from leaking nonsensical VXLAN ids.

bpf/bpf_host.c

cilium#30154 and cilium#31073 introduced new datapath trace reasons and had an impact on Hubble, but the sig-hubble team doesn't get automatically pulled in for review. This patch adds the sig-hubble team to review datapath_trace.go changes. Signed-off-by: Alexandre Perrin <alex@isovalent.com>

julianwiedmann

looks good, thank you!

bpf/Makefile

pchaigno · 2024-03-26T14:00:38Z

@ldelossa We just merged the IPsec fixes so there's a conflict. Ping me on Slack if you need help on that.

This commit adds a new reserved security identity for signaling overlay traffic which must be IPSec encrypted. When the eBPF datapath encounters an egress packet with this security identity in an overlay header (currently only VXLan supported) it will subject the packet to IPSec encryption and rewrite the overlay header with the correct security identity before the packet leaves the host. Therefore, this identity should NEVER be seen on traffic ingress or egress the node from the network. Signed-off-by: Louis DeLosSantos <louis.delos@isovalent.com>

This commits and both the agent and datapath flag required to enable the "Encrypted Overlay" feature. The datapath will use ENABLE_ENCRYPTED_OVERLAY feature flag. The agent will use "encryption.ipsec.encryptOverlay" Signed-off-by: Louis DeLosSantos <louis.delos@isovalent.com>

ldelossa · 2024-03-26T14:04:33Z

@pchaigno handling this now, the diff looks pretty simple on my end. Just a change in includes in encrypt.h

Just pushed conflict resolution.

This commit introduces the ability to encrypt overlay traffic before it leaves the host. The 'cil_to_netdev' function is updated to sniff into overlay packets (only VXLAN supported for now) and determine if the ENCRYPTED_OVERLAY_ID security identifier is present in the overlay's header. If it is, a new function in encrypt.h will set the appropriate packet mark on the skb and redirect the packet to the ingress of the interface it was egressing on. When the packet is seen on the ingress side of the device it will be submitted to the XFRM hooks in the output routing path and the XFRM subsystem will encrypt the packet. Subsequent changes to the IPSec control plane to create the appropriate states and policies to support this are required. Signed-off-by: ldelossa <louis.delos@isovalent.com>

Add a trace notification when we are redirecting a packet back into the stack for XFRM encryption. Trace example: -> stack flow 0xc218244b , identity unknown->unknown state encrypt-overlay ifindex 0 orig-ip 0.0.0.0: 172.18.0.3:58167 -> 172.18.0.2:8472 udp Signed-off-by: ldelossa <louis.delos@isovalent.com>

Add unit tests for new vxlan helper functions in tunnel.h Signed-off-by: Louis DeLosSantos <louis.delos@isovalent.com>

ldelossa · 2024-03-26T14:30:41Z

/test

ldelossa · 2024-03-26T16:47:05Z

https://github.com/cilium/cilium/actions/runs/8437738465 is failing, but this is an unrelated error.

Marking this PR as ready to merge.

cilium#30154 and cilium#31073 introduced new datapath trace reasons and had an impact on Hubble, but the sig-hubble team doesn't get automatically pulled in for review. This patch adds the sig-hubble team to review datapath_trace.go changes. Signed-off-by: Alexandre Perrin <alex@isovalent.com>

#30154 and #31073 introduced new datapath trace reasons and had an impact on Hubble, but the sig-hubble team doesn't get automatically pulled in for review. This patch adds the sig-hubble team to review datapath_trace.go changes. Signed-off-by: Alexandre Perrin <alex@isovalent.com>

ldelossa added sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. release-note/minor This PR changes functionality that users may find relevant to operating Cilium. labels Feb 29, 2024

ldelossa requested review from a team as code owners February 29, 2024 22:15

ldelossa requested review from julianwiedmann, brb, a team and lambdanis and removed request for a team February 29, 2024 22:15

ldelossa commented Feb 29, 2024

View reviewed changes

bpf/bpf_host.c Outdated Show resolved Hide resolved

ldelossa requested a review from pchaigno February 29, 2024 22:42

ldelossa force-pushed the ldelossa/encrypted-overlay-oss branch 2 times, most recently from b5f0705 to ceb1eec Compare March 1, 2024 00:32

ldelossa force-pushed the ldelossa/encrypted-overlay-oss branch 2 times, most recently from f2f6671 to 9433bad Compare March 1, 2024 02:27

brb approved these changes Mar 1, 2024

View reviewed changes

pchaigno added release-note/major This PR introduces major new functionality to Cilium. feature/ipsec Relates to Cilium's IPsec feature and removed release-note/minor This PR changes functionality that users may find relevant to operating Cilium. labels Mar 1, 2024

pchaigno reviewed Mar 1, 2024

View reviewed changes

bpf/bpf_host.c Outdated Show resolved Hide resolved

bpf/lib/encrypt.h Outdated Show resolved Hide resolved

bpf/lib/encrypt.h Outdated Show resolved Hide resolved

bpf/lib/trace.h Show resolved Hide resolved

kaworu requested changes Mar 1, 2024

View reviewed changes

bpf/lib/trace.h Show resolved Hide resolved

rolinh assigned kaworu Mar 1, 2024

joestringer reviewed Mar 1, 2024

View reviewed changes

bpf/bpf_host.c Show resolved Hide resolved

ldelossa force-pushed the ldelossa/encrypted-overlay-oss branch from 9433bad to a01c9d1 Compare March 2, 2024 00:26

kaworu mentioned this pull request Mar 6, 2024

Hubble: expose the trace reason field #31202

Closed

1 task

ldelossa force-pushed the ldelossa/encrypted-overlay-oss branch from a01c9d1 to 27ef8dc Compare March 6, 2024 22:57

julianwiedmann approved these changes Mar 26, 2024

View reviewed changes

This comment was marked as outdated.

Sign in to view

pchaigno approved these changes Mar 26, 2024

View reviewed changes

bpf/Makefile Show resolved Hide resolved

This comment was marked as outdated.

Sign in to view

ldelossa added 2 commits March 26, 2024 10:02

ldelossa and others added 3 commits March 26, 2024 10:05

bpf,test: add tests for vxlan helper functions

bcd5eec

Add unit tests for new vxlan helper functions in tunnel.h Signed-off-by: Louis DeLosSantos <louis.delos@isovalent.com>

ldelossa force-pushed the ldelossa/encrypted-overlay-oss branch from 40d4347 to bcd5eec Compare March 26, 2024 14:05

ldelossa added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Mar 26, 2024

pchaigno added this pull request to the merge queue Mar 27, 2024

Merged via the queue into cilium:main with commit d00547a Mar 27, 2024
62 checks passed

pchaigno mentioned this pull request Mar 27, 2024

workflows: Cover IPsec encrypted overlay mode in end-to-end tests #31637

Merged

ldelossa deleted the ldelossa/encrypted-overlay-oss branch March 27, 2024 15:15

julianwiedmann mentioned this pull request Apr 4, 2024

bpf: avoid SNAT tracking for overlay traffic #31082

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BPF: Encrypted Overlay Support #31073

BPF: Encrypted Overlay Support #31073

ldelossa commented Feb 29, 2024 •

edited

ldelossa commented Mar 1, 2024

brb left a comment

pchaigno left a comment

kaworu left a comment

ldelossa commented Mar 1, 2024

julianwiedmann left a comment

This comment was marked as outdated.

This comment was marked as outdated.

pchaigno commented Mar 26, 2024

ldelossa commented Mar 26, 2024 •

edited

ldelossa commented Mar 26, 2024

ldelossa commented Mar 26, 2024

BPF: Encrypted Overlay Support #31073

BPF: Encrypted Overlay Support #31073

Conversation

ldelossa commented Feb 29, 2024 • edited

ldelossa commented Mar 1, 2024

brb left a comment

Choose a reason for hiding this comment

pchaigno left a comment

Choose a reason for hiding this comment

kaworu left a comment

Choose a reason for hiding this comment

ldelossa commented Mar 1, 2024

julianwiedmann left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

pchaigno commented Mar 26, 2024

ldelossa commented Mar 26, 2024 • edited

ldelossa commented Mar 26, 2024

ldelossa commented Mar 26, 2024

ldelossa commented Feb 29, 2024 •

edited

ldelossa commented Mar 26, 2024 •

edited