Implement BPF-based masquerading for IPv6 #18694

pchaigno · 2022-02-01T18:17:53Z

Summary of commits:

Preparatory: use a bit of inline BPF asm to avoid bad optimization from LLVM.
Preparatory: refactor ct_is_reply4 as a macro to facilitate implem. of IPv6 counterpart.
Implement the datapath changes by following the IPv4 implementation.
Preparatory: refactor InitBPFMasqueradeAddrs to facilitate implem. of IPv6 counterpart.
Implement agent changes for the flags and macros.
Update documentation.
Extend the tests to cover IPv6 BPF masquerading.

See commits for details.

Fixes: #14350.

christarazi

🚀

qmonnet

Looks good, nice work!

Maybe one more thing to update:

$ grep supported Documentation/concepts/networking/masquerading.rst
    eBPF based masquerading is currently not supported for IPv6 traffic.

bpf/lib/nodeport.h

brb · 2022-07-29T07:43:29Z

The complexity has been improved - #20425 (comment). @qmonnet @pchaigno this PR is no longer blocked.

Since ctx->{data,data_end,data_meta} are 32-bits fields, LLVM sometimes generates 32-bit assignments for those pointers. This leads to verifier errors as the verifier is unable to track the packet pointers through the 32-bit assignments, as show below. ; return (void *)(unsigned long)ctx->data; 2: (61) r9 = *(u32 *)(r7 +76) ; R7_w=ctx(id=0,off=0,imm=0) R9_w=pkt(id=0,off=0,r=0,imm=0) ; return (void *)(unsigned long)ctx->data; 3: (bc) w6 = w9 ; R6_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R9_w=pkt(id=0,off=0,r=0,imm=0) ; if (data + tot_len > data_end) 4: (bf) r2 = r6 ; R2_w=inv(id=1,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6_w=inv(id=1,umax_value=4294967295,var_off=(0x0; 0xffffffff)) 5: (07) r2 += 54 ; R2_w=inv(id=0,umin_value=54,umax_value=4294967349,var_off=(0x0; 0x1ffffffff)) ; if (data + tot_len > data_end) 6: (2d) if r2 > r1 goto pc+466 ; R1_w=pkt_end(id=0,off=0,imm=0) R2_w=inv(id=0,umin_value=54,umax_value=4294967349,var_off=(0x0; 0x1ffffffff)) ; tmp = a->d1 - b->d1; 7: (71) r2 = *(u8 *)(r6 +22) R6 invalid mem access 'inv' As a workaround, Yonghong Song suggested to read those fields using asm bytecode. With that trick, LLVM is unaware that the original field is actually on 32 bits and can't perform the above "optimization". This workaround is needed only now because a subsequent commit introduces some BPF C code that causes this bytecode to be generated (snat_v6_needed). Related: https://lore.kernel.org/bpf/c5a76b4d-abed-51f6-bf16-040eb0baf290@fb.com/T/#m933626f7404562a6e98af78a8f44c30977681ffa Suggested-by: Yonghong Song <yhs@fb.com> Signed-off-by: Paul Chaignon <paul@cilium.io>

This will allow us to easily define the IPv6 equivalent in a subsequent commit. This commit should include no functional changes. Signed-off-by: Paul Chaignon <paul@cilium.io>

This commit implements BPF masquerading for IPv6 by following the implementation for IPv4. The main function changed here is snat_v6_needed with almost the exact same steps as IPv4. Support for the ipmasq agent in IPv6 is not implemented here. [ Quentin: Rebased on master branch, addressed minor conflicts in nodeport.h due to commit 0ea4341 ("bpf: Make union v6addr const where possible"). ] Signed-off-by: Paul Chaignon <paul@cilium.io> Signed-off-by: Quentin Monnet <quentin@isovalent.com>

This commit refactors InitBPFMasqueradeAddrs to easily extend it for IPv6 in a subsequent commit. This commit shouldn't have any functional changes. Signed-off-by: Paul Chaignon <paul@cilium.io>

This commit allows enabling BPF masquerading for IPv6 and defines all the proper macros for the datapath implementation. [ Quentin: Rebased on master branch, addressed minor conflicts on daemon.go and loader.go. ] Signed-off-by: Paul Chaignon <paul@cilium.io> Signed-off-by: Quentin Monnet <quentin@isovalent.com>

BPF-based IPv6 masquerading is now supported, except for the ip-masq agent. This commit therefore updates the documentation accordingly. Signed-off-by: Paul Chaignon <paul@cilium.io>

Since Cilium supports BPF masquerading for IPv6, we can now test it. The upgrade/downgrade test needs a small exception because IPv6 BPF masquerading is still not supported on v1.11, from which we upgrade. The exception logic comes with an assertion, to ensure we don't forget to remove the exception once the upgrade is performed from v1.12. Signed-off-by: Paul Chaignon <paul@cilium.io>

qmonnet · 2022-07-29T11:01:30Z

@pchaigno I rebased, but I can't push on your repo. I pushed to cilium/pr/pchaigno/ipv6-bpf-masquerading instead, any chance you can swap to that branch for the PR?

I addressed minor conflicts on daemon.go, loader.go, nodeport.h. Nothing that looked significant.

Diff on nodeport.h

diff --git a/bpf/lib/nodeport.h b/bpf/lib/nodeport.h
index fa29b2908c90..22b7a7f8575d 100644
--- a/bpf/lib/nodeport.h
+++ b/bpf/lib/nodeport.h
@@ -179,10 +179,10 @@ static __always_inline bool nodeport_uses_dsr6(const struct ipv6_ct_tuple *tuple
  * then the helper function won't depend the dsr checks.
  */
 static __always_inline bool snat_v6_needed(struct __ctx_buff *ctx,
-					   const union v6addr *addr)
+					   union v6addr *addr)
 {
 	union v6addr masq_addr __maybe_unused;
-	union v6addr dr_addr = IPV6_DIRECT_ROUTING;
+	const union v6addr dr_addr = IPV6_DIRECT_ROUTING;
 	struct remote_endpoint_info *remote_ep __maybe_unused;
 	struct endpoint_info *local_ep __maybe_unused;
 	void *data, *data_end;

pchaigno · 2022-07-31T19:55:16Z

Still hitting a complexity issue on 4.9, in bpf_host, when the host firewall is enabled.

brb · 2022-08-02T08:29:51Z

Still hitting a complexity issue on 4.9, in bpf_host, when the host firewall is enabled.

We can either allow IPv6 BPF masq only on kernels with HAVE_LARGE_INSN_LIMIT, or wait until the second part of the SNAT refactoring has been done in #19712.

pchaigno · 2022-08-02T09:02:33Z

The 4.9 complexity issue seems to be affecting function handle_lxc_traffic in bpf_host.c. So that one should be really easy to fix by introducing tail calls, to separate the from_host_to_lxc and to_host_from_lxc code paths. It's ~20min of work if you rely on CI to run the tests.

There seem to be another complexity issue on 4.19 with the host endpoint as well. Hopefully it's the same, but it's impossible to know since logs are missing (I suspect because of #20738).

batistein · 2022-09-15T07:32:24Z

@pchaigno Will this also work in tunneling mode without an IPv4 address?

qmonnet · 2023-06-16T22:11:20Z

Completed in #23165.

networkop · 2023-07-21T16:02:02Z

would it make sense to disable the log.Fatal in this ip6_table module check?

qmonnet · 2023-07-24T10:43:37Z

Discussed offline with Joe:

Pretty sure even with that we still depend on ip6tables for some functionality, so by fataling we can make it crystal clear to the user that they have configured Cilium in a way that will cause IPv6 connectivity issues.

So we keep the current behaviour for now.

pchaigno added sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. release-note/major This PR introduces major new functionality to Cilium. feature/ipv6 Relates to IPv6 protocol support labels Feb 1, 2022

pchaigno force-pushed the ipv6-bpf-masquerading branch 6 times, most recently from 84df485 to 796fdcb Compare February 6, 2022 17:10

pchaigno force-pushed the ipv6-bpf-masquerading branch 3 times, most recently from 92cff28 to 9e8f18d Compare February 10, 2022 16:22

pchaigno requested a review from qmonnet February 10, 2022 16:28

maintainer-s-little-helper bot assigned qmonnet Feb 10, 2022

pchaigno force-pushed the ipv6-bpf-masquerading branch from 9e8f18d to 30820cf Compare February 10, 2022 21:45

pchaigno marked this pull request as ready for review February 10, 2022 22:10

pchaigno requested a review from a team February 10, 2022 22:10

pchaigno requested a review from a team as a code owner February 10, 2022 22:10

pchaigno requested a review from a team February 10, 2022 22:10

pchaigno requested a review from a team as a code owner February 10, 2022 22:10

pchaigno requested review from ldelossa and christarazi February 10, 2022 22:10

maintainer-s-little-helper bot assigned ldelossa and christarazi Feb 10, 2022

christarazi approved these changes Feb 11, 2022

View reviewed changes

maintainer-s-little-helper bot unassigned christarazi Feb 11, 2022

qmonnet approved these changes Feb 14, 2022

View reviewed changes

maintainer-s-little-helper bot unassigned qmonnet Feb 14, 2022

pchaigno force-pushed the ipv6-bpf-masquerading branch from 30820cf to df0cdb8 Compare February 14, 2022 22:36

pchaigno requested a review from a team as a code owner February 14, 2022 22:36

pchaigno force-pushed the ipv6-bpf-masquerading branch 2 times, most recently from 3b2eef3 to 3cc9787 Compare May 23, 2022 17:09

julianwiedmann reviewed May 24, 2022

View reviewed changes

bpf/lib/nodeport.h Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jul 7, 2022

github-actions bot removed the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jul 16, 2022

pchaigno added 7 commits July 29, 2022 11:36

bpf: Refactor ct_is_reply as a macro

9ea62a1

This will allow us to easily define the IPv6 equivalent in a subsequent commit. This commit should include no functional changes. Signed-off-by: Paul Chaignon <paul@cilium.io>

node: Refactor InitBPFMasqueradeAddrs

d2ba82d

This commit refactors InitBPFMasqueradeAddrs to easily extend it for IPv6 in a subsequent commit. This commit shouldn't have any functional changes. Signed-off-by: Paul Chaignon <paul@cilium.io>

docs: Update documentation for BPF IPv6 masquerading

40e3030

BPF-based IPv6 masquerading is now supported, except for the ip-masq agent. This commit therefore updates the documentation accordingly. Signed-off-by: Paul Chaignon <paul@cilium.io>

pchaigno force-pushed the ipv6-bpf-masquerading branch from 3cc9787 to ae16b5d Compare July 29, 2022 18:55

ti-mo mentioned this pull request Aug 3, 2022

bpf: grow verifier log buffer to 10MiB to prevent truncation #20773

Merged

This comment was marked as resolved.

Sign in to view

github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Sep 2, 2022

qmonnet added pinned These issues are not marked stale by our issue bot. and removed stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. labels Sep 2, 2022

qmonnet mentioned this pull request Jan 18, 2023

Implement BPF-based masquerading for IPv6 (rebased) #23165

Merged

qmonnet closed this Jun 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement BPF-based masquerading for IPv6 #18694

Implement BPF-based masquerading for IPv6 #18694

pchaigno commented Feb 1, 2022 •

edited

christarazi left a comment

qmonnet left a comment

This comment was marked as resolved.

brb commented Jul 29, 2022

qmonnet commented Jul 29, 2022

pchaigno commented Jul 31, 2022

brb commented Aug 2, 2022 •

edited

pchaigno commented Aug 2, 2022

This comment was marked as resolved.

batistein commented Sep 15, 2022

qmonnet commented Jun 16, 2023

networkop commented Jul 21, 2023

qmonnet commented Jul 24, 2023

Implement BPF-based masquerading for IPv6 #18694

Implement BPF-based masquerading for IPv6 #18694

Conversation

pchaigno commented Feb 1, 2022 • edited

christarazi left a comment

Choose a reason for hiding this comment

qmonnet left a comment

Choose a reason for hiding this comment

This comment was marked as resolved.

brb commented Jul 29, 2022

qmonnet commented Jul 29, 2022

pchaigno commented Jul 31, 2022

brb commented Aug 2, 2022 • edited

pchaigno commented Aug 2, 2022

This comment was marked as resolved.

batistein commented Sep 15, 2022

qmonnet commented Jun 16, 2023

networkop commented Jul 21, 2023

qmonnet commented Jul 24, 2023

pchaigno commented Feb 1, 2022 •

edited

brb commented Aug 2, 2022 •

edited