Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: ebpf unit tests fail with Ubuntu 22.04.1 kernel 5.15 #24053

Closed
wanlin31 opened this issue Feb 27, 2023 · 7 comments · Fixed by #24094
Closed

CI: ebpf unit tests fail with Ubuntu 22.04.1 kernel 5.15 #24053

wanlin31 opened this issue Feb 27, 2023 · 7 comments · Fixed by #24094
Labels
area/build Anything to do with the build, more general then area/CI area/CI Continuous Integration testing issue or flake sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.

Comments

@wanlin31
Copy link
Contributor

CI failure: ebpf unit tests fail with Ubuntu 22.04.1 kernel 5.15

OS + Kernel

uname -a
Linux wanlindu-cilium-ebpf-testvm-image-ubuntu2204 5.15.0-57-generic #63-Ubuntu SMP Thu Nov 24 13:43:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Checked out commit

git log
commit 66e5ec75ed8bbb96c4129634a75f01662cc439b3 (HEAD -> master, origin/master, origin/HEAD)

Commands to run the test

# Install the newest version of golang.
sudo add-apt-repository ppa:longsleep/golang-backports -y && sudo apt update && sudo apt install golang-go -y

# Install clang-11 and llvm-11 and make
sudo apt install clang-11 llvm-11 -y make

# Update the clang command used for test
export CLANG=clang-11 
export LLC=llc-11

# Clone the upstream cilium.
mkdir upstream_cilium && cd upstream_cilium && git clone --recursive https://github.com/cilium/cilium.git

# Run ebpf unit test in /upstream_cilium and /internal_cilium
cd ~/upstream_cilium/cilium && make -C test run_bpf_tests

Error message

make: Entering directory '/home/wanlindu/upstream-cilium/cilium/test'
make -C ../bpf/tests all
make[1]: Entering directory '/home/wanlindu/upstream-cilium/cilium/bpf/tests'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/wanlindu/upstream-cilium/cilium/bpf/tests'
go test ./bpf_tests -exec sudo -bpf-test-path /home/wanlindu/upstream-cilium/cilium/bpf/tests 
--- FAIL: TestBPF (0.89s)
    --- FAIL: TestBPF/session_affinity_test.o (0.10s)
        bpf_test.go:150: verifier error: load program: permission denied:
            	; int tail_nodeport_nat_egress_ipv4(struct __ctx_buff *ctx)
            	0: (bf) r6 = r1
            	; return ctx->ingress_ifindex;
            	1: (61) r1 = *(u32 *)(r6 +12)
            	2: (7b) *(u64 *)(r10 -168) = r1
            	3: (b7) r2 = 0
            	; struct ipv4_ct_tuple tuple = {};
            	4: (6b) *(u16 *)(r10 -140) = r2
            	last_idx 4 first_idx 0
            	regs=4 stack=0 before 3: (b7) r2 = 0
            	5: (63) *(u32 *)(r10 -144) = r2
            	6: (7b) *(u64 *)(r10 -152) = r2
            	7: (18) r8 = 0xffffff7a
            	; return (void *)(unsigned long)ctx->data_end;
            	9: (61) r1 = *(u32 *)(r6 +4)
            	; return (void *)(unsigned long)ctx->data;
            	10: (61) r9 = *(u32 *)(r6 +0)
            	; if (data + tot_len > data_end)
            	11: (bf) r3 = r9
            	12: (07) r3 += 34
            	; if (data + tot_len > data_end)
            	13: (2d) if r3 > r1 goto pc+2968
            	 R1_w=pkt_end(id=0,off=0,imm=0) R2_w=invP0 R3_w=pkt(id=0,off=34,r=34,imm=0) R6_w=ctx(id=0,off=0,imm=0) R8_w=inv4294967162 R9_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??000000 fp-152_w=00000000 fp-168_w=inv
            	; tuple->nexthdr = ip4->protocol;
            	14: (71) r1 = *(u8 *)(r9 +23)
            	; tuple->nexthdr = ip4->protocol;
            	15: (73) *(u8 *)(r10 -140) = r1
            	; tuple->daddr = ip4->daddr;
            	16: (61) r3 = *(u32 *)(r9 +30)
            	; tuple->daddr = ip4->daddr;
            	17: (63) *(u32 *)(r10 -152) = r3
            	; tuple->saddr = ip4->saddr;
            	18: (61) r3 = *(u32 *)(r9 +26)
            	; tuple->flags = dir;
            	19: (73) *(u8 *)(r10 -139) = r2
            	; tuple->saddr = ip4->saddr;
            	20: (63) *(u32 *)(r10 -148) = r3
            	; return ip4->ihl * 4;
            	21: (71) r2 = *(u8 *)(r9 +14)
            	; return ip4->ihl * 4;
            	22: (67) r2 <<= 2
            	23: (57) r2 &= 60
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	24: (07) r2 += 14
            	25: (7b) *(u64 *)(r10 -160) = r2
            	; switch (tuple.nexthdr) {
            	26: (15) if r1 == 0x1 goto pc+28
            	 R1_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6_w=ctx(id=0,off=0,imm=0) R8_w=inv4294967162 R9_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0m0000 fp-152_w=mmmmmmmm fp-160_w=inv fp-168_w=inv
            	27: (15) if r1 == 0x11 goto pc+3
            	 R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R8=inv4294967162 R9=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=inv fp-168=inv
            	;
            	28: (18) r8 = 0xffffff53
            	; switch (tuple.nexthdr) {
            	30: (55) if r1 != 0x6 goto pc+2951
            	 R1=inv6 R2=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R8_w=inv4294967123 R9=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=inv fp-168=inv
            	; asm volatile("r1 = *(u32 *)(%[ctx] +0)\n\t"
            	31: (79) r3 = *(u64 *)(r10 -160)
            	32: (61) r1 = *(u32 *)(r6 +0)
            	33: (61) r2 = *(u32 *)(r6 +4)
            	34: (57) r3 &= 255
            	35: (0f) r1 += r3
            	last_idx 35 first_idx 27
            	regs=8 stack=0 before 34: (57) r3 &= 255
            	regs=8 stack=0 before 33: (61) r2 = *(u32 *)(r6 +4)
            	regs=8 stack=0 before 32: (61) r1 = *(u32 *)(r6 +0)
            	regs=8 stack=0 before 31: (79) r3 = *(u64 *)(r10 -160)
            	regs=0 stack=80000 before 30: (55) if r1 != 0x6 goto pc+2951
            	regs=0 stack=80000 before 28: (18) r8 = 0xffffff53
            	regs=0 stack=80000 before 27: (15) if r1 == 0x11 goto pc+3
            	 R1_rw=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6_rw=ctx(id=0,off=0,imm=0) R8_w=inv4294967162 R9_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0m0000 fp-152_w=mmmmmmmm fp-160_rw=invP fp-168_w=inv
            	parent didn't have regs=0 stack=80000 marks
            	last_idx 26 first_idx 0
            	regs=0 stack=80000 before 26: (15) if r1 == 0x1 goto pc+28
            	regs=0 stack=80000 before 25: (7b) *(u64 *)(r10 -160) = r2
            	regs=4 stack=0 before 24: (07) r2 += 14
            	regs=4 stack=0 before 23: (57) r2 &= 60
            	regs=4 stack=0 before 22: (67) r2 <<= 2
            	regs=4 stack=0 before 21: (71) r2 = *(u8 *)(r9 +14)
            	36: (bf) r4 = r1
            	37: (07) r1 += 4
            	38: (2d) if r1 > r2 goto pc+2
            	 R1=pkt(id=1,off=4,r=4,umin_value=2,umax_value=74,var_off=(0x2; 0x7c),s32_max_value=126,u32_max_value=126) R2=pkt_end(id=0,off=0,imm=0) R3=invP(id=0,umin_value=2,umax_value=74,var_off=(0x2; 0x7c)) R4=pkt(id=1,off=0,r=4,umin_value=2,umax_value=74,var_off=(0x2; 0x7c),s32_max_value=126,u32_max_value=126) R6=ctx(id=0,off=0,imm=0) R8=inv4294967123 R9=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=inv fp-168=inv
            	39: (b7) r3 = 0
            	40: (05) goto pc+1
            	42: (67) r3 <<= 32
            	43: (bf) r1 = r3
            	44: (77) r1 >>= 32
            	; if (!ret)
            	45: (55) if r1 != 0x0 goto pc+4
            	last_idx 45 first_idx 38
            	regs=2 stack=0 before 44: (77) r1 >>= 32
            	regs=2 stack=0 before 43: (bf) r1 = r3
            	regs=8 stack=0 before 42: (67) r3 <<= 32
            	regs=8 stack=0 before 40: (05) goto pc+1
            	regs=8 stack=0 before 39: (b7) r3 = 0
            	; case  4: __it_mob(d, s, 32);
            	46: (61) r1 = *(u32 *)(r4 +0)
            	47: (bf) r2 = r1
            	48: (77) r2 >>= 16
            	49: (05) goto pc+23
            	;
            	73: (6b) *(u16 *)(r10 -142) = r1
            	74: (6b) *(u16 *)(r10 -144) = r2
            	75: (bf) r2 = r10
            	76: (07) r2 += -152
            	; return map_lookup_elem(map, tuple);
            	77: (18) r1 = 0xffff9ca4c89c5c00
            	79: (85) call bpf_map_lookup_elem#1
            	;
            	80: (bf) r2 = r0
            	81: (07) r2 += 32
            	R2 pointer arithmetic on map_value_or_null prohibited, null-check it first
            	processed 55 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 3
    --- FAIL: TestBPF/tc_egressgw.o (0.06s)
        bpf_test.go:150: verifier error: load program: permission denied:
            	; int tail_nodeport_nat_egress_ipv4(struct __ctx_buff *ctx)
            	0: (bf) r6 = r1
            	1: (b7) r1 = 0
            	; struct bpf_fib_lookup_padded fib_params = {
            	2: (7b) *(u64 *)(r10 -232) = r1
            	last_idx 2 first_idx 0
            	regs=2 stack=0 before 1: (b7) r1 = 0
            	3: (b7) r2 = 2
            	; .l = {
            	4: (73) *(u8 *)(r10 -232) = r2
            	; struct bpf_fib_lookup_padded fib_params = {
            	5: (63) *(u32 *)(r10 -168) = r1
            	6: (7b) *(u64 *)(r10 -176) = r1
            	7: (7b) *(u64 *)(r10 -184) = r1
            	8: (7b) *(u64 *)(r10 -192) = r1
            	9: (7b) *(u64 *)(r10 -200) = r1
            	10: (7b) *(u64 *)(r10 -208) = r1
            	11: (7b) *(u64 *)(r10 -216) = r1
            	12: (7b) *(u64 *)(r10 -224) = r1
            	; return ctx->ifindex;
            	13: (61) r2 = *(u32 *)(r6 +40)
            	; .l = {
            	14: (63) *(u32 *)(r10 -224) = r2
            	; struct ipv4_ct_tuple tuple = {};
            	15: (6b) *(u16 *)(r10 -148) = r1
            	16: (63) *(u32 *)(r10 -152) = r1
            	17: (7b) *(u64 *)(r10 -160) = r1
            	18: (18) r7 = 0xffffff7a
            	; return (void *)(unsigned long)ctx->data_end;
            	20: (61) r2 = *(u32 *)(r6 +80)
            	; return (void *)(unsigned long)ctx->data;
            	21: (61) r9 = *(u32 *)(r6 +76)
            	; if (data + tot_len > data_end)
            	22: (bf) r3 = r9
            	23: (07) r3 += 34
            	; if (data + tot_len > data_end)
            	24: (2d) if r3 > r2 goto pc+2648
            	 R1_w=invP0 R2_w=pkt_end(id=0,off=0,imm=0) R3_w=pkt(id=0,off=34,r=34,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv4294967162 R9_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??000000 fp-160_w=00000000 fp-168=????0000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=0000mmmm fp-232_w=0000000m
            	; tuple->nexthdr = ip4->protocol;
            	25: (71) r4 = *(u8 *)(r9 +23)
            	; tuple->nexthdr = ip4->protocol;
            	26: (73) *(u8 *)(r10 -148) = r4
            	; tuple->daddr = ip4->daddr;
            	27: (61) r3 = *(u32 *)(r9 +30)
            	; tuple->daddr = ip4->daddr;
            	28: (63) *(u32 *)(r10 -160) = r3
            	; tuple->saddr = ip4->saddr;
            	29: (61) r3 = *(u32 *)(r9 +26)
            	; tuple->flags = dir;
            	30: (73) *(u8 *)(r10 -147) = r1
            	; tuple->saddr = ip4->saddr;
            	31: (63) *(u32 *)(r10 -156) = r3
            	; return ip4->ihl * 4;
            	32: (71) r2 = *(u8 *)(r9 +14)
            	; return ip4->ihl * 4;
            	33: (67) r2 <<= 2
            	34: (57) r2 &= 60
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	35: (07) r2 += 14
            	36: (7b) *(u64 *)(r10 -240) = r2
            	; switch (tuple.nexthdr) {
            	37: (15) if r4 == 0x1 goto pc+31
            	 R1_w=invP0 R2_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv4294967162 R9_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160_w=mmmmmmmm fp-168=????0000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=00000000 fp-224_w=0000mmmm fp-232_w=0000000m fp-240_w=inv
            	38: (15) if r4 == 0x11 goto pc+3
            	 R1=invP0 R2=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6=ctx(id=0,off=0,imm=0) R7=inv4294967162 R9=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-168=????0000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=00000000 fp-208=00000000 fp-216=00000000 fp-224=0000mmmm fp-232=0000000m fp-240=inv
            	;
            	39: (18) r7 = 0xffffff53
            	; switch (tuple.nexthdr) {
            	41: (55) if r4 != 0x6 goto pc+2631
            	 R1=invP0 R2=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4=inv6 R6=ctx(id=0,off=0,imm=0) R7_w=inv4294967123 R9=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-168=????0000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=00000000 fp-208=00000000 fp-216=00000000 fp-224=0000mmmm fp-232=0000000m fp-240=inv
            	42: (bf) r3 = r10
            	; if (ctx_load_bytes(ctx, off, &l4hdr, sizeof(l4hdr)) < 0)
            	43: (07) r3 += -104
            	44: (bf) r1 = r6
            	45: (79) r2 = *(u64 *)(r10 -240)
            	46: (b7) r4 = 4
            	47: (85) call bpf_skb_load_bytes#26
            	last_idx 47 first_idx 38
            	regs=10 stack=0 before 46: (b7) r4 = 4
            	48: (67) r0 <<= 32
            	49: (c7) r0 s>>= 32
            	;
            	50: (18) r7 = 0xffffff7a
            	; if (ctx_load_bytes(ctx, off, &l4hdr, sizeof(l4hdr)) < 0)
            	52: (c5) if r0 s< 0x0 goto pc+2620
            	 R0_w=inv(id=0,umax_value=2147483647,var_off=(0x0; 0x7fffffff)) R6=ctx(id=0,off=0,imm=0) R7_w=inv4294967162 R9=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-104=????mmmm fp-152=??0m0000 fp-160=mmmmmmmm fp-168=????0000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=00000000 fp-208=00000000 fp-216=00000000 fp-224=0000mmmm fp-232=0000000m fp-240=inv
            	; tuple.dport = l4hdr.dport;
            	53: (69) r1 = *(u16 *)(r10 -102)
            	; tuple.dport = l4hdr.dport;
            	54: (6b) *(u16 *)(r10 -152) = r1
            	; tuple.sport = l4hdr.sport;
            	55: (69) r1 = *(u16 *)(r10 -104)
            	;
            	56: (6b) *(u16 *)(r10 -150) = r1
            	57: (bf) r2 = r10
            	58: (07) r2 += -160
            	; return map_lookup_elem(map, tuple);
            	59: (18) r1 = 0xffff9ca4c89c1800
            	61: (85) call bpf_map_lookup_elem#1
            	;
            	62: (bf) r5 = r0
            	63: (07) r5 += 32
            	R5 pointer arithmetic on map_value_or_null prohibited, null-check it first
            	processed 60 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 2
    --- FAIL: TestBPF/tc_nodeport_lb4_dsr_lb.o (0.04s)
        bpf_test.go:150: verifier error: load program: permission denied:
            	; int tail_handle_snat_fwd_ipv4(struct __ctx_buff *ctx)
            	0: (bf) r6 = r1
            	; return (void *)(unsigned long)ctx->data_end;
            	1: (61) r1 = *(u32 *)(r6 +80)
            	; return (void *)(unsigned long)ctx->data;
            	2: (61) r8 = *(u32 *)(r6 +76)
            	; if (data + tot_len > data_end)
            	3: (bf) r2 = r8
            	4: (07) r2 += 34
            	; if (data + tot_len > data_end)
            	5: (2d) if r2 > r1 goto pc+536
            	 R1_w=pkt_end(id=0,off=0,imm=0) R2_w=pkt(id=0,off=34,r=34,imm=0) R6_w=ctx(id=0,off=0,imm=0) R8_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0
            	; ip4->saddr == IPV4_DIRECT_ROUTING) {
            	6: (61) r1 = *(u32 *)(r8 +26)
            	; if (DIRECT_ROUTING_DEV_IFINDEX == NATIVE_DEV_IFINDEX &&
            	7: (15) if r1 == 0xa000a01 goto pc+335
            	
            	from 7 to 343: R1=inv167774721 R2=pkt(id=0,off=34,r=34,imm=0) R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0
            	;
            	343: (bf) r2 = r8
            	344: (07) r2 += 14
            	345: (b7) r4 = 0
            	; struct ipv4_ct_tuple tuple = {};
            	346: (63) *(u32 *)(r10 -152) = r4
            	last_idx 346 first_idx 343
            	regs=10 stack=0 before 345: (b7) r4 = 0
            	; tuple->nexthdr = ip4->protocol;
            	347: (71) r1 = *(u8 *)(r8 +23)
            	; tuple->nexthdr = ip4->protocol;
            	348: (73) *(u8 *)(r10 -148) = r1
            	; tuple->daddr = ip4->daddr;
            	349: (61) r3 = *(u32 *)(r8 +30)
            	; tuple->daddr = ip4->daddr;
            	350: (63) *(u32 *)(r10 -160) = r3
            	; tuple->saddr = ip4->saddr;
            	351: (61) r3 = *(u32 *)(r8 +26)
            	; tuple->flags = dir;
            	352: (73) *(u8 *)(r10 -147) = r4
            	; tuple->saddr = ip4->saddr;
            	353: (63) *(u32 *)(r10 -156) = r3
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	354: (1f) r2 -= r8
            	; return ip4->ihl * 4;
            	355: (71) r3 = *(u8 *)(r8 +14)
            	; return ip4->ihl * 4;
            	356: (67) r3 <<= 2
            	357: (57) r3 &= 60
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	358: (0f) r2 += r3
            	; switch (tuple.nexthdr) {
            	359: (7b) *(u64 *)(r10 -176) = r2
            	360: (15) if r1 == 0x1 goto pc+34
            	 R1_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0) R3_w=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R4_w=invP0 R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-176_w=mmmmmmmm
            	361: (15) if r1 == 0x11 goto pc+3
            	 R1_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0) R3_w=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R4_w=invP0 R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-176_w=mmmmmmmm
            	;
            	362: (18) r7 = 0xffffff53
            	; switch (tuple.nexthdr) {
            	364: (55) if r1 != 0x6 goto pc+2742
            	 R1=inv6 R2=inv(id=0) R3=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R4=invP0 R6=ctx(id=0,off=0,imm=0) R7=inv4294967123 R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-176=mmmmmmmm
            	365: (bf) r3 = r10
            	; if (ctx_load_bytes(ctx, off, &l4hdr, sizeof(l4hdr)) < 0)
            	366: (07) r3 += -104
            	367: (bf) r1 = r6
            	368: (79) r2 = *(u64 *)(r10 -176)
            	369: (b7) r4 = 4
            	370: (85) call bpf_skb_load_bytes#26
            	last_idx 370 first_idx 364
            	regs=10 stack=0 before 369: (b7) r4 = 4
            	371: (18) r7 = 0xffffff7a
            	373: (67) r0 <<= 32
            	374: (c7) r0 s>>= 32
            	; if (ctx_load_bytes(ctx, off, &l4hdr, sizeof(l4hdr)) < 0)
            	375: (c5) if r0 s< 0x0 goto pc+2731
            	 R0=inv(id=0,umax_value=2147483647,var_off=(0x0; 0x7fffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv4294967162 R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-104=????mmmm fp-152=??0m0000 fp-160=mmmmmmmm fp-176=mmmmmmmm
            	; tuple.dport = l4hdr.dport;
            	376: (69) r1 = *(u16 *)(r10 -102)
            	; tuple.dport = l4hdr.dport;
            	377: (6b) *(u16 *)(r10 -152) = r1
            	; tuple.sport = l4hdr.sport;
            	378: (69) r1 = *(u16 *)(r10 -104)
            	;
            	379: (6b) *(u16 *)(r10 -150) = r1
            	380: (57) r1 &= 128
            	;
            	381: (18) r7 = 0xffffff53
            	; return (!target->from_local_endpoint && !target->src_from_world &&
            	383: (15) if r1 == 0x0 goto pc+2723
            	 R0=inv(id=0,umax_value=2147483647,var_off=(0x0; 0x7fffffff)) R1_w=inv(id=0,umax_value=128,var_off=(0x0; 0x80)) R6=ctx(id=0,off=0,imm=0) R7_w=inv4294967123 R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-104=????mmmm fp-152=??0mmmmm fp-160=mmmmmmmm fp-176=mmmmmmmm
            	384: (bf) r2 = r10
            	;
            	385: (07) r2 += -160
            	; return map_lookup_elem(map, tuple);
            	386: (18) r1 = 0xffff9ca8d0d04000
            	388: (85) call bpf_map_lookup_elem#1
            	;
            	389: (bf) r5 = r0
            	390: (07) r5 += 32
            	R5 pointer arithmetic on map_value_or_null prohibited, null-check it first
            	processed 522 insns (limit 1000000) max_states_per_insn 1 total_states 39 peak_states 39 mark_read 22
    --- FAIL: TestBPF/tc_nodeport_lb4_nat_backend.o (0.03s)
        bpf_test.go:150: verifier error: load program: permission denied:
            	; int tail_handle_snat_fwd_ipv4(struct __ctx_buff *ctx)
            	0: (bf) r6 = r1
            	; return (void *)(unsigned long)ctx->data_end;
            	1: (61) r1 = *(u32 *)(r6 +80)
            	; return (void *)(unsigned long)ctx->data;
            	2: (61) r8 = *(u32 *)(r6 +76)
            	; if (data + tot_len > data_end)
            	3: (bf) r2 = r8
            	4: (07) r2 += 34
            	; if (data + tot_len > data_end)
            	5: (2d) if r2 > r1 goto pc+533
            	 R1_w=pkt_end(id=0,off=0,imm=0) R2_w=pkt(id=0,off=34,r=34,imm=0) R6_w=ctx(id=0,off=0,imm=0) R8_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0
            	; ip4->saddr == IPV4_DIRECT_ROUTING) {
            	6: (61) r1 = *(u32 *)(r8 +26)
            	; if (DIRECT_ROUTING_DEV_IFINDEX == NATIVE_DEV_IFINDEX &&
            	7: (15) if r1 == 0x0 goto pc+335
            	
            	from 7 to 343: R1=inv0 R2=pkt(id=0,off=34,r=34,imm=0) R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0
            	;
            	343: (bf) r2 = r8
            	344: (07) r2 += 14
            	345: (b7) r4 = 0
            	; struct ipv4_ct_tuple tuple = {};
            	346: (63) *(u32 *)(r10 -152) = r4
            	last_idx 346 first_idx 343
            	regs=10 stack=0 before 345: (b7) r4 = 0
            	; tuple->nexthdr = ip4->protocol;
            	347: (71) r1 = *(u8 *)(r8 +23)
            	; tuple->nexthdr = ip4->protocol;
            	348: (73) *(u8 *)(r10 -148) = r1
            	; tuple->daddr = ip4->daddr;
            	349: (61) r3 = *(u32 *)(r8 +30)
            	; tuple->daddr = ip4->daddr;
            	350: (63) *(u32 *)(r10 -160) = r3
            	; tuple->saddr = ip4->saddr;
            	351: (61) r3 = *(u32 *)(r8 +26)
            	; tuple->flags = dir;
            	352: (73) *(u8 *)(r10 -147) = r4
            	; tuple->saddr = ip4->saddr;
            	353: (63) *(u32 *)(r10 -156) = r3
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	354: (1f) r2 -= r8
            	; return ip4->ihl * 4;
            	355: (71) r3 = *(u8 *)(r8 +14)
            	; return ip4->ihl * 4;
            	356: (67) r3 <<= 2
            	357: (57) r3 &= 60
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	358: (0f) r2 += r3
            	; switch (tuple.nexthdr) {
            	359: (7b) *(u64 *)(r10 -176) = r2
            	360: (15) if r1 == 0x1 goto pc+34
            	 R1_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0) R3_w=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R4_w=invP0 R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-176_w=mmmmmmmm
            	361: (15) if r1 == 0x11 goto pc+3
            	 R1_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0) R3_w=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R4_w=invP0 R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-176_w=mmmmmmmm
            	;
            	362: (18) r7 = 0xffffff53
            	; switch (tuple.nexthdr) {
            	364: (55) if r1 != 0x6 goto pc+2708
            	 R1=inv6 R2=inv(id=0) R3=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R4=invP0 R6=ctx(id=0,off=0,imm=0) R7=inv4294967123 R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-176=mmmmmmmm
            	365: (bf) r3 = r10
            	; if (ctx_load_bytes(ctx, off, &l4hdr, sizeof(l4hdr)) < 0)
            	366: (07) r3 += -104
            	367: (bf) r1 = r6
            	368: (79) r2 = *(u64 *)(r10 -176)
            	369: (b7) r4 = 4
            	370: (85) call bpf_skb_load_bytes#26
            	last_idx 370 first_idx 364
            	regs=10 stack=0 before 369: (b7) r4 = 4
            	371: (18) r7 = 0xffffff7a
            	373: (67) r0 <<= 32
            	374: (c7) r0 s>>= 32
            	; if (ctx_load_bytes(ctx, off, &l4hdr, sizeof(l4hdr)) < 0)
            	375: (c5) if r0 s< 0x0 goto pc+2697
            	 R0=inv(id=0,umax_value=2147483647,var_off=(0x0; 0x7fffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv4294967162 R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-104=????mmmm fp-152=??0m0000 fp-160=mmmmmmmm fp-176=mmmmmmmm
            	; tuple.dport = l4hdr.dport;
            	376: (69) r1 = *(u16 *)(r10 -102)
            	; tuple.dport = l4hdr.dport;
            	377: (6b) *(u16 *)(r10 -152) = r1
            	; tuple.sport = l4hdr.sport;
            	378: (69) r1 = *(u16 *)(r10 -104)
            	;
            	379: (6b) *(u16 *)(r10 -150) = r1
            	380: (57) r1 &= 128
            	;
            	381: (18) r7 = 0xffffff53
            	; return (!target->from_local_endpoint && !target->src_from_world &&
            	383: (15) if r1 == 0x0 goto pc+2689
            	 R0=inv(id=0,umax_value=2147483647,var_off=(0x0; 0x7fffffff)) R1_w=inv(id=0,umax_value=128,var_off=(0x0; 0x80)) R6=ctx(id=0,off=0,imm=0) R7_w=inv4294967123 R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-104=????mmmm fp-152=??0mmmmm fp-160=mmmmmmmm fp-176=mmmmmmmm
            	384: (bf) r2 = r10
            	;
            	385: (07) r2 += -160
            	; return map_lookup_elem(map, tuple);
            	386: (18) r1 = 0xffff9ca804a2f000
            	388: (85) call bpf_map_lookup_elem#1
            	;
            	389: (bf) r5 = r0
            	390: (07) r5 += 32
            	R5 pointer arithmetic on map_value_or_null prohibited, null-check it first
            	processed 522 insns (limit 1000000) max_states_per_insn 1 total_states 39 peak_states 39 mark_read 22
    --- FAIL: TestBPF/tc_nodeport_lb4_nat_lb.o (0.03s)
        bpf_test.go:150: verifier error: load program: permission denied:
            	; int tail_handle_snat_fwd_ipv4(struct __ctx_buff *ctx)
            	0: (bf) r6 = r1
            	; return (void *)(unsigned long)ctx->data_end;
            	1: (61) r1 = *(u32 *)(r6 +80)
            	; return (void *)(unsigned long)ctx->data;
            	2: (61) r8 = *(u32 *)(r6 +76)
            	; if (data + tot_len > data_end)
            	3: (bf) r2 = r8
            	4: (07) r2 += 34
            	; if (data + tot_len > data_end)
            	5: (2d) if r2 > r1 goto pc+536
            	 R1_w=pkt_end(id=0,off=0,imm=0) R2_w=pkt(id=0,off=34,r=34,imm=0) R6_w=ctx(id=0,off=0,imm=0) R8_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0
            	; ip4->saddr == IPV4_DIRECT_ROUTING) {
            	6: (61) r1 = *(u32 *)(r8 +26)
            	; if (DIRECT_ROUTING_DEV_IFINDEX == NATIVE_DEV_IFINDEX &&
            	7: (15) if r1 == 0xa000a01 goto pc+335
            	
            	from 7 to 343: R1=inv167774721 R2=pkt(id=0,off=34,r=34,imm=0) R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0
            	;
            	343: (bf) r2 = r8
            	344: (07) r2 += 14
            	345: (b7) r4 = 0
            	; struct ipv4_ct_tuple tuple = {};
            	346: (63) *(u32 *)(r10 -152) = r4
            	last_idx 346 first_idx 343
            	regs=10 stack=0 before 345: (b7) r4 = 0
            	; tuple->nexthdr = ip4->protocol;
            	347: (71) r1 = *(u8 *)(r8 +23)
            	; tuple->nexthdr = ip4->protocol;
            	348: (73) *(u8 *)(r10 -148) = r1
            	; tuple->daddr = ip4->daddr;
            	349: (61) r3 = *(u32 *)(r8 +30)
            	; tuple->daddr = ip4->daddr;
            	350: (63) *(u32 *)(r10 -160) = r3
            	; tuple->saddr = ip4->saddr;
            	351: (61) r3 = *(u32 *)(r8 +26)
            	; tuple->flags = dir;
            	352: (73) *(u8 *)(r10 -147) = r4
            	; tuple->saddr = ip4->saddr;
            	353: (63) *(u32 *)(r10 -156) = r3
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	354: (1f) r2 -= r8
            	; return ip4->ihl * 4;
            	355: (71) r3 = *(u8 *)(r8 +14)
            	; return ip4->ihl * 4;
            	356: (67) r3 <<= 2
            	357: (57) r3 &= 60
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	358: (0f) r2 += r3
            	; switch (tuple.nexthdr) {
            	359: (7b) *(u64 *)(r10 -176) = r2
            	360: (15) if r1 == 0x1 goto pc+34
            	 R1_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0) R3_w=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R4_w=invP0 R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-176_w=mmmmmmmm
            	361: (15) if r1 == 0x11 goto pc+3
            	 R1_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0) R3_w=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R4_w=invP0 R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-176_w=mmmmmmmm
            	;
            	362: (18) r7 = 0xffffff53
            	; switch (tuple.nexthdr) {
            	364: (55) if r1 != 0x6 goto pc+2742
            	 R1=inv6 R2=inv(id=0) R3=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R4=invP0 R6=ctx(id=0,off=0,imm=0) R7=inv4294967123 R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-152=??0m0000 fp-160=mmmmmmmm fp-176=mmmmmmmm
            	365: (bf) r3 = r10
            	; if (ctx_load_bytes(ctx, off, &l4hdr, sizeof(l4hdr)) < 0)
            	366: (07) r3 += -104
            	367: (bf) r1 = r6
            	368: (79) r2 = *(u64 *)(r10 -176)
            	369: (b7) r4 = 4
            	370: (85) call bpf_skb_load_bytes#26
            	last_idx 370 first_idx 364
            	regs=10 stack=0 before 369: (b7) r4 = 4
            	371: (18) r7 = 0xffffff7a
            	373: (67) r0 <<= 32
            	374: (c7) r0 s>>= 32
            	; if (ctx_load_bytes(ctx, off, &l4hdr, sizeof(l4hdr)) < 0)
            	375: (c5) if r0 s< 0x0 goto pc+2731
            	 R0=inv(id=0,umax_value=2147483647,var_off=(0x0; 0x7fffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv4294967162 R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-104=????mmmm fp-152=??0m0000 fp-160=mmmmmmmm fp-176=mmmmmmmm
            	; tuple.dport = l4hdr.dport;
            	376: (69) r1 = *(u16 *)(r10 -102)
            	; tuple.dport = l4hdr.dport;
            	377: (6b) *(u16 *)(r10 -152) = r1
            	; tuple.sport = l4hdr.sport;
            	378: (69) r1 = *(u16 *)(r10 -104)
            	;
            	379: (6b) *(u16 *)(r10 -150) = r1
            	380: (57) r1 &= 128
            	;
            	381: (18) r7 = 0xffffff53
            	; return (!target->from_local_endpoint && !target->src_from_world &&
            	383: (15) if r1 == 0x0 goto pc+2723
            	 R0=inv(id=0,umax_value=2147483647,var_off=(0x0; 0x7fffffff)) R1_w=inv(id=0,umax_value=128,var_off=(0x0; 0x80)) R6=ctx(id=0,off=0,imm=0) R7_w=inv4294967123 R8=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-104=????mmmm fp-152=??0mmmmm fp-160=mmmmmmmm fp-176=mmmmmmmm
            	384: (bf) r2 = r10
            	;
            	385: (07) r2 += -160
            	; return map_lookup_elem(map, tuple);
            	386: (18) r1 = 0xffff9ca8d0d05c00
            	388: (85) call bpf_map_lookup_elem#1
            	;
            	389: (bf) r5 = r0
            	390: (07) r5 += 32
            	R5 pointer arithmetic on map_value_or_null prohibited, null-check it first
            	processed 522 insns (limit 1000000) max_states_per_insn 1 total_states 39 peak_states 39 mark_read 22
    --- FAIL: TestBPF/xdp_nodeport_lb4_dsr_lb.o (0.03s)
        bpf_test.go:150: verifier error: load program: permission denied:
            	; int tail_nodeport_nat_egress_ipv4(struct __ctx_buff *ctx)
            	0: (b7) r4 = 0
            	; struct bpf_fib_lookup_padded fib_params = {
            	1: (7b) *(u64 *)(r10 -176) = r4
            	last_idx 1 first_idx 0
            	regs=10 stack=0 before 0: (b7) r4 = 0
            	2: (7b) *(u64 *)(r10 -168) = r4
            	3: (63) *(u32 *)(r10 -160) = r4
            	4: (6b) *(u16 *)(r10 -156) = r4
            	5: (63) *(u32 *)(r10 -184) = r4
            	6: (6b) *(u16 *)(r10 -180) = r4
            	; struct ipv4_ct_tuple tuple = {};
            	7: (6b) *(u16 *)(r10 -140) = r4
            	8: (63) *(u32 *)(r10 -144) = r4
            	9: (7b) *(u64 *)(r10 -152) = r4
            	10: (18) r8 = 0xffffff7a
            	; return (void *)(unsigned long)ctx->data_end;
            	12: (61) r2 = *(u32 *)(r1 +4)
            	13: (7b) *(u64 *)(r10 -192) = r1
            	; return (void *)(unsigned long)ctx->data;
            	14: (61) r6 = *(u32 *)(r1 +0)
            	; if (data + tot_len > data_end)
            	15: (bf) r3 = r6
            	16: (07) r3 += 34
            	; if (data + tot_len > data_end)
            	17: (2d) if r3 > r2 goto pc+2966
            	 R1=ctx(id=0,off=0,imm=0) R2_w=pkt_end(id=0,off=0,imm=0) R3_w=pkt(id=0,off=34,r=34,imm=0) R4_w=invP0 R6_w=pkt(id=0,off=0,r=34,imm=0) R8_w=inv4294967162 R10=fp0 fp-144=??000000 fp-152_w=00000000 fp-160=??000000 fp-168_w=00000000 fp-176_w=00000000 fp-184=??000000 fp-192_w=ctx
            	; tuple->nexthdr = ip4->protocol;
            	18: (71) r2 = *(u8 *)(r6 +23)
            	; tuple->nexthdr = ip4->protocol;
            	19: (73) *(u8 *)(r10 -140) = r2
            	; tuple->daddr = ip4->daddr;
            	20: (61) r3 = *(u32 *)(r6 +30)
            	; tuple->daddr = ip4->daddr;
            	21: (63) *(u32 *)(r10 -152) = r3
            	; tuple->saddr = ip4->saddr;
            	22: (61) r3 = *(u32 *)(r6 +26)
            	; tuple->flags = dir;
            	23: (73) *(u8 *)(r10 -139) = r4
            	; tuple->saddr = ip4->saddr;
            	24: (63) *(u32 *)(r10 -148) = r3
            	; return ip4->ihl * 4;
            	25: (71) r5 = *(u8 *)(r6 +14)
            	; return ip4->ihl * 4;
            	26: (67) r5 <<= 2
            	27: (57) r5 &= 60
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	28: (07) r5 += 14
            	; switch (tuple.nexthdr) {
            	29: (15) if r2 == 0x1 goto pc+28
            	 R1=ctx(id=0,off=0,imm=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4_w=invP0 R5_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R6_w=pkt(id=0,off=0,r=34,imm=0) R8_w=inv4294967162 R10=fp0 fp-144=??0m0000 fp-152_w=mmmmmmmm fp-160=??000000 fp-168_w=00000000 fp-176_w=00000000 fp-184=??000000 fp-192_w=ctx
            	30: (15) if r2 == 0x11 goto pc+3
            	 R1=ctx(id=0,off=0,imm=0) R2=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4=invP0 R5=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R6=pkt(id=0,off=0,r=34,imm=0) R8=inv4294967162 R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=??000000 fp-168=00000000 fp-176=00000000 fp-184=??000000 fp-192=ctx
            	;
            	31: (18) r8 = 0xffffff53
            	; switch (tuple.nexthdr) {
            	33: (55) if r2 != 0x6 goto pc+2950
            	 R1=ctx(id=0,off=0,imm=0) R2=inv6 R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4=invP0 R5=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R6=pkt(id=0,off=0,r=34,imm=0) R8_w=inv4294967123 R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=??000000 fp-168=00000000 fp-176=00000000 fp-184=??000000 fp-192=ctx
            	; asm volatile("r1 = *(u32 *)(%[ctx] +0)\n\t"
            	34: (79) r3 = *(u64 *)(r10 -192)
            	35: (61) r1 = *(u32 *)(r3 +0)
            	36: (61) r2 = *(u32 *)(r3 +4)
            	37: (57) r5 &= 255
            	38: (0f) r1 += r5
            	last_idx 38 first_idx 30
            	regs=20 stack=0 before 37: (57) r5 &= 255
            	regs=20 stack=0 before 36: (61) r2 = *(u32 *)(r3 +4)
            	regs=20 stack=0 before 35: (61) r1 = *(u32 *)(r3 +0)
            	regs=20 stack=0 before 34: (79) r3 = *(u64 *)(r10 -192)
            	regs=20 stack=0 before 33: (55) if r2 != 0x6 goto pc+2950
            	regs=20 stack=0 before 31: (18) r8 = 0xffffff53
            	regs=20 stack=0 before 30: (15) if r2 == 0x11 goto pc+3
            	 R1=ctx(id=0,off=0,imm=0) R2_rw=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4_w=invP0 R5_rw=invP(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R6_w=pkt(id=0,off=0,r=34,imm=0) R8_w=inv4294967162 R10=fp0 fp-144=??0m0000 fp-152_w=mmmmmmmm fp-160=??000000 fp-168_w=00000000 fp-176_w=00000000 fp-184=??000000 fp-192_rw=ctx
            	parent didn't have regs=20 stack=0 marks
            	last_idx 29 first_idx 0
            	regs=20 stack=0 before 29: (15) if r2 == 0x1 goto pc+28
            	regs=20 stack=0 before 28: (07) r5 += 14
            	regs=20 stack=0 before 27: (57) r5 &= 60
            	regs=20 stack=0 before 26: (67) r5 <<= 2
            	regs=20 stack=0 before 25: (71) r5 = *(u8 *)(r6 +14)
            	39: (bf) r4 = r1
            	40: (07) r1 += 4
            	41: (2d) if r1 > r2 goto pc+2
            	 R1=pkt(id=1,off=4,r=4,umin_value=2,umax_value=74,var_off=(0x2; 0x7c),s32_max_value=126,u32_max_value=126) R2=pkt_end(id=0,off=0,imm=0) R3=ctx(id=0,off=0,imm=0) R4=pkt(id=1,off=0,r=4,umin_value=2,umax_value=74,var_off=(0x2; 0x7c),s32_max_value=126,u32_max_value=126) R5=invP(id=0,umin_value=2,umax_value=74,var_off=(0x2; 0x7c)) R6=pkt(id=0,off=0,r=34,imm=0) R8=inv4294967123 R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=??000000 fp-168=00000000 fp-176=00000000 fp-184=??000000 fp-192=ctx
            	42: (b7) r3 = 0
            	43: (05) goto pc+1
            	45: (67) r3 <<= 32
            	46: (bf) r1 = r3
            	47: (77) r1 >>= 32
            	; if (!ret)
            	48: (55) if r1 != 0x0 goto pc+4
            	last_idx 48 first_idx 41
            	regs=2 stack=0 before 47: (77) r1 >>= 32
            	regs=2 stack=0 before 46: (bf) r1 = r3
            	regs=8 stack=0 before 45: (67) r3 <<= 32
            	regs=8 stack=0 before 43: (05) goto pc+1
            	regs=8 stack=0 before 42: (b7) r3 = 0
            	; case  4: __it_mob(d, s, 32);
            	49: (61) r1 = *(u32 *)(r4 +0)
            	50: (bf) r2 = r1
            	51: (77) r2 >>= 16
            	52: (05) goto pc+23
            	; if (ctx_load_bytes(ctx, off, &icmphdr, sizeof(icmphdr)) < 0)
            	76: (7b) *(u64 *)(r10 -200) = r5
            	;
            	77: (6b) *(u16 *)(r10 -142) = r1
            	78: (6b) *(u16 *)(r10 -144) = r2
            	79: (bf) r2 = r10
            	80: (07) r2 += -152
            	; return map_lookup_elem(map, tuple);
            	81: (18) r1 = 0xffff9ca82c914c00
            	83: (85) call bpf_map_lookup_elem#1
            	;
            	84: (bf) r2 = r0
            	85: (07) r2 += 32
            	R2 pointer arithmetic on map_value_or_null prohibited, null-check it first
            	processed 59 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 3
    --- FAIL: TestBPF/xdp_nodeport_lb4_nat_backend.o (0.04s)
        bpf_test.go:150: verifier error: load program: permission denied:
            	; int tail_nodeport_nat_egress_ipv4(struct __ctx_buff *ctx)
            	0: (bf) r6 = r1
            	1: (b7) r1 = 0
            	; struct bpf_fib_lookup_padded fib_params = {
            	2: (7b) *(u64 *)(r10 -224) = r1
            	last_idx 2 first_idx 0
            	regs=2 stack=0 before 1: (b7) r1 = 0
            	3: (b7) r2 = 2
            	; .l = {
            	4: (73) *(u8 *)(r10 -224) = r2
            	; struct bpf_fib_lookup_padded fib_params = {
            	5: (63) *(u32 *)(r10 -160) = r1
            	6: (7b) *(u64 *)(r10 -168) = r1
            	7: (7b) *(u64 *)(r10 -176) = r1
            	8: (7b) *(u64 *)(r10 -184) = r1
            	9: (7b) *(u64 *)(r10 -192) = r1
            	10: (7b) *(u64 *)(r10 -200) = r1
            	11: (7b) *(u64 *)(r10 -208) = r1
            	12: (7b) *(u64 *)(r10 -216) = r1
            	; return ctx->ingress_ifindex;
            	13: (61) r2 = *(u32 *)(r6 +12)
            	; .l = {
            	14: (63) *(u32 *)(r10 -216) = r2
            	; struct ipv4_ct_tuple tuple = {};
            	15: (6b) *(u16 *)(r10 -140) = r1
            	16: (63) *(u32 *)(r10 -144) = r1
            	17: (7b) *(u64 *)(r10 -152) = r1
            	18: (18) r9 = 0xffffff7a
            	; return (void *)(unsigned long)ctx->data_end;
            	20: (61) r2 = *(u32 *)(r6 +4)
            	; return (void *)(unsigned long)ctx->data;
            	21: (61) r8 = *(u32 *)(r6 +0)
            	; if (data + tot_len > data_end)
            	22: (bf) r3 = r8
            	23: (07) r3 += 34
            	; if (data + tot_len > data_end)
            	24: (2d) if r3 > r2 goto pc+2972
            	 R1_w=invP0 R2_w=pkt_end(id=0,off=0,imm=0) R3_w=pkt(id=0,off=34,r=34,imm=0) R6_w=ctx(id=0,off=0,imm=0) R8_w=pkt(id=0,off=0,r=34,imm=0) R9_w=inv4294967162 R10=fp0 fp-144=??000000 fp-152_w=00000000 fp-160=????0000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=0000mmmm fp-224_w=0000000m
            	; tuple->nexthdr = ip4->protocol;
            	25: (71) r2 = *(u8 *)(r8 +23)
            	; tuple->nexthdr = ip4->protocol;
            	26: (73) *(u8 *)(r10 -140) = r2
            	; tuple->daddr = ip4->daddr;
            	27: (61) r3 = *(u32 *)(r8 +30)
            	; tuple->daddr = ip4->daddr;
            	28: (63) *(u32 *)(r10 -152) = r3
            	; tuple->saddr = ip4->saddr;
            	29: (61) r3 = *(u32 *)(r8 +26)
            	; tuple->flags = dir;
            	30: (73) *(u8 *)(r10 -139) = r1
            	; tuple->saddr = ip4->saddr;
            	31: (63) *(u32 *)(r10 -148) = r3
            	; return ip4->ihl * 4;
            	32: (71) r1 = *(u8 *)(r8 +14)
            	; return ip4->ihl * 4;
            	33: (67) r1 <<= 2
            	34: (57) r1 &= 60
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	35: (07) r1 += 14
            	36: (7b) *(u64 *)(r10 -232) = r1
            	; switch (tuple.nexthdr) {
            	37: (15) if r2 == 0x1 goto pc+28
            	 R1_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6_w=ctx(id=0,off=0,imm=0) R8_w=pkt(id=0,off=0,r=34,imm=0) R9_w=inv4294967162 R10=fp0 fp-144=??0m0000 fp-152_w=mmmmmmmm fp-160=????0000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=0000mmmm fp-224_w=0000000m fp-232_w=inv
            	38: (15) if r2 == 0x11 goto pc+3
            	 R1=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R2=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R9=inv4294967162 R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=????0000 fp-168=00000000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=00000000 fp-208=00000000 fp-216=0000mmmm fp-224=0000000m fp-232=inv
            	;
            	39: (18) r9 = 0xffffff53
            	; switch (tuple.nexthdr) {
            	41: (55) if r2 != 0x6 goto pc+2955
            	 R1=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R2=inv6 R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R9_w=inv4294967123 R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=????0000 fp-168=00000000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=00000000 fp-208=00000000 fp-216=0000mmmm fp-224=0000000m fp-232=inv
            	; asm volatile("r1 = *(u32 *)(%[ctx] +0)\n\t"
            	42: (79) r3 = *(u64 *)(r10 -232)
            	43: (61) r1 = *(u32 *)(r6 +0)
            	44: (61) r2 = *(u32 *)(r6 +4)
            	45: (57) r3 &= 255
            	46: (0f) r1 += r3
            	last_idx 46 first_idx 38
            	regs=8 stack=0 before 45: (57) r3 &= 255
            	regs=8 stack=0 before 44: (61) r2 = *(u32 *)(r6 +4)
            	regs=8 stack=0 before 43: (61) r1 = *(u32 *)(r6 +0)
            	regs=8 stack=0 before 42: (79) r3 = *(u64 *)(r10 -232)
            	regs=0 stack=10000000 before 41: (55) if r2 != 0x6 goto pc+2955
            	regs=0 stack=10000000 before 39: (18) r9 = 0xffffff53
            	regs=0 stack=10000000 before 38: (15) if r2 == 0x11 goto pc+3
            	 R1_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R2_rw=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6_rw=ctx(id=0,off=0,imm=0) R8_w=pkt(id=0,off=0,r=34,imm=0) R9_w=inv4294967162 R10=fp0 fp-144=??0m0000 fp-152_w=mmmmmmmm fp-160=????0000 fp-168_w=00000000 fp-176_w=00000000 fp-184_w=00000000 fp-192_w=00000000 fp-200_w=00000000 fp-208_w=00000000 fp-216_w=0000mmmm fp-224_w=0000000m fp-232_rw=invP
            	parent didn't have regs=0 stack=10000000 marks
            	last_idx 37 first_idx 0
            	regs=0 stack=10000000 before 37: (15) if r2 == 0x1 goto pc+28
            	regs=0 stack=10000000 before 36: (7b) *(u64 *)(r10 -232) = r1
            	regs=2 stack=0 before 35: (07) r1 += 14
            	regs=2 stack=0 before 34: (57) r1 &= 60
            	regs=2 stack=0 before 33: (67) r1 <<= 2
            	regs=2 stack=0 before 32: (71) r1 = *(u8 *)(r8 +14)
            	47: (bf) r4 = r1
            	48: (07) r1 += 4
            	49: (2d) if r1 > r2 goto pc+2
            	 R1=pkt(id=1,off=4,r=4,umin_value=2,umax_value=74,var_off=(0x2; 0x7c),s32_max_value=126,u32_max_value=126) R2=pkt_end(id=0,off=0,imm=0) R3=invP(id=0,umin_value=2,umax_value=74,var_off=(0x2; 0x7c)) R4=pkt(id=1,off=0,r=4,umin_value=2,umax_value=74,var_off=(0x2; 0x7c),s32_max_value=126,u32_max_value=126) R6=ctx(id=0,off=0,imm=0) R8=pkt(id=0,off=0,r=34,imm=0) R9=inv4294967123 R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=????0000 fp-168=00000000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=00000000 fp-208=00000000 fp-216=0000mmmm fp-224=0000000m fp-232=inv
            	50: (b7) r3 = 0
            	51: (05) goto pc+1
            	53: (67) r3 <<= 32
            	54: (bf) r1 = r3
            	55: (77) r1 >>= 32
            	; if (!ret)
            	56: (55) if r1 != 0x0 goto pc+4
            	last_idx 56 first_idx 49
            	regs=2 stack=0 before 55: (77) r1 >>= 32
            	regs=2 stack=0 before 54: (bf) r1 = r3
            	regs=8 stack=0 before 53: (67) r3 <<= 32
            	regs=8 stack=0 before 51: (05) goto pc+1
            	regs=8 stack=0 before 50: (b7) r3 = 0
            	; case  4: __it_mob(d, s, 32);
            	57: (61) r1 = *(u32 *)(r4 +0)
            	58: (bf) r2 = r1
            	59: (77) r2 >>= 16
            	60: (05) goto pc+23
            	;
            	84: (6b) *(u16 *)(r10 -142) = r1
            	85: (6b) *(u16 *)(r10 -144) = r2
            	86: (bf) r2 = r10
            	87: (07) r2 += -152
            	; return map_lookup_elem(map, tuple);
            	88: (18) r1 = 0xffff9ca683e52000
            	90: (85) call bpf_map_lookup_elem#1
            	;
            	91: (bf) r2 = r0
            	92: (07) r2 += 32
            	R2 pointer arithmetic on map_value_or_null prohibited, null-check it first
            	processed 66 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 3
    --- FAIL: TestBPF/xdp_nodeport_lb4_nat_lb.o (0.03s)
        bpf_test.go:150: verifier error: load program: permission denied:
            	; int tail_nodeport_nat_egress_ipv4(struct __ctx_buff *ctx)
            	0: (b7) r8 = 0
            	; struct bpf_fib_lookup_padded fib_params = {
            	1: (7b) *(u64 *)(r10 -176) = r8
            	last_idx 1 first_idx 0
            	regs=100 stack=0 before 0: (b7) r8 = 0
            	2: (7b) *(u64 *)(r10 -168) = r8
            	3: (63) *(u32 *)(r10 -160) = r8
            	4: (6b) *(u16 *)(r10 -156) = r8
            	5: (63) *(u32 *)(r10 -184) = r8
            	6: (6b) *(u16 *)(r10 -180) = r8
            	; struct ipv4_ct_tuple tuple = {};
            	7: (6b) *(u16 *)(r10 -140) = r8
            	8: (63) *(u32 *)(r10 -144) = r8
            	9: (7b) *(u64 *)(r10 -152) = r8
            	10: (18) r9 = 0xffffff7a
            	; return (void *)(unsigned long)ctx->data_end;
            	12: (61) r3 = *(u32 *)(r1 +4)
            	13: (7b) *(u64 *)(r10 -192) = r1
            	; return (void *)(unsigned long)ctx->data;
            	14: (61) r6 = *(u32 *)(r1 +0)
            	; if (data + tot_len > data_end)
            	15: (bf) r2 = r6
            	16: (07) r2 += 34
            	; if (data + tot_len > data_end)
            	17: (2d) if r2 > r3 goto pc+2974
            	 R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=34,r=34,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R6_w=pkt(id=0,off=0,r=34,imm=0) R8_w=invP0 R9_w=inv4294967162 R10=fp0 fp-144=??000000 fp-152_w=00000000 fp-160=??000000 fp-168_w=00000000 fp-176_w=00000000 fp-184=??000000 fp-192_w=ctx
            	; tuple->nexthdr = ip4->protocol;
            	18: (71) r1 = *(u8 *)(r6 +23)
            	; tuple->nexthdr = ip4->protocol;
            	19: (73) *(u8 *)(r10 -140) = r1
            	; tuple->daddr = ip4->daddr;
            	20: (61) r2 = *(u32 *)(r6 +30)
            	; tuple->daddr = ip4->daddr;
            	21: (63) *(u32 *)(r10 -152) = r2
            	; tuple->saddr = ip4->saddr;
            	22: (61) r2 = *(u32 *)(r6 +26)
            	; tuple->flags = dir;
            	23: (73) *(u8 *)(r10 -139) = r8
            	; tuple->saddr = ip4->saddr;
            	24: (63) *(u32 *)(r10 -148) = r2
            	; return ip4->ihl * 4;
            	25: (71) r2 = *(u8 *)(r6 +14)
            	; return ip4->ihl * 4;
            	26: (67) r2 <<= 2
            	27: (57) r2 &= 60
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	28: (07) r2 += 14
            	29: (7b) *(u64 *)(r10 -200) = r2
            	; switch (tuple.nexthdr) {
            	30: (15) if r1 == 0x1 goto pc+29
            	 R1_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3_w=pkt_end(id=0,off=0,imm=0) R6_w=pkt(id=0,off=0,r=34,imm=0) R8_w=invP0 R9_w=inv4294967162 R10=fp0 fp-144=??0m0000 fp-152_w=mmmmmmmm fp-160=??000000 fp-168_w=00000000 fp-176_w=00000000 fp-184=??000000 fp-192_w=ctx fp-200_w=inv
            	31: (15) if r1 == 0x11 goto pc+3
            	 R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3=pkt_end(id=0,off=0,imm=0) R6=pkt(id=0,off=0,r=34,imm=0) R8=invP0 R9=inv4294967162 R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=??000000 fp-168=00000000 fp-176=00000000 fp-184=??000000 fp-192=ctx fp-200=inv
            	;
            	32: (18) r9 = 0xffffff53
            	; switch (tuple.nexthdr) {
            	34: (55) if r1 != 0x6 goto pc+2957
            	 R1=inv6 R2=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3=pkt_end(id=0,off=0,imm=0) R6=pkt(id=0,off=0,r=34,imm=0) R8=invP0 R9_w=inv4294967123 R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=??000000 fp-168=00000000 fp-176=00000000 fp-184=??000000 fp-192=ctx fp-200=inv
            	; asm volatile("r1 = *(u32 *)(%[ctx] +0)\n\t"
            	35: (79) r3 = *(u64 *)(r10 -192)
            	36: (79) r4 = *(u64 *)(r10 -200)
            	37: (61) r1 = *(u32 *)(r3 +0)
            	38: (61) r2 = *(u32 *)(r3 +4)
            	39: (57) r4 &= 255
            	40: (0f) r1 += r4
            	last_idx 40 first_idx 31
            	regs=10 stack=0 before 39: (57) r4 &= 255
            	regs=10 stack=0 before 38: (61) r2 = *(u32 *)(r3 +4)
            	regs=10 stack=0 before 37: (61) r1 = *(u32 *)(r3 +0)
            	regs=10 stack=0 before 36: (79) r4 = *(u64 *)(r10 -200)
            	regs=0 stack=1000000 before 35: (79) r3 = *(u64 *)(r10 -192)
            	regs=0 stack=1000000 before 34: (55) if r1 != 0x6 goto pc+2957
            	regs=0 stack=1000000 before 32: (18) r9 = 0xffffff53
            	regs=0 stack=1000000 before 31: (15) if r1 == 0x11 goto pc+3
            	 R1_rw=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R3_w=pkt_end(id=0,off=0,imm=0) R6_w=pkt(id=0,off=0,r=34,imm=0) R8_w=invP0 R9_w=inv4294967162 R10=fp0 fp-144=??0m0000 fp-152_w=mmmmmmmm fp-160=??000000 fp-168_w=00000000 fp-176_w=00000000 fp-184=??000000 fp-192_rw=ctx fp-200_rw=invP
            	parent didn't have regs=0 stack=1000000 marks
            	last_idx 30 first_idx 0
            	regs=0 stack=1000000 before 30: (15) if r1 == 0x1 goto pc+29
            	regs=0 stack=1000000 before 29: (7b) *(u64 *)(r10 -200) = r2
            	regs=4 stack=0 before 28: (07) r2 += 14
            	regs=4 stack=0 before 27: (57) r2 &= 60
            	regs=4 stack=0 before 26: (67) r2 <<= 2
            	regs=4 stack=0 before 25: (71) r2 = *(u8 *)(r6 +14)
            	41: (bf) r4 = r1
            	42: (07) r1 += 4
            	43: (2d) if r1 > r2 goto pc+2
            	 R1=pkt(id=1,off=4,r=4,umin_value=2,umax_value=74,var_off=(0x2; 0x7c),s32_max_value=126,u32_max_value=126) R2=pkt_end(id=0,off=0,imm=0) R3=ctx(id=0,off=0,imm=0) R4=pkt(id=1,off=0,r=4,umin_value=2,umax_value=74,var_off=(0x2; 0x7c),s32_max_value=126,u32_max_value=126) R6=pkt(id=0,off=0,r=34,imm=0) R8=invP0 R9=inv4294967123 R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=??000000 fp-168=00000000 fp-176=00000000 fp-184=??000000 fp-192=ctx fp-200=inv
            	44: (b7) r3 = 0
            	45: (05) goto pc+1
            	47: (67) r3 <<= 32
            	48: (bf) r1 = r3
            	49: (77) r1 >>= 32
            	; if (!ret)
            	50: (55) if r1 != 0x0 goto pc+4
            	last_idx 50 first_idx 43
            	regs=2 stack=0 before 49: (77) r1 >>= 32
            	regs=2 stack=0 before 48: (bf) r1 = r3
            	regs=8 stack=0 before 47: (67) r3 <<= 32
            	regs=8 stack=0 before 45: (05) goto pc+1
            	regs=8 stack=0 before 44: (b7) r3 = 0
            	; case  4: __it_mob(d, s, 32);
            	51: (61) r1 = *(u32 *)(r4 +0)
            	52: (bf) r2 = r1
            	53: (77) r2 >>= 16
            	54: (05) goto pc+24
            	;
            	79: (6b) *(u16 *)(r10 -142) = r1
            	80: (6b) *(u16 *)(r10 -144) = r2
            	81: (bf) r2 = r10
            	82: (07) r2 += -152
            	; return map_lookup_elem(map, tuple);
            	83: (18) r1 = 0xffff9ca4cded2c00
            	85: (85) call bpf_map_lookup_elem#1
            	;
            	86: (bf) r2 = r0
            	87: (07) r2 += 32
            	R2 pointer arithmetic on map_value_or_null prohibited, null-check it first
            	processed 60 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 3
    --- FAIL: TestBPF/xdp_nodeport_lb4_test.o (0.03s)
        bpf_test.go:150: verifier error: load program: permission denied:
            	; int tail_handle_snat_fwd_ipv4(struct __ctx_buff *ctx)
            	0: (bf) r6 = r1
            	1: (b7) r7 = 2
            	; return (void *)(unsigned long)ctx->data_end;
            	2: (61) r1 = *(u32 *)(r6 +4)
            	; return (void *)(unsigned long)ctx->data;
            	3: (61) r9 = *(u32 *)(r6 +0)
            	; if (data + tot_len > data_end)
            	4: (bf) r2 = r9
            	5: (07) r2 += 34
            	; if (data + tot_len > data_end)
            	6: (2d) if r2 > r1 goto pc+579
            	 R1_w=pkt_end(id=0,off=0,imm=0) R2_w=pkt(id=0,off=34,r=34,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv2 R9_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0
            	; ip4->saddr == IPV4_DIRECT_ROUTING) {
            	7: (61) r1 = *(u32 *)(r9 +26)
            	8: (7b) *(u64 *)(r10 -160) = r6
            	; if (DIRECT_ROUTING_DEV_IFINDEX == NATIVE_DEV_IFINDEX &&
            	9: (15) if r1 == 0x0 goto pc+315
            	
            	from 9 to 325: R1_w=inv0 R2_w=pkt(id=0,off=34,r=34,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv2 R9_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-160_w=ctx
            	;
            	325: (bf) r8 = r9
            	326: (07) r8 += 14
            	327: (b7) r2 = 0
            	; struct ipv4_ct_tuple tuple = {};
            	328: (63) *(u32 *)(r10 -144) = r2
            	last_idx 328 first_idx 0
            	regs=4 stack=0 before 327: (b7) r2 = 0
            	; tuple->nexthdr = ip4->protocol;
            	329: (71) r1 = *(u8 *)(r9 +23)
            	; tuple->nexthdr = ip4->protocol;
            	330: (73) *(u8 *)(r10 -140) = r1
            	; tuple->daddr = ip4->daddr;
            	331: (61) r3 = *(u32 *)(r9 +30)
            	; tuple->daddr = ip4->daddr;
            	332: (63) *(u32 *)(r10 -152) = r3
            	; tuple->saddr = ip4->saddr;
            	333: (61) r3 = *(u32 *)(r9 +26)
            	; tuple->flags = dir;
            	334: (73) *(u8 *)(r10 -139) = r2
            	; tuple->saddr = ip4->saddr;
            	335: (63) *(u32 *)(r10 -148) = r3
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	336: (1f) r8 -= r9
            	; return ip4->ihl * 4;
            	337: (71) r2 = *(u8 *)(r9 +14)
            	; return ip4->ihl * 4;
            	338: (67) r2 <<= 2
            	339: (57) r2 &= 60
            	; off = ((void *)ip4 - data) + ipv4_hdrlen(ip4);
            	340: (0f) r8 += r2
            	; switch (tuple.nexthdr) {
            	341: (15) if r1 == 0x1 goto pc+42
            	 R1_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv2 R8_w=inv(id=0) R9_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160_w=ctx
            	342: (15) if r1 == 0x11 goto pc+1
            	 R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv2 R8=inv(id=0) R9=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=ctx
            	343: (55) if r1 != 0x6 goto pc+242
            	 R1=inv6 R2=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6=ctx(id=0,off=0,imm=0) R7=inv2 R8=inv(id=0) R9=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=ctx
            	344: (bf) r5 = r9
            	; asm volatile("r1 = *(u32 *)(%[ctx] +0)\n\t"
            	345: (61) r1 = *(u32 *)(r6 +0)
            	346: (61) r2 = *(u32 *)(r6 +4)
            	347: (57) r8 &= 255
            	348: (0f) r1 += r8
            	last_idx 348 first_idx 342
            	regs=100 stack=0 before 347: (57) r8 &= 255
            	regs=100 stack=0 before 346: (61) r2 = *(u32 *)(r6 +4)
            	regs=100 stack=0 before 345: (61) r1 = *(u32 *)(r6 +0)
            	regs=100 stack=0 before 344: (bf) r5 = r9
            	regs=100 stack=0 before 343: (55) if r1 != 0x6 goto pc+242
            	regs=100 stack=0 before 342: (15) if r1 == 0x11 goto pc+1
            	 R1_rw=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2_w=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R3_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6_rw=ctx(id=0,off=0,imm=0) R7_w=inv2 R8_rw=invP(id=0) R9_rw=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160_w=ctx
            	parent didn't have regs=100 stack=0 marks
            	last_idx 341 first_idx 0
            	regs=100 stack=0 before 341: (15) if r1 == 0x1 goto pc+42
            	regs=100 stack=0 before 340: (0f) r8 += r2
            	regs=104 stack=0 before 339: (57) r2 &= 60
            	regs=104 stack=0 before 338: (67) r2 <<= 2
            	regs=104 stack=0 before 337: (71) r2 = *(u8 *)(r9 +14)
            	regs=100 stack=0 before 336: (1f) r8 -= r9
            	regs=300 stack=0 before 335: (63) *(u32 *)(r10 -148) = r3
            	regs=300 stack=0 before 334: (73) *(u8 *)(r10 -139) = r2
            	regs=300 stack=0 before 333: (61) r3 = *(u32 *)(r9 +26)
            	regs=300 stack=0 before 332: (63) *(u32 *)(r10 -152) = r3
            	regs=300 stack=0 before 331: (61) r3 = *(u32 *)(r9 +30)
            	regs=300 stack=0 before 330: (73) *(u8 *)(r10 -140) = r1
            	regs=300 stack=0 before 329: (71) r1 = *(u8 *)(r9 +23)
            	regs=300 stack=0 before 328: (63) *(u32 *)(r10 -144) = r2
            	regs=300 stack=0 before 327: (b7) r2 = 0
            	regs=300 stack=0 before 326: (07) r8 += 14
            	regs=300 stack=0 before 325: (bf) r8 = r9
            	regs=200 stack=0 before 9: (15) if r1 == 0x0 goto pc+315
            	regs=200 stack=0 before 8: (7b) *(u64 *)(r10 -160) = r6
            	regs=200 stack=0 before 7: (61) r1 = *(u32 *)(r9 +26)
            	regs=200 stack=0 before 6: (2d) if r2 > r1 goto pc+579
            	regs=200 stack=0 before 5: (07) r2 += 34
            	regs=200 stack=0 before 4: (bf) r2 = r9
            	regs=200 stack=0 before 3: (61) r9 = *(u32 *)(r6 +0)
            	349: (bf) r4 = r1
            	350: (07) r1 += 4
            	351: (2d) if r1 > r2 goto pc+2
            	 R1=pkt(id=45,off=4,r=4,umax_value=255,var_off=(0x0; 0xff)) R2=pkt_end(id=0,off=0,imm=0) R3=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R4=pkt(id=45,off=0,r=4,umax_value=255,var_off=(0x0; 0xff)) R5=pkt(id=0,off=0,r=34,imm=0) R6=ctx(id=0,off=0,imm=0) R7=inv2 R8=invP(id=0,umax_value=255,var_off=(0x0; 0xff)) R9=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0m0000 fp-152=mmmmmmmm fp-160=ctx
            	352: (b7) r3 = 0
            	353: (05) goto pc+1
            	355: (67) r3 <<= 32
            	356: (bf) r1 = r3
            	357: (77) r1 >>= 32
            	; if (!ret)
            	358: (55) if r1 != 0x0 goto pc+21
            	last_idx 358 first_idx 351
            	regs=2 stack=0 before 357: (77) r1 >>= 32
            	regs=2 stack=0 before 356: (bf) r1 = r3
            	regs=8 stack=0 before 355: (67) r3 <<= 32
            	regs=8 stack=0 before 353: (05) goto pc+1
            	regs=8 stack=0 before 352: (b7) r3 = 0
            	; case  4: __it_mob(d, s, 32);
            	359: (61) r1 = *(u32 *)(r4 +0)
            	360: (bf) r2 = r1
            	361: (77) r2 >>= 16
            	362: (05) goto pc+40
            	;
            	403: (6b) *(u16 *)(r10 -144) = r2
            	404: (6b) *(u16 *)(r10 -142) = r1
            	405: (57) r1 &= 128
            	406: (79) r6 = *(u64 *)(r10 -160)
            	407: (bf) r9 = r5
            	; return (!target->from_local_endpoint && !target->src_from_world &&
            	408: (15) if r1 == 0x0 goto pc+177
            	 R1_w=inv(id=0,umax_value=128,var_off=(0x0; 0x80)) R2=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R3=inv0 R4=pkt(id=45,off=0,r=4,umax_value=255,var_off=(0x0; 0xff)) R5=pkt(id=0,off=0,r=34,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7=inv2 R8=invP(id=0,umax_value=255,var_off=(0x0; 0xff)) R9_w=pkt(id=0,off=0,r=34,imm=0) R10=fp0 fp-144=??0mmmmm fp-152=mmmmmmmm fp-160=ctx
            	409: (bf) r2 = r10
            	;
            	410: (07) r2 += -152
            	; return map_lookup_elem(map, tuple);
            	411: (18) r1 = 0xffff9ca4cd2d8c00
            	413: (85) call bpf_map_lookup_elem#1
            	414: (bf) r7 = r0
            	;
            	415: (bf) r2 = r7
            	416: (07) r2 += 32
            	R2 pointer arithmetic on map_value_or_null prohibited, null-check it first
            	processed 575 insns (limit 1000000) max_states_per_insn 1 total_states 45 peak_states 45 mark_read 24
FAIL
FAIL	github.com/cilium/cilium/test/bpf_tests	0.935s
FAIL
make: *** [Makefile:92: run_bpf_tests] Error 1
make: Leaving directory '/home/wanlindu/upstream-cilium/cilium/test'

@anfernee
@wanlin31 wanlin31 added area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! labels Feb 27, 2023
@sayboras
Copy link
Member

👋 can you give it a try with clang 10 ?

@nickolaev
Copy link
Contributor

nickolaev commented Feb 28, 2023 •
edited

Hello,
so the whole datapath and the relevant unit tests are being tested and verified with llvm-10 toolchain. To make sure you have these on Ubuntu 20.04 and 22.04, please add this repository:

$ cat /etc/apt/sources.list.d/llvm.list
deb http://apt.llvm.org/focal/ llvm-toolchain-focal-10 main
deb-src http://apt.llvm.org/focal/ llvm-toolchain-focal-10 main

Note that there is no llvm-10 for 22.04, but you can safely use the 20.04 one (focal).

Installing it is as simple as sudo apt install clang-10 llvm-10. Then make sure the testing folder is compiled with the right toolchain:

make -C bpf/tests clean
CLANG=clang-10 LLC=llc-10 make -C test/ run_bpf_tests

This is relevant also for: #24052 , #24051, #24050 , #24049

@sayboras sayboras added sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. area/build Anything to do with the build, more general then area/CI labels Feb 28, 2023
This was referenced Feb 28, 2023
Closed
Closed
Closed
Closed
@pchaigno pchaigno removed the ci/flake This is a known failure that occurs in the tree. Please investigate me! label Feb 28, 2023
@wanlin31
Copy link
Contributor Author

I would try to install clang-10 and llvm-10 for Ubuntu22.04.
But for 20.04.4 + 5.13, 20.04.3 + 5.11, 20.04.2 + 5.8, 20.04+ 5.4 they are all clang-10 and llvm-10

20.04.4 + 5.13

wanlindu@wanlindu-cilium-ebpf-testvm-image-ubuntu20044-513:~$ uname -a
Linux wanlindu-cilium-ebpf-testvm-image-ubuntu20044-513 5.13.0-52-generic #59~20.04.1-Ubuntu SMP Thu Jun 16 21:21:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
wanlindu@wanlindu-cilium-ebpf-testvm-image-ubuntu20044-513:~$ clang --version
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
wanlindu@wanlindu-cilium-ebpf-testvm-image-ubuntu20044-513:~$ llc --version
LLVM (http://llvm.org/):
  LLVM version 10.0.0

  Optimized build.
  Default target: x86_64-pc-linux-gnu
  Host CPU: cascadelake

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    avr        - Atmel AVR Microcontroller
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    hexagon    - Hexagon
    lanai      - Lanai
    mips       - MIPS (32-bit big endian)
    mips64     - MIPS (64-bit big endian)
    mips64el   - MIPS (64-bit little endian)
    mipsel     - MIPS (32-bit little endian)
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    riscv32    - 32-bit RISC-V
    riscv64    - 64-bit RISC-V
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore

20.04.3 + 5.11,

uname -a
Linux wanlindu-cilium-ebpf-testvm-image-ubuntu20043-511 5.11.0-1017-gcp #19~20.04.1-Ubuntu SMP Thu Aug 12 05:25:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
wanlindu@wanlindu-cilium-ebpf-testvm-image-ubuntu20043-511:~$ clang --version
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
wanlindu@wanlindu-cilium-ebpf-testvm-image-ubuntu20043-511:~$ llc --version
LLVM (http://llvm.org/):
  LLVM version 10.0.0

  Optimized build.
  Default target: x86_64-pc-linux-gnu
  Host CPU: cascadelake

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    avr        - Atmel AVR Microcontroller
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    hexagon    - Hexagon
    lanai      - Lanai
    mips       - MIPS (32-bit big endian)
    mips64     - MIPS (64-bit big endian)
    mips64el   - MIPS (64-bit little endian)
    mipsel     - MIPS (32-bit little endian)
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    riscv32    - 32-bit RISC-V
    riscv64    - 64-bit RISC-V
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore

20.04.2 + 5.8

wanlindu@wanlindu-cilium-ebpf-testvm-image-ubuntu20042-58:~$ uname -a
Linux wanlindu-cilium-ebpf-testvm-image-ubuntu20042-58 5.8.0-63-generic #71~20.04.1-Ubuntu SMP Thu Jul 15 17:46:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
wanlindu@wanlindu-cilium-ebpf-testvm-image-ubuntu20042-58:~$ clang --version
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
wanlindu@wanlindu-cilium-ebpf-testvm-image-ubuntu20042-58:~$ llc --version
LLVM (http://llvm.org/):
  LLVM version 10.0.0

  Optimized build.
  Default target: x86_64-pc-linux-gnu
  Host CPU: cascadelake

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    avr        - Atmel AVR Microcontroller
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    hexagon    - Hexagon
    lanai      - Lanai
    mips       - MIPS (32-bit big endian)
    mips64     - MIPS (64-bit big endian)
    mips64el   - MIPS (64-bit little endian)
    mipsel     - MIPS (32-bit little endian)
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    riscv32    - 32-bit RISC-V
    riscv64    - 64-bit RISC-V
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore

20.04+ 5.4

wanlindu@wanlindu-cilium-ebpf-testvm-ubuntu20040-base-image:~$ uname -a
Linux wanlindu-cilium-ebpf-testvm-ubuntu20040-base-image 5.4.0-84-generic #94-Ubuntu SMP Thu Aug 26 20:27:37 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
wanlindu@wanlindu-cilium-ebpf-testvm-ubuntu20040-base-image:~$ clang --version
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
wanlindu@wanlindu-cilium-ebpf-testvm-ubuntu20040-base-image:~$ llc --version
LLVM (http://llvm.org/):
  LLVM version 10.0.0

  Optimized build.
  Default target: x86_64-pc-linux-gnu
  Host CPU: haswell

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    avr        - Atmel AVR Microcontroller
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    hexagon    - Hexagon
    lanai      - Lanai
    mips       - MIPS (32-bit big endian)
    mips64     - MIPS (64-bit big endian)
    mips64el   - MIPS (64-bit little endian)
    mipsel     - MIPS (32-bit little endian)
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    riscv32    - 32-bit RISC-V
    riscv64    - 64-bit RISC-V
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore

@wanlin31
Copy link
Contributor Author

I have tried 22.04.1 kernel 5.15 with clang-10 llvm-10, worked out okay, thanks for the instruction. For the rest of the OS, they were already using clang-10 and llvm-10, I cleaned up and re-run them with CLANG=clang-10 LLC=llc-10, still not failing.

sudo add-apt-repository -y "deb http://apt.llvm.org/focal/ llvm-toolchain-focal-10 main"

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys <PUBLIC_KEY>

sudo apt install clang-10 llvm-10

CLANG=clang-10 LLC=llc-10 make -C test/ run_bpf_tests
make: Entering directory '/home/wanlindu/upstream_cilium/cilium/test'
make -C ../bpf/tests all
make[1]: Entering directory '/home/wanlindu/upstream_cilium/cilium/bpf/tests'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/wanlindu/upstream_cilium/cilium/bpf/tests'
go test ./bpf_tests -exec sudo -bpf-test-path /home/wanlindu/upstream_cilium/cilium/bpf/tests
ok  	github.com/cilium/cilium/test/bpf_tests	1.603s
make: Leaving directory '/home/wanlindu/upstream_cilium/cilium/test'

@wanlin31
Copy link
Contributor Author

Now when enable to report option COVER=1, we see error below, this is consistent with all OS tested when enable COVER=1:

CLANG=clang-10 LLC=llc-10 make -C test/ run_bpf_tests COVER=1
make: Entering directory '/home/wanlindu/upstream_cilium/cilium/test'
make -C ../bpf/tests all
make[1]: Entering directory '/home/wanlindu/upstream_cilium/cilium/bpf/tests'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/wanlindu/upstream_cilium/cilium/bpf/tests'
go test ./bpf_tests -exec sudo -bpf-test-path /home/wanlindu/upstream_cilium/cilium/bpf/tests -coverage-report /home/wanlindu/upstream_cilium/cilium/bpf-coverage.html -coverage-format html
--- FAIL: TestBPF (18.96s)
    --- FAIL: TestBPF/session_affinity_test.o (11.68s)
panic: runtime error: index out of range [2451] with length 2451 [recovered]
	panic: runtime error: index out of range [2451] with length 2451

goroutine 130 [running]:
testing.tRunner.func1.2({0x1c00500, 0xc0075b6498})
	/usr/lib/go-1.20/src/testing/testing.go:1526 +0x24e
testing.tRunner.func1()
	/usr/lib/go-1.20/src/testing/testing.go:1529 +0x39f
panic({0x1c00500, 0xc0075b6498})
	/usr/lib/go-1.20/src/runtime/panic.go:884 +0x213
github.com/cilium/coverbee.InstrumentCollection(0xc000206510, {0x0, 0x0})
	/home/wanlindu/upstream_cilium/cilium/vendor/github.com/cilium/coverbee/instrumentation.go:277 +0x25ff
github.com/cilium/coverbee.InstrumentAndLoadCollection(0x4fda00?, {{{0x0, 0x0}, {0x0, 0x0, 0x0}}, {0x0, 0x4000000, 0x0, 0x0}, ...}, ...)
	/home/wanlindu/upstream_cilium/cilium/vendor/github.com/cilium/coverbee/instrumentation.go:27 +0x46
github.com/cilium/cilium/test/bpf_tests.loadAndRunSpec(0xc000982340, {0x2012350?, 0xc0007a7d40?}, {0x0, 0x0})
	/home/wanlindu/upstream_cilium/cilium/test/bpf_tests/bpf_test.go:137 +0x21a
github.com/cilium/cilium/test/bpf_tests.TestBPF.func1(0xc000883860?)
	/home/wanlindu/upstream_cilium/cilium/test/bpf_tests/bpf_test.go:90 +0x3c
testing.tRunner(0xc000982340, 0xc0044fed20)
	/usr/lib/go-1.20/src/testing/testing.go:1576 +0x10b
created by testing.(*T).Run
	/usr/lib/go-1.20/src/testing/testing.go:1629 +0x3ea
FAIL	github.com/cilium/cilium/test/bpf_tests	19.012s
FAIL
make: *** [Makefile:92: run_bpf_tests] Error 1

@sayboras
Copy link
Member

sayboras commented Mar 1, 2023

I can replicate the above issue in my local when passed COVER=1 as well.

cc @dylandreimerink if you have any idea.

make -C bpf/tests clean
CLANG=clang-10 LLC=llc-10 make -C test/ run_bpf_tests COVER=1

@dylandreimerink dylandreimerink mentioned this issue Mar 1, 2023
Merged
@dylandreimerink
Copy link
Member

Thanks for the report! I was also able to reproduce the issue. Turns out that clang is not smart enough to eliminate all dead code, and CoverBee didn't handle that to well. Should be fixed after #24094 merges

dylandreimerink added a commit to dylandreimerink/cilium that referenced this issue Mar 1, 2023
@dylandreimerink
vendor,test: Updated CoverBee to v0.3.1 to resolve panic during testing
f339933 
We were getting panics in the CoverBee code when coverage testing
certain test programs. It turns out that this was due to "dead code"
which the verifier would never evaluate, which CoverBee did not expect
to be possible.

This version of CoverBee fixes the issue without additional changes.

Fixes: cilium#24053
Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
sayboras pushed a commit that referenced this issue Mar 1, 2023
@dylandreimerink @sayboras
vendor,test: Updated CoverBee to v0.3.1 to resolve panic during testing
accd342 
We were getting panics in the CoverBee code when coverage testing
certain test programs. It turns out that this was due to "dead code"
which the verifier would never evaluate, which CoverBee did not expect
to be possible.

This version of CoverBee fixes the issue without additional changes.

Fixes: #24053
Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/build Anything to do with the build, more general then area/CI area/CI Continuous Integration testing issue or flake sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

vendor,test: Updated CoverBee to v0.3.1 to resolve panic during testing dylandreimerink/cilium
5 participants
@pchaigno @dylandreimerink @nickolaev @sayboras @wanlin31