eBPF kenel panic - net/core/skbuff.c:4082

[vojtechDB]

In our cluster we use MetalLB for exposing L2 addresses (not BGP mode). If we generate high load for any IP announced by MetalLB, the node where the IP is announce from crash due to kernel panic

Expected Behavior

Avoid any crash of the node

Current Behavior

vmcore-dmesg:

[ 1539.094593] ------------[ cut here ]------------
[ 1539.094645] kernel BUG at net/core/skbuff.c:4082!
[ 1539.094710] invalid opcode: 0000 [#1] SMP PTI
[ 1539.094757] CPU: 1 PID: 0 Comm: swapper/1 Kdump: loaded Not tainted 4.18.0-372.26.1.el8_6.x86_64 #1
[ 1539.094821] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 1539.094888] RIP: 0010:skb_segment+0xbda/0xe20
[ 1539.094942] Code: 89 4c 24 50 44 89 54 24 60 e8 a2 aa ff ff 44 8b 54 24 60 48 8b 4c 24 50 85 c0 44 8b 44 24 4c 0f 84 f0 fb ff ff e9 0e fd ff ff <0f> 0b 29 c1 89 ce 09 ca e9 5d ff ff ff 0f 0b 45 8b 87 84 00 00 00
[ 1539.095075] RSP: 0018:ffffa5dac1d70830 EFLAGS: 00010246
[ 1539.095126] RAX: ffff8ff98f36cac0 RBX: ffff8ff9465d2900 RCX: 0000000000000000
[ 1539.095185] RDX: 000000000000bca8 RSI: ffff8ff96762d500 RDI: 000000000000bb02
[ 1539.095244] RBP: ffffa5dac1d70908 R08: 000000000000bca8 R09: 6f04b4400a080101
[ 1539.095301] R10: 000000000000bcba R11: f16f8b406f04b440 R12: 000000000000bca8
[ 1539.095358] R13: ffff8ffa21e7e100 R14: 0000000000000010 R15: ffff8ff94ee1a700
[ 1539.095428] FS:  0000000000000000(0000) GS:ffff8ffc6de40000(0000) knlGS:0000000000000000
[ 1539.095453] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1539.095473] CR2: 00007f91940fd018 CR3: 00000003e7010002 CR4: 00000000001706e0
[ 1539.095509] Call Trace:
[ 1539.095520]  <IRQ>
[ 1539.095532]  ? skb_send_sock_locked+0x20/0x20
[ 1539.095548]  ? reqsk_fastopen_remove+0x150/0x150
[ 1539.095562]  tcp_gso_segment+0xf1/0x4c0
[ 1539.095575]  inet_gso_segment+0x161/0x3e0
[ 1539.095589]  skb_mac_gso_segment+0xad/0x110
[ 1539.095603]  skb_udp_tunnel_segment+0x2a9/0x5d0
[ 1539.095618]  inet_gso_segment+0x161/0x3e0
[ 1539.095629]  skb_mac_gso_segment+0xad/0x110
[ 1539.095642]  __skb_gso_segment+0xbd/0x170
[ 1539.095655]  validate_xmit_skb+0x14e/0x310
[ 1539.096162]  validate_xmit_skb_list+0x46/0x70
[ 1539.096575]  sch_direct_xmit+0x183/0x360
[ 1539.096969]  __dev_queue_xmit+0x893/0xa40
[ 1539.097353]  __bpf_redirect+0x17c/0x2e0
[ 1539.097737]  skb_do_redirect+0x20b/0x700
[ 1539.098119]  ? cls_bpf_classify+0x10b/0x320 [cls_bpf]
[ 1539.098493]  ? __ip_finish_output+0x1c0/0x1c0
[ 1539.098858]  __netif_receive_skb_core+0x9d4/0xcb0
[ 1539.099218]  ? tcp_gro_receive+0x1d5/0x2d0
[ 1539.099586]  ? ktime_get_with_offset+0x54/0xc0
[ 1539.099953]  netif_receive_skb_internal+0x3d/0xb0
[ 1539.100319]  ? inet_gro_complete+0xa7/0xe0
[ 1539.100688]  ? napi_gro_complete.constprop.170+0xb9/0xd0
[ 1539.101057]  dev_gro_receive+0x2c7/0x760
[ 1539.101432]  napi_gro_receive+0x56/0x150
[ 1539.101806]  vmxnet3_rq_rx_complete+0x8b7/0xdf0 [vmxnet3]
[ 1539.102203]  vmxnet3_poll_rx_only+0x31/0x90 [vmxnet3]
[ 1539.102586]  __napi_poll+0x2d/0x130
[ 1539.102961]  net_rx_action+0x253/0x320
[ 1539.103329]  __do_softirq+0xd7/0x2c4
[ 1539.103704]  irq_exit_rcu+0xcb/0xd0
[ 1539.104077]  irq_exit+0xa/0x10
[ 1539.104457]  do_IRQ+0x7f/0xd0
[ 1539.104807]  common_interrupt+0xf/0xf
[ 1539.105168]  </IRQ>
[ 1539.105485] RIP: 0010:native_safe_halt+0xe/0x10
[ 1539.105796] Code: ff ff 7f c3 65 48 8b 04 25 40 5c 01 00 f0 80 48 02 20 48 8b 00 a8 08 75 c4 eb 80 90 e9 07 00 00 00 0f 00 2d 96 30 46 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 86 30 46 00 f4 c3 90 90 0f 1f 44 00
[ 1539.106455] RSP: 0018:ffffa5dac1943e30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd3
[ 1539.106798] RAX: 0000000080004000 RBX: 0000000000000001 RCX: ffff8ffc6de6bc00
[ 1539.107150] RDX: 0000000000000001 RSI: ffffffff97ac3660 RDI: ffff8ff94299c464
[ 1539.107481] RBP: ffff8ff94299c464 R08: 0000000000000001 R09: ffff8ff94299c400
[ 1539.107821] R10: 000000000000605c R11: ffff8ffc6de69b44 R12: 0000000000000001
[ 1539.108165] R13: ffffffff97ac3660 R14: 0000000000000001 R15: 0000000000000001
[ 1539.108494]  acpi_idle_do_entry+0x46/0x50
[ 1539.108829]  acpi_idle_enter+0x5a/0xd0
[ 1539.109172]  cpuidle_enter_state+0x86/0x3d0
[ 1539.109497]  cpuidle_enter+0x2c/0x40
[ 1539.109823]  do_idle+0x264/0x2c0
[ 1539.110147]  cpu_startup_entry+0x6f/0x80
[ 1539.110461]  start_secondary+0x1a6/0x1e0
[ 1539.110785]  secondary_startup_64_no_verify+0xc2/0xcb

Possible Solution

With disabled GRO for main interface like a workaround, the issue is not reproduceable
ethtool --offload eth0 generic-receive-offload off

Context

Flatcar and Ubuntu kernels are facing with the same issue: flatcar/Flatcar#378

Brief analysis by RH:

crashed due to a bad pointer reference and kernel triggered a BUG condition due to this

+++
crashed at:
4082                     BUG_ON(!list_skb->head_frag);
+++


The network stack was not expecting a NULL value at  skb->head_frag.  But it got NULL and caused panic.


From the vmcore, the calculated list_skb->head_frag is 0x0 which is NULL.

+++
crash> struct sk_buff ffff8ff9c3374a40 |grep head_frag
  head_frag = 0x0,
+++

and we crashed.


Below is the backtrace of the crashed program:
+++
 #6 [ffffa5dac1d70780] invalid_op at ffffffff96800d64
    [exception RIP: skb_segment+0xbda]
    RIP: ffffffff965cb83a  RSP: ffffa5dac1d70830  RFLAGS: [00010246](https://access.redhat.com/support/cases/#/case/00010246)
    RAX: ffff8ff98f36cac0  RBX: ffff8ff9465d2900  RCX: 0000000000000000
    RDX: 000000000000bca8  RSI: ffff8ff96762d500  RDI: 000000000000bb02
    RBP: ffffa5dac1d70908   R8: 000000000000bca8   R9: 6f04b4400a080101
    R10: 000000000000bcba  R11: f16f8b406f04b440  R12: 000000000000bca8
    R13: ffff8ffa21e7e100  R14: 0000000000000010  R15: ffff8ff94ee1a700
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffffa5dac1d70910] tcp_gso_segment at ffffffff966a9a21
 #8 [ffffa5dac1d70960] inet_gso_segment at ffffffff966beb31
 #9 [ffffa5dac1d709b0] skb_mac_gso_segment at ffffffff965ded0d
#10 [ffffa5dac1d709d8] skb_udp_tunnel_segment at ffffffff966b35b9
#11 [ffffa5dac1d70a38] inet_gso_segment at ffffffff966beb31
#12 [ffffa5dac1d70a88] skb_mac_gso_segment at ffffffff965ded0d
#13 [ffffa5dac1d70ab0] __skb_gso_segment at ffffffff965dee2d
#14 [ffffa5dac1d70ad8] validate_xmit_skb at ffffffff965df2ae
#15 [ffffa5dac1d70b10] validate_xmit_skb_list at ffffffff965df4b6
#16 [ffffa5dac1d70b40] sch_direct_xmit at ffffffff9663df13
#17 [ffffa5dac1d70b88] __dev_queue_xmit at ffffffff965e0233
#18 [ffffa5dac1d70c08] __bpf_redirect at ffffffff96605c9c
#19 [ffffa5dac1d70c28] skb_do_redirect at ffffffff96608a8b
#20 [ffffa5dac1d70cd8] __netif_receive_skb_core at ffffffff965e1484
+++

Here we can see that network stack receives the packet and then calls 'skb_do_redirect'  via '__bpf_redirect' and does a packet redirection.

Your Environment

• Calico version

/usr/local/bin/calicoctl version
Client Version:    v3.23.3
Git commit:        3a3559be1
Unable to detect installed Calico version

• Orchestrator version (e.g. kubernetes, mesos, rkt):

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4",....}

• Operating System and version:

Red Hat Enterprise Linux release 8.6 (Ootpa)
4.18.0-372.26.1.el8_6.x86_64

[seh]

Thank you filing this issue. I noticed that to work around this problem, you disabled GRO, but not GSO. @igcherkaev had recommended disabling both in flatcar/Flatcar#378 (comment), and we had found that disabling just one of the two insufficient, but that was with an older kernel version than we've been using over the last couple of weeks. Perhaps we should try enabling GSO again.

[tomastigera]

https://lore.kernel.org/netdev/CAM=1FV1Nt9a5-d7LneS=-o0S2=FnDeCxLPXrB==XYJvjJ8v=+A@mail.gmail.com/T/#u

[tomastigera]

The ebpf code triggers a bug in the kernel. ebpf actions should not trigger that, it should be safe. However, that does not mean that there is not a bug in calico code. There likely is and we are looking into it.

[seh]

Perhaps we should try enabling GSO again.

So far, my testing confirm's @vojtechDB's observation:

GRO	GSO	Kernel Bug
🚫	✅	🆗
✅	🚫	💥

[tomastigera]

@seh @vojtechDB so far it seems to be a genuine kernel bug

[jbenc]

Could you try this kernel patch?
https://people.redhat.com/~jbenc/net-gso-fix-panic-on-frag_list-with-mixed-head-alloc.patch

[vojtechDB]

thank you @jbenc! I confirm I'm not able to reproduce kernel panic anymore. With a kernel without the patch I can reproduce it within a minute

@tomastigera you were right

Thank you guys!

[tomastigera]

Thank you @jbenc for providing the fix.

[vojtechDB]

For a reference. Fixed in following RedHat errata https://access.redhat.com/errata/RHSA-2023:2951