Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apply linux-5.10.35.patch #1

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open

apply linux-5.10.35.patch #1

wants to merge 27 commits into from

Conversation

kingluo
Copy link

@kingluo kingluo commented May 21, 2024

@kingluo
Copy link
Author

kingluo commented May 21, 2024

0. system freeze

When the system is idle for a long time, such as when the qemu host sleeping overnight, it may crash.

Useful links:
https://access.redhat.com/solutions/5534961
https://www.kernel.org/doc/Documentation/RCU/stallwarn.txt
https://access.redhat.com/solutions/3901121

  • log1
[43272.637898] rcu: INFO: rcu_preempt self-detected stall on CPU
[43272.637935] rcu:     0-...!: (1 ticks this GP) idle=25b4/1/0x4000000000000000 softirq=3748/3748 fqs=0
[43272.637953] rcu:     (t=8796097 jiffies g=4213 q=6 ncpus=4)
[43272.637965] rcu: rcu_preempt kthread timer wakeup didn't happen for 8796096 jiffies! g4213 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[43272.637983] rcu:     Possible timer handling issue on cpu=0 timer-softirq=3263
[43272.637997] rcu: rcu_preempt kthread starved for 8796097 jiffies! g4213 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
[43272.638014] rcu:     Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[43272.638037] rcu: RCU grace-period kthread stack dump:
[43272.638050] task:rcu_preempt     state:I stack:0     pid:17    tgid:17    ppid:2      flags:0x00004000
[43272.638054] Call Trace:
[43272.638057]  <TASK>
[43272.638062]  __schedule+0x375/0xb30
[43272.638072]  ? __pfx_rcu_gp_kthread+0x10/0x10
[43272.638078]  schedule+0x39/0xe0
[43272.638080]  schedule_timeout+0x8d/0x160
[43272.638083]  ? __pfx_process_timeout+0x10/0x10
[43272.638085]  rcu_gp_fqs_loop+0x115/0x4d0
[43272.638087]  rcu_gp_kthread+0xd8/0x170
[43272.638090]  kthread+0xfb/0x130
[43272.638094]  ? __pfx_kthread+0x10/0x10
[43272.638096]  ret_from_fork+0x40/0x60
[43272.638100]  ? __pfx_kthread+0x10/0x10
[43272.638102]  ret_from_fork_asm+0x1b/0x30
[43272.638106]  </TASK>
[43272.638112] CPU: 0 PID: 571 Comm: systemd-timesyn Kdump: loaded Not tainted 6.8.9+ #45
[43272.638114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[43272.638115] RIP: 0010:e1000_xmit_frame+0xbd4/0xfe0 [e1000]
[43272.638130] Code: c0 74 11 48 8b 43 18 f6 80 d0 00 00 00 03 0f 84 d3 fd ff ff 48 8b 5d c0 41 0f b7 45 2a 41 8b 55 18 48 03 83 48 0e 00 00 89 10 <e9> 90 fa ff ff 44 89 eb 4d 89 fd 4c 8b 7d b8 e9 e2 f9 ff ff 31 f6
[43272.638131] RSP: 0018:ffff9d110097f850 EFLAGS: 00010286
[43272.638133] RAX: ffff9d11009e3818 RBX: ffff8fa7b1d92000 RCX: 00000000000000c5
[43272.638135] RDX: 00000000000000c5 RSI: ffff8fa7a8c4c600 RDI: 0000000000000100
[43272.638136] RBP: ffff9d110097f8e8 R08: 0000000000000000 R09: ffff8fa7b1d92000
[43272.638137] R10: 0000000000000000 R11: 00000000aff2a800 R12: 00000000000000c3
[43272.638137] R13: ffff8fa7a8c4c600 R14: 000000000000005a R15: ffff8fa7a85ab500
[43272.638139] FS:  00007f589e66d180(0000) GS:ffff8fa7bbc00000(0000) knlGS:0000000000000000
[43272.638140] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[43272.638141] CR2: 00007f4a9ad3f000 CR3: 0000000131e1c000 CR4: 0000000000b50ef0
[43272.638144] Call Trace:
[43272.638145]  <IRQ>
[43272.638146]  ? show_regs+0x6e/0x80
[43272.638149]  ? dump_cpu_task+0x6e/0x80
[43272.638152]  ? rcu_dump_cpu_stacks+0xc3/0x100
[43272.638155]  ? rcu_sched_clock_irq+0x675/0x1070
[43272.638156]  ? update_cfs_group+0xab/0xc0
[43272.638158]  ? enqueue_task_fair+0x222/0x3e0
[43272.638160]  ? __cgroup_account_cputime_field+0x3f/0x60
[43272.638164]  ? account_system_index_time+0x95/0xb0
[43272.638167]  ? update_process_times+0x67/0xa0
[43272.638169]  ? __pfx_tick_nohz_highres_handler+0x10/0x10
[43272.638173]  ? tick_sched_handle+0x28/0x70
[43272.638175]  ? __pfx_tick_nohz_highres_handler+0x10/0x10
[43272.638177]  ? tick_nohz_highres_handler+0x7b/0xa0
[43272.638179]  ? __hrtimer_run_queues+0x114/0x2b0
[43272.638181]  ? hrtimer_interrupt+0x105/0x230
[43272.638184]  ? __sysvec_apic_timer_interrupt+0x57/0x150
[43272.638188]  ? sysvec_apic_timer_interrupt+0x7f/0x90
[43272.638190]  </IRQ>
[43272.638190]  <TASK>
[43272.638191]  ? asm_sysvec_apic_timer_interrupt+0x1f/0x30
[43272.638195]  ? e1000_xmit_frame+0xbd4/0xfe0 [e1000]
[43272.638201]  dev_hard_start_xmit+0x92/0x1e0
[43272.638205]  ? validate_xmit_skb_list+0x51/0x80
[43272.638207]  sch_direct_xmit+0xa5/0x3c0
[43272.638211]  __dev_queue_xmit+0xb1b/0xde0
[43272.638213]  ? __neigh_event_send+0xcd/0x440
[43272.638216]  neigh_resolve_output+0x116/0x1c0
[43272.638217]  ip_finish_output2+0x196/0x550
[43272.638222]  __ip_finish_output+0xb7/0x190
[43272.638224]  ip_finish_output+0x2d/0xe0
[43272.638226]  ip_output+0x63/0xf0
[43272.638228]  ? __pfx_ip_finish_output+0x10/0x10
[43272.638230]  ip_send_skb+0x90/0xa0
[43272.638232]  udp_send_skb+0x164/0x3a0
[43272.638234]  udp_sendmsg+0xc21/0xf60
[43272.638236]  ? _get_random_bytes.part.0+0x78/0x190
[43272.638241]  ? __pfx_ip_generic_getfrag+0x10/0x10
[43272.638243]  ? tomoyo_socket_sendmsg_permission+0x94/0xc0
[43272.638247]  ? switched_to_dl+0x42/0x1d0
[43272.638250]  inet_sendmsg+0x71/0x80
[43272.638253]  ? inet_sendmsg+0x71/0x80
[43272.638255]  __sys_sendto+0x207/0x240
[43272.638257]  ? do_epoll_ctl+0x273/0xcc0
[43272.638261]  __x64_sys_sendto+0x28/0x30
[43272.638263]  x64_sys_call+0x2038/0x20c0
[43272.638265]  do_syscall_64+0x54/0x120
[43272.638267]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[43272.638268] RIP: 0033:0x7f589e927a66
[43272.638271] Code: 8f f6 ff 44 8b 4c 24 2c 4c 8b 44 24 20 41 89 c4 44 8b 54 24 28 48 8b 54 24 18 b8 2c 00 00 00 48 8b 74 24 10 8b 7c 24 08 0f 05 <48> 3d 00 f0 ff ff 77
3a 44 89 e7 48 89 44 24 08 e8 e5 8f f6 ff 48
[43272.638272] RSP: 002b:00007ffeb38be760 EFLAGS: 00000293 ORIG_RAX: 000000000000002c
[43272.638274] RAX: ffffffffffffffda RBX: 00007ffeb38be8b0 RCX: 00007f589e927a66
[43272.638275] RDX: 0000000000000030 RSI: 00007ffeb38be7b0 RDI: 0000000000000010
[43272.638276] RBP: 0000557ed8463a30 R08: 0000557ed846c248 R09: 0000000000000010
[43272.638276] R10: 0000000000000040 R11: 0000000000000293 R12: 0000000000000000
[43272.638277] R13: 00007ffeb38be7a4 R14: 0000000000000002 R15: 0000000000000000
[43272.638279]  </TASK>
[43272.638281] Sending NMI from CPU 0 to CPUs 1:
[43272.638289] NMI backtrace for cpu 1
[43272.638292] CPU: 1 PID: 718 Comm: containerd Kdump: loaded Not tainted 6.8.9+ #45
[43272.638295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[43272.638296] RIP: 0033:0x55d52dd9fd04
[43272.638299] Code: 08 0f 1f 44 00 00 e8 1b 87 02 00 8b 44 24 08 e9 92 f7 ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 49 3b 66 10 <76> 67 55 48 89 e5 48
83 ec 30 48 89 44 24 40 e8 68 00 00 00 48 8b
[43272.638301] RSP: 002b:00007f972cfffc00 EFLAGS: 00000202
[43272.638303] RAX: 000000c00004c000 RBX: 0000000000000000 RCX: 000000c0001021a0
[43272.638304] RDX: 000000c000100000 RSI: 000000c0001021a0 RDI: 000000c000100148
[43272.638305] RBP: 00007f972cfffc28 R08: 0000000000000000 R09: 0000000000000000
[43272.638306] R10: 0000000000000000 R11: 0000000000000286 R12: 000000c00004c000
[43272.638307] R13: 000000c000075ee2 R14: 000000c0001021a0 R15: 0000000000000000
[43272.638308] FS:  00007f972d000640 GS:  0000000000000000
[43272.639279] Sending NMI from CPU 0 to CPUs 2:
[43272.639292] NMI backtrace for cpu 2
[43272.639294] CPU: 2 PID: 328 Comm: systemd-journal Kdump: loaded Not tainted 6.8.9+ #45
[43272.639297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[43272.639299] RIP: 0033:0x7fb45a343cca
[43272.639302] Code: f3 0f 1e fa c3 66 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 54 31 c0 ba ff ff ff ff 53 48 83 ec 08 f0 0f b1 15 3e 30 17 00 <83> f8 ff 74 39 41 89 c4 85 c0 75 21 31 c0 bf 27 00 00 00 e8 de f7
[43272.639303] RSP: 002b:00007ffcaa78ba50 EFLAGS: 00000297
[43272.639305] RAX: 0000000000000148 RBX: 0000000000000148 RCX: 0000555bc43b3f00
[43272.639306] RDX: 00000000ffffffff RSI: 0000000000000001 RDI: 0000555bc43b3cc0
[43272.639307] RBP: 0000555bc43b3cc0 R08: 0000000000000001 R09: 0000000000000000
[43272.639308] R10: 00007ffcaa78b9fc R11: 0000000000000213 R12: 0000000000000001
[43272.639309] R13: 00007ffcaa78bac8 R14: 0000000000000003 R15: 0000000000000001
[43272.639310] FS:  00007fb45a02b900 GS:  0000000000000000
[43272.640281] Sending NMI from CPU 0 to CPUs 3:
[43272.640294] NMI backtrace for cpu 3
[43272.640297] CPU: 3 PID: 837 Comm: snapd Kdump: loaded Not tainted 6.8.9+ #45
[43272.640300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[43272.640301] RIP: 0033:0x56027b941de9
[43272.640305] Code: cc 4c 8d 64 24 f0 4d 3b 66 10 0f 86 17 03 00 00 48 81 ec 90 00 00 00 48 89 ac 24 88 00 00 00 48 8d ac 24 88 00 00 00 0f 18 08 <48> 8b 15 00 a5 56 01
84 02 90 48 be 00 00 00 00 00 80 00 00 48 01
[43272.640306] RSP: 002b:00007fc4d8bffc88 EFLAGS: 00000206
[43272.640308] RAX: 000000c00018b4a0 RBX: 000000c000048138 RCX: 000000c00018b4a0
[43272.640309] RDX: 000000c000202b60 RSI: 0000000000000000 RDI: 0000000000000000
[43272.640310] RBP: 00007fc4d8bffd10 R08: 7fffffffffed76b7 R09: 000000c0003b7800
[43272.640311] R10: 000000000000007a R11: 0000000000000079 R12: 00007fc4d8bffd08
[43272.640312] R13: 0000000000000055 R14: 000000c00057eb60 R15: 0000000000000000
[43272.640313] FS:  00007fc4d8c00640 GS:  0000000000000000
[43272.641283] Kernel panic - not syncing: RCU Stall
[43272.641476] CPU: 0 PID: 571 Comm: systemd-timesyn Kdump: loaded Not tainted 6.8.9+ #45
[43272.641498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[43272.641522] Call Trace:
[43272.641532]  <IRQ>
[43272.641542]  dump_stack_lvl+0x2b/0x90
[43272.641564]  dump_stack+0x14/0x20
[43272.641574]  panic+0x339/0x350
[43272.641588]  rcu_sched_clock_irq+0x106d/0x1070
[43272.641603]  ? update_cfs_group+0xab/0xc0
[43272.641618]  ? enqueue_task_fair+0x222/0x3e0
[43272.641632]  ? __cgroup_account_cputime_field+0x3f/0x60
[43272.641649]  ? account_system_index_time+0x95/0xb0
[43272.641662]  update_process_times+0x67/0xa0
[43272.641675]  ? __pfx_tick_nohz_highres_handler+0x10/0x10
[43272.641690]  tick_sched_handle+0x28/0x70
[43272.641702]  ? __pfx_tick_nohz_highres_handler+0x10/0x10
[43272.641715]  tick_nohz_highres_handler+0x7b/0xa0
[43272.641726]  __hrtimer_run_queues+0x114/0x2b0
[43272.641739]  hrtimer_interrupt+0x105/0x230
[43272.641753]  __sysvec_apic_timer_interrupt+0x57/0x150
[43272.641767]  sysvec_apic_timer_interrupt+0x7f/0x90
[43272.641781]  </IRQ>
[43272.641786]  <TASK>
[43272.641792]  asm_sysvec_apic_timer_interrupt+0x1f/0x30
[43272.641810] RIP: 0010:e1000_xmit_frame+0xbd4/0xfe0 [e1000]
[43272.642234] Code: c0 74 11 48 8b 43 18 f6 80 d0 00 00 00 03 0f 84 d3 fd ff ff 48 8b 5d c0 41 0f b7 45 2a 41 8b 55 18 48 03 83 48 0e 00 00 89 10 <e9> 90 fa ff ff 44 89 eb 4d 89 fd 4c 8b 7d b8 e9 e2 f9 ff ff 31 f6
[43272.642879] RSP: 0018:ffff9d110097f850 EFLAGS: 00010286
[43272.643189] RAX: ffff9d11009e3818 RBX: ffff8fa7b1d92000 RCX: 00000000000000c5
[43272.643630] RDX: 00000000000000c5 RSI: ffff8fa7a8c4c600 RDI: 0000000000000100
[43272.644053] RBP: ffff9d110097f8e8 R08: 0000000000000000 R09: ffff8fa7b1d92000
[43272.644312] R10: 0000000000000000 R11: 00000000aff2a800 R12: 00000000000000c3
[43272.644593] R13: ffff8fa7a8c4c600 R14: 000000000000005a R15: ffff8fa7a85ab500
[43272.644890]  dev_hard_start_xmit+0x92/0x1e0
[43272.645156]  ? validate_xmit_skb_list+0x51/0x80
[43272.645430]  sch_direct_xmit+0xa5/0x3c0
[43272.645676]  __dev_queue_xmit+0xb1b/0xde0
[43272.645958]  ? __neigh_event_send+0xcd/0x440
[43272.646248]  neigh_resolve_output+0x116/0x1c0
[43272.646807]  ip_finish_output2+0x196/0x550
[43272.647060]  __ip_finish_output+0xb7/0x190
[43272.647325]  ip_finish_output+0x2d/0xe0
[43272.647553]  ip_output+0x63/0xf0
[43272.647774]  ? __pfx_ip_finish_output+0x10/0x10
[43272.648001]  ip_send_skb+0x90/0xa0
[43272.652927]  udp_send_skb+0x164/0x3a0
[43272.653309]  udp_sendmsg+0xc21/0xf60
[43272.653989]  ? _get_random_bytes.part.0+0x78/0x190
[43272.654485]  ? __pfx_ip_generic_getfrag+0x10/0x10
[43272.654769]  ? tomoyo_socket_sendmsg_permission+0x94/0xc0
[43272.655030]  ? switched_to_dl+0x42/0x1d0
[43272.666353]  inet_sendmsg+0x71/0x80
[43272.666623]  ? inet_sendmsg+0x71/0x80
[43272.666927]  __sys_sendto+0x207/0x240
[43272.667129]  ? do_epoll_ctl+0x273/0xcc0
[43272.669332]  __x64_sys_sendto+0x28/0x30
[43272.670192]  x64_sys_call+0x2038/0x20c0
[43272.670468]  do_syscall_64+0x54/0x120
[43272.670700]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[43272.670929] RIP: 0033:0x7f589e927a66
[43272.671158] Code: 8f f6 ff 44 8b 4c 24 2c 4c 8b 44 24 20 41 89 c4 44 8b 54 24 28 48 8b 54 24 18 b8 2c 00 00 00 48 8b 74 24 10 8b 7c 24 08 0f 05 <48> 3d 00 f0 ff ff 77 3a 44 89 e7 48 89 44 24 08 e8 e5 8f f6 ff 48
[43272.672055] RSP: 002b:00007ffeb38be760 EFLAGS: 00000293 ORIG_RAX: 000000000000002c
[43272.672323] RAX: ffffffffffffffda RBX: 00007ffeb38be8b0 RCX: 00007f589e927a66
[43272.672859] RDX: 0000000000000030 RSI: 00007ffeb38be7b0 RDI: 0000000000000010
[43272.673480] RBP: 0000557ed8463a30 R08: 0000557ed846c248 R09: 0000000000000010
[43272.673736] R10: 0000000000000040 R11: 0000000000000293 R12: 0000000000000000
[43272.673994] R13: 00007ffeb38be7a4 R14: 0000000000000002 R15: 0000000000000000
[43272.674259]  </TASK>
  • log2
[13472.301062] process 'tempesta/tls/t/tgen_ec256' started with executable stack
[23747.512118] systemd[1]: snapd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
[23747.512682] systemd[1]: snapd.service: Failed with result 'watchdog'.
[23747.513709] systemd[1]: snapd.service: Consumed 3.958s CPU time.
[59207.878421] rcu: INFO: rcu_preempt self-detected stall on CPU
[59207.878441] watchdog: BUG: soft lockup - CPU#1 stuck for 33025s! [kworker/1:2:11146]
[59207.878891] rcu:     2-...!: (1 ticks this GP) idle=cffc/0/0x1 softirq=36569/36569 fqs=1
[59207.879287] Modules linked in:
[59207.880023] rcu:     (t=69144 jiffies g=52789 q=494 ncpus=4)
[59207.880192]  tls
[59207.880767] rcu: rcu_preempt kthread timer wakeup didn't happen for 69141 jiffies! g52789 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[59207.881102]  xt_conntrack
[59207.881660] rcu:     Possible timer handling issue on cpu=3 timer-softirq=71090
[59207.882056]  nft_chain_nat
[59207.882572] rcu: rcu_preempt kthread starved for 69142 jiffies! g52789 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=3
[59207.882854]  xt_MASQUERADE
[59207.883223] rcu:     Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[59207.883528]  nf_nat
[59207.883830] rcu: RCU grace-period kthread stack dump:
[59207.884193]  nf_conntrack_netlink
[59207.884454] task:rcu_preempt     state:I
[59207.884770]  nf_conntrack
[59207.885077]  stack:0     pid:17    tgid:17    ppid:2      flags:0x00004000
[59207.885464]  nf_defrag_ipv6
[59207.885754] Call Trace:
[59207.886053]  nf_defrag_ipv4
[59207.886348]  <TASK>
[59207.886638]  xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc overlay intel_rapl_msr intel_rapl_common
[59207.887062]  __schedule+0x375/0xb30
[59207.887545]  crct10dif_pclmul
[59207.887956]  ? __pfx_rcu_gp_kthread+0x10/0x10
[59207.888300]  crc32_pclmul
[59207.888579]  schedule+0x39/0xe0
[59207.888937]  ghash_clmulni_intel
[59207.889211]  schedule_timeout+0x8d/0x160
[59207.889549]  sha512_ssse3
[59207.889815]  ? __pfx_process_timeout+0x10/0x10
[59207.890145]  sha256_ssse3
[59207.890396]  rcu_gp_fqs_loop+0x115/0x4d0
[59207.890724]  rapl
[59207.890971]  rcu_gp_kthread+0xd8/0x170
[59207.890976]  kthread+0xfb/0x130
[59207.891317]  ppdev
[59207.891560]  ? __pfx_kthread+0x10/0x10
[59207.891870]  parport_pc
[59207.892112]  ret_from_fork+0x40/0x60
[59207.892381]  e1000
[59207.892650]  ? __pfx_kthread+0x10/0x10
[59207.892977]  i2c_piix4
[59207.893206]  ret_from_fork_asm+0x1b/0x30
[59207.893538]  floppy
[59207.893744]  </TASK>
[59207.894038]  parport
[59207.894255] rcu: Stack dump where RCU GP kthread last ran:
[59207.894540]  qemu_fw_cfg
[59207.894881] Sending NMI from CPU 2 to CPUs 3:
[59207.895124]  binfmt_misc sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efi_pstore ip_tables x_tables autofs4 raid10 raid456 libcrc32c async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0 bochs drm_vram_helper drm_ttm_helper ttm drm_kms_helper drm input_leds psmouse serio_raw pata_acpi mac_hid aesni_intel crypto_simd cryptd [last unloaded: tempesta_lib(OE)]
[59207.897951] CPU: 1 PID: 11146 Comm: kworker/1:2 Kdump: loaded Tainted: G           OE      6.8.9+ #53
[59207.898261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[59207.898565] Workqueue: events drm_fb_helper_damage_work [drm_kms_helper]
[59207.899609] RIP: 0010:memcpy_toio+0x78/0xc0
[59207.900098] Code: 5d c3 cc cc cc cc 48 85 db 74 f0 41 f6 c4 01 75 47 48 83 fb 01 76 06 41 f6 c4 02 75 27 48 89 d9 4c 89 e7 4c 89 ee 48 c1 e9 02 <f3> a5 f6 c3 02 74 02
66 a5 f6 c3 01 74 01 a4 5b 41 5c 41 5d 5d c3
[59207.900656] RSP: 0018:ffff9aafc2b57d78 EFLAGS: 00010206
[59207.901019] RAX: 0000000000000000 RBX: 0000000000001400 RCX: 0000000000000212
[59207.901336] RDX: 0000000000001400 RSI: ffff9aafc05c57b8 RDI: ffff9aafc0d087b8
[59207.901644] RBP: ffff9aafc2b57d90 R08: ffff8bd60004eeb0 R09: ffff8bd51efbaec0
[59207.901966] R10: 0000000000000007 R11: 0000000000000007 R12: ffff9aafc0d07c00
[59207.902247] R13: ffff9aafc05c4c00 R14: 00000000000000d3 R15: ffff9aafc2b57e20
[59207.902528] FS:  0000000000000000(0000) GS:ffff8bd63bc80000(0000) knlGS:0000000000000000
[59207.902811] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[59207.903100] CR2: 0000556b0db820e0 CR3: 000000010006c000 CR4: 0000000000b50ef0
[59207.903411] Call Trace:
[59207.903724]  <IRQ>
[59207.904031]  ? show_regs+0x6e/0x80
[59207.904459]  ? watchdog_timer_fn+0x200/0x280
[59207.904923]  ? __pfx_watchdog_timer_fn+0x10/0x10
[59207.905208]  ? __hrtimer_run_queues+0x114/0x2b0
[59207.905489]  ? hrtimer_interrupt+0x105/0x230
[59207.905768]  ? __sysvec_apic_timer_interrupt+0x57/0x150
[59207.906161]  ? sysvec_apic_timer_interrupt+0x7f/0x90
[59207.906466]  </IRQ>
[59207.906739]  <TASK>
[59207.907007]  ? asm_sysvec_apic_timer_interrupt+0x1f/0x30
[59207.907382]  ? memcpy_toio+0x78/0xc0
[59207.907674]  ? memcpy_toio+0x25/0xc0
[59207.908006]  drm_fbdev_generic_helper_fb_dirty+0x1c2/0x2d0 [drm_kms_helper]
[59207.908430]  drm_fb_helper_damage_work+0x93/0x160 [drm_kms_helper]
[59207.908883]  process_one_work+0x179/0x350
[59207.909431]  ? __pfx_worker_thread+0x10/0x10
[59207.909808]  worker_thread+0x2f7/0x420
[59207.910149]  ? __pfx_worker_thread+0x10/0x10
[59207.910493]  kthread+0xfb/0x130
[59207.911022]  ? __pfx_kthread+0x10/0x10
[59207.911350]  ret_from_fork+0x40/0x60
[59207.911671]  ? __pfx_kthread+0x10/0x10
[59207.911993]  ret_from_fork_asm+0x1b/0x30
[59207.912405]  </TASK>
[59207.912738] NMI backtrace for cpu 3
[59207.912743] CPU: 3 PID: 13001 Comm: pool-udisksd Kdump: loaded Tainted: G           OEL     6.8.9+ #53
[59207.912746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[59207.912735] Kernel panic - not syncing: softlockup: hung tasks
[59207.912748] RIP: 0010:__printk_cpu_sync_wait+0xf/0x20
[59207.912817] Code: c3 cc cc cc cc 66 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 f3 90 <8b> 05 4f be 90 01 83 f8 ff 75 f3 5d c3 cc cc cc cc 90 90 90 90 90
[59207.912819] RSP: 0000:ffff9aafc3cbfe20 EFLAGS: 00000013
[59207.912822] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 000000000000001f
[59207.912823] RDX: 0000000000000003 RSI: 000000002829102e RDI: fffffffc34c5e070
[59207.912824] RBP: ffff9aafc3cbfe20 R08: 0000000000000000 R09: 7fffffffffffffff
[59207.912825] R10: 000035d972e84500 R11: 0000204060131a13 R12: 0000000000008101
[59207.912826] R13: ffff9aafc3cbff58 R14: 0000000000000007 R15: ffff8bd63bda5240
[59207.912827] FS:  00007f0efd600640(0000) GS:ffff8bd63bd80000(0000) knlGS:0000000000000000
[59207.912829] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[59207.912830] CR2: 000055e8d6949000 CR3: 000000012ce4e000 CR4: 0000000000b50ef0
[59207.912833] Call Trace:
[59207.912836]  <NMI>
[59207.912838]  ? show_regs+0x6e/0x80
[59207.912841]  ? nmi_cpu_backtrace+0xa1/0x120
[59207.912907]  ? nmi_cpu_backtrace_handler+0x15/0x20
[59207.912910]  ? nmi_handle+0x63/0x160
[59207.912912]  ? default_do_nmi+0x45/0x120
[59207.912928]  ? exc_nmi+0x12e/0x1b0
[59207.912931]  ? end_repeat_nmi+0xf/0x60
[59207.912935]  ? __printk_cpu_sync_wait+0xf/0x20
[59207.912937]  ? __printk_cpu_sync_wait+0xf/0x20
[59207.912938]  ? __printk_cpu_sync_wait+0xf/0x20
[59207.912940]  </NMI>
[59207.912940]  <TASK>
[59207.912941]  watchdog_timer_fn+0x1a9/0x280
[59207.912944]  ? __pfx_watchdog_timer_fn+0x10/0x10
[59207.912947]  __hrtimer_run_queues+0x114/0x2b0
[59207.912950]  hrtimer_interrupt+0x105/0x230
[59207.912953]  __sysvec_apic_timer_interrupt+0x57/0x150
[59207.912956]  sysvec_apic_timer_interrupt+0x42/0x90
[59207.912957]  asm_sysvec_apic_timer_interrupt+0x1f/0x30
[59207.912959] RIP: 0033:0x7f0efeafb308
[59207.913002] Code: 24 50 48 8b 44 24 38 4c 8b 0e 49 8d 76 04 4c 89 44 24 20 45 89 e8 49 89 44 24 38 49 89 c7 48 89 74 24 28 0f 1f 80 00 00 00 00 <48> 63 44 9d 00 41 89
de 48 c1 e0 04 4c 01 c8 8b 50 08 0f b6 78 08
[59207.913003] RSP: 002b:00007f0efd5ff630 EFLAGS: 00000212
[59207.913004] RAX: 0000000000000000 RBX: 0000000000000009 RCX: 0000000000000086
[59207.913005] RDX: 0000000000000002 RSI: 00007f0ef0095170 RDI: 000000000000000a
[59207.913006] RBP: 00007f0ef00950a0 R08: 0000000000000038 R09: 00007f0ef006eda0
[59207.913022] R10: 0000000000000004 R11: 0000000000000000 R12: 00007f0ef002e390
[59207.913023] R13: 0000000000000000 R14: 0000000000000008 R15: 00007f0ef002e398
[59207.913025]  </TASK>
[59207.913027] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 17.490 msecs
[59207.913027] watchdog: BUG: soft lockup - CPU#0 stuck for 33025s! [swapper/0:0]
[59207.913079] Modules linked in: tls xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc overlay intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 rapl ppdev parport_pc e1000 i2c_piix4 floppy parport qemu_fw_cfg binfmt_misc sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efi_pstore ip_tables x_tables autofs4 raid10 raid456 libcrc32c async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0 bochs drm_vram_helper drm_ttm_
helper ttm drm_kms_helper drm input_leds psmouse serio_raw pata_acpi mac_hid aesni_intel crypto_simd cryptd [last unloaded: tempesta_lib(OE)]
[59207.913166] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Tainted: G           OEL     6.8.9+ #53
[59207.913169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[59207.913171] RIP: 0010:pv_native_safe_halt+0xf/0x20
[59207.913176] Code: 22 d7 c3 cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 05 ab 32 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
[59207.913178] RSP: 0018:ffffffffa0403e28 EFLAGS: 00000206
[59207.913179] RAX: ffff8bd63bc2f7a0 RBX: ffffffffa040a980 RCX: 4000000000000000
[59207.913181] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00000000012b6364
[59207.913182] RBP: ffffffffa0403e30 R08: 0000159926c419c0 R09: 0000000000000001
[59207.913183] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[59207.913184] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffffa040a038
[59207.913186] FS:  0000000000000000(0000) GS:ffff8bd63bc00000(0000) knlGS:0000000000000000
[59207.913187] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[59207.913189] CR2: 00007fd650be7000 CR3: 00000000132c4000 CR4: 0000000000b50ef0
[59207.913191] Call Trace:
[59207.913193]  <IRQ>
[59207.913195]  ? show_regs+0x6e/0x80
[59207.913199]  ? watchdog_timer_fn+0x200/0x280
[59207.913202]  ? __pfx_watchdog_timer_fn+0x10/0x10
[59207.913205]  ? __hrtimer_run_queues+0x114/0x2b0
[59207.913208]  ? hrtimer_interrupt+0x105/0x230
[59207.913211]  ? tick_nohz_stop_idle+0x4b/0x70
[59207.913288]  ? __sysvec_apic_timer_interrupt+0x57/0x150
[59207.913292]  ? sysvec_apic_timer_interrupt+0x7f/0x90
[59207.913294]  </IRQ>
[59207.913295]  <TASK>
[59207.913295]  ? asm_sysvec_apic_timer_interrupt+0x1f/0x30
[59207.913299]  ? pv_native_safe_halt+0xf/0x20
[59207.913300]  ? ct_kernel_exit.constprop.0+0x7d/0x90
[59207.913303]  ? default_idle+0xd/0x20
[59207.913305]  arch_cpu_idle+0xd/0x20
[59207.913307]  default_idle_call+0x30/0xe0
[59207.913359] CPU: 2 PID: 0 Comm: swapper/2 Kdump: loaded Tainted: G           OEL     6.8.9+ #53
[59207.913363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[59207.913364] RIP: 0010:pv_native_safe_halt+0xf/0x20
[59207.913368] Code: 22 d7 c3 cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 05 ab 32 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
[59207.913370] RSP: 0018:ffff9aafc00b3ea0 EFLAGS: 00000206
[59207.913372] RAX: ffff8bd63bd2f7a0 RBX: ffff8bd6203499c0 RCX: 4000000000000000
[59207.913373] RDX: 0000000000000001 RSI: 0000000000000002 RDI: 0000000000ffcff4
[59207.913374] RBP: ffff9aafc00b3ea8 R08: 0000159926f3efa7 R09: 0000000000000001
[59207.913375] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[59207.913309]  do_idle+0x1f6/0x240
[59207.913419] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[59207.913420]  cpu_startup_entry+0x31/0x40
[59207.913420] FS:  0000000000000000(0000) GS:ffff8bd63bd00000(0000) knlGS:0000000000000000
[59207.913422] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[59207.913423] CR2: 00007fd6d9601020 CR3: 000000001260a000 CR4: 0000000000b50ef0
[59207.913423]  rest_init+0xd9/0xe0
[59207.913426] Call Trace:
[59207.913427]  <IRQ>
[59207.913429]  ? show_regs+0x6e/0x80
[59207.913432]  ? dump_cpu_task+0x6e/0x80
[59207.913473]  ? rcu_dump_cpu_stacks+0xc3/0x100
[59207.913478]  ? rcu_sched_clock_irq+0x675/0x1070
[59207.913480]  ? blk_mq_run_hw_queues+0x87/0x110
[59207.913426]  arch_call_rest_init+0x12/0x50
[59207.913594]  start_kernel+0x4d9/0x780
[59207.913596]  x86_64_start_reservations+0x1c/0x30
[59207.913705]  x86_64_start_kernel+0x80/0x80
[59207.913707]  secondary_startup_64_no_verify+0x175/0x17b
[59207.913712]  </TASK>
[59207.913713] watchdog: BUG: soft lockup - CPU#3 stuck for 33025s! [pool-udisksd:13001]
[59207.913715] Modules linked in: tls xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc overlay intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 rapl ppdev parport_pc e1000 i2c_piix4 floppy parport qemu_fw_cfg binfmt_misc sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efi_pstore ip_tables x_tables autofs4 raid10 raid456 libcrc32c async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0 bochs drm_vram_helper drm_ttm_helper ttm drm_kms_helper drm input_leds psmouse serio_raw pata_acpi mac_hid aesni_intel crypto_simd cryptd [last unloaded: tempesta_lib(OE)]
[59207.913745] CPU: 3 PID: 13001 Comm: pool-udisksd Kdump: loaded Tainted: G           OEL     6.8.9+ #53
[59207.913747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[59207.913747] RIP: 0033:0x7f0efeafb308
[59207.913749] Code: 24 50 48 8b 44 24 38 4c 8b 0e 49 8d 76 04 4c 89 44 24 20 45 89 e8 49 89 44 24 38 49 89 c7 48 89 74 24 28 0f 1f 80 00 00 00 00 <48> 63 44 9d 00 41 89 de 48 c1 e0 04 4c 01 c8 8b 50 08 0f b6 78 08
[59207.913750] RSP: 002b:00007f0efd5ff630 EFLAGS: 00000212
[59207.913751] RAX: 0000000000000000 RBX: 0000000000000009 RCX: 0000000000000086
[59207.913752] RDX: 0000000000000002 RSI: 00007f0ef0095170 RDI: 000000000000000a
[59207.913753] RBP: 00007f0ef00950a0 R08: 0000000000000038 R09: 00007f0ef006eda0
[59207.913754] R10: 0000000000000004 R11: 0000000000000000 R12: 00007f0ef002e390
[59207.913754] R13: 0000000000000000 R14: 0000000000000008 R15: 00007f0ef002e398
[59207.913755] FS:  00007f0efd600640 GS:  0000000000000000
[59207.913583]  ? scsi_run_queue_async+0x79/0x80
[59207.913779]  ? update_process_times+0x67/0xa0
[59207.913782]  ? __pfx_tick_nohz_highres_handler+0x10/0x10
[59207.913785]  ? tick_sched_handle+0x28/0x70
[59207.913787]  ? __pfx_tick_nohz_highres_handler+0x10/0x10
[59207.913790]  ? tick_nohz_highres_handler+0x7b/0xa0
[59207.913792]  ? __hrtimer_run_queues+0x114/0x2b0
[59207.913795]  ? hrtimer_interrupt+0x105/0x230
[59207.913797]  ? tick_nohz_stop_idle+0x4b/0x70
[59207.913800]  ? __sysvec_apic_timer_interrupt+0x57/0x150
[59207.913803]  ? sysvec_apic_timer_interrupt+0x7f/0x90
[59207.913805]  </IRQ>
[59207.913805]  <TASK>
[59207.913806]  ? asm_sysvec_apic_timer_interrupt+0x1f/0x30
[59207.913846]  ? pv_native_safe_halt+0xf/0x20
[59207.913849]  ? default_idle+0xd/0x20
[59207.913851]  arch_cpu_idle+0xd/0x20
[59207.913853]  default_idle_call+0x30/0xe0
[59207.913855]  do_idle+0x1f6/0x240
[59207.913858]  cpu_startup_entry+0x31/0x40
[59207.913860]  start_secondary+0x11d/0x140
[59207.913862]  secondary_startup_64_no_verify+0x175/0x17b
[59207.913866]  </TASK>
[59207.961114] CPU: 1 PID: 11146 Comm: kworker/1:2 Kdump: loaded Tainted: G           OEL     6.8.9+ #53
[59207.961395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[59207.961680] Workqueue: events drm_fb_helper_damage_work [drm_kms_helper]
[59207.961942] Call Trace:
[59207.962181]  <IRQ>
[59207.962416]  dump_stack_lvl+0x2b/0x90
[59207.962706]  dump_stack+0x14/0x20
[59207.913855]  do_idle+0x1f6/0x240
[59207.913858]  cpu_startup_entry+0x31/0x40
[59207.913860]  start_secondary+0x11d/0x140
[59207.913862]  secondary_startup_64_no_verify+0x175/0x17b
[59207.913866]  </TASK>
[59207.961114] CPU: 1 PID: 11146 Comm: kworker/1:2 Kdump: loaded Tainted: G           OEL     6.8.9+ #53
[59207.961395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[59207.961680] Workqueue: events drm_fb_helper_damage_work [drm_kms_helper]
[59207.961942] Call Trace:
[59207.962181]  <IRQ>
[59207.962416]  dump_stack_lvl+0x2b/0x90
[59207.962706]  dump_stack+0x14/0x20
[59207.962962]  panic+0x339/0x350
[59207.963219]  watchdog_timer_fn+0x23e/0x280
[59207.963434]  ? __pfx_watchdog_timer_fn+0x10/0x10
[59207.963642]  __hrtimer_run_queues+0x114/0x2b0
[59207.963853]  hrtimer_interrupt+0x105/0x230
[59207.964046]  __sysvec_apic_timer_interrupt+0x57/0x150
[59207.964245]  sysvec_apic_timer_interrupt+0x7f/0x90
[59207.964425]  </IRQ>
[59207.964608]  <TASK>
[59207.964773]  asm_sysvec_apic_timer_interrupt+0x1f/0x30
[59207.964950] RIP: 0010:memcpy_toio+0x78/0xc0
[59207.965108] Code: 5d c3 cc cc cc cc 48 85 db 74 f0 41 f6 c4 01 75 47 48 83 fb 01 76 06 41 f6 c4 02 75 27 48 89 d9 4c 89 e7 4c 89 ee 48 c1 e9 02 <f3> a5 f6 c3 02 74 02 66 a5 f6 c3 01 74 01 a4 5b 41 5c 41 5d 5d c3
[59207.965451] RSP: 0018:ffff9aafc2b57d78 EFLAGS: 00010206
[59207.965657] RAX: 0000000000000000 RBX: 0000000000001400 RCX: 0000000000000212
[59207.965833] RDX: 0000000000001400 RSI: ffff9aafc05c57b8 RDI: ffff9aafc0d087b8
[59207.966011] RBP: ffff9aafc2b57d90 R08: ffff8bd60004eeb0 R09: ffff8bd51efbaec0
[59207.966175] R10: 0000000000000007 R11: 0000000000000007 R12: ffff9aafc0d07c00
[59207.966343] R13: ffff9aafc05c4c00 R14: 00000000000000d3 R15: ffff9aafc2b57e20
[59207.966513]  ? memcpy_toio+0x25/0xc0
[59207.966688]  drm_fbdev_generic_helper_fb_dirty+0x1c2/0x2d0 [drm_kms_helper]
[59207.966878]  drm_fb_helper_damage_work+0x93/0x160 [drm_kms_helper]
[59207.967078]  process_one_work+0x179/0x350
[59207.967263]  ? __pfx_worker_thread+0x10/0x10
[59207.967436]  worker_thread+0x2f7/0x420
[59207.967616]  ? __pfx_worker_thread+0x10/0x10
[59207.967787]  kthread+0xfb/0x130
[59207.967959]  ? __pfx_kthread+0x10/0x10
[59207.968129]  ret_from_fork+0x40/0x60
[59207.968313]  ? __pfx_kthread+0x10/0x10
[59207.968481]  ret_from_fork_asm+0x1b/0x30
[59207.968696]  </TASK>

net/core/skbuff.c Outdated Show resolved Hide resolved
Now fpu can be enabled in the whole softirq context. Unlike the old
version, the new kernel raises softirq even in the boot phase, when fpu
related stuff (such as registers or process fpu state) is not ready for
manipulation (otherwise exception raised), so it should be enabled in
the first kernel_fpu_begin() (i.e. the first possible crypto API call
happens in the softirq context) instead of __do_softirq().
@kingluo kingluo marked this pull request as ready for review June 17, 2024 11:34
Copy link

@const-t const-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During testing I faced following problems:
1.txt
2.txt

2.txt was reproduced under load: h2load -c100 -m95 -t2 -D10 https://ubuntu h2load was ran on the same VM with Tempesta.

net/ipv4/tcp.c Outdated Show resolved Hide resolved
include/linux/fortify-string.h Outdated Show resolved Hide resolved
net/ipv6/tcp_ipv6.c Show resolved Hide resolved
arch/x86/kernel/cpu/common.c Outdated Show resolved Hide resolved
net/core/skbuff.c Outdated Show resolved Hide resolved
kernel/softirq.c Show resolved Hide resolved
@biathlon3
Copy link

biathlon3 commented Jun 27, 2024

Send request from host to VM
curl http://192.168.122.104:80

Backend is apache. Curl receives an html page from the server.

tempesta_fw.conf

listen 192.168.122.104:80;

srv_group default {server 127.0.0.1:82 conns_n=4;}

tls_match_any_server_name;

vhost default {
    proxy_pass default;
    #tls_match_any_server_name;
    frang_limits {
        http_strict_host_checking false;
        http_body_len 50000;
    }

}

http_chain {
  -> default;
}

log

[   69.753549] BUG: kernel NULL pointer dereference, address: 0000000000000008
[   69.756239] #PF: supervisor read access in kernel mode
[   69.757989] #PF: error_code(0x0000) - not-present page
[   69.759588] PGD 0 P4D 0
[   69.760552] Oops: 0000 [#1] PREEMPT SMP PTI
[   69.762106] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W  OE      6.8.9+ #1
[   69.764417] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.0-0-gd239552c-rebuilt.opensuse.org 04/01/2014
[   69.767997] RIP: 0010:selinux_netlbl_sk_security_free+0x14/0xe0
[   69.769928] Code: cc 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 41 56 41 55 41 54 53 <4c> 8b 6f 08 4d 85 ed 74 49 41 8b 45 00 49 89 fc a9 00 00 00 01 75
[   69.775025] RSP: 0018:ffffa359000e8b80 EFLAGS: 00010286
[   69.776813] RAX: ffffffff8cc4cde0 RBX: ffffffff8e069930 RCX: 00000000020c2001
[   69.778953] RDX: ffff909ee0830000 RSI: 0000000000000000 RDI: 0000000000000000
[   69.781079] RBP: ffffa359000e8ba0 R08: 0000000000000000 R09: 0000000000000000
[   69.783473] R10: 0000000000000c00 R11: 0000000000000000 R12: 0000000000000000
[   69.785782] R13: ffff909ee2bc2a00 R14: 0000000000000000 R15: ffff909ee3a4c7c0
[   69.787904] FS:  0000000000000000(0000) GS:ffff909f3bd00000(0000) knlGS:0000000000000000
[   69.790125] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   69.791469] CR2: 0000000000000008 CR3: 0000000129c38004 CR4: 0000000000370ef0
[   69.793005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   69.794612] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   69.796252] Call Trace:
[   69.797308]  <IRQ>
[   69.798240]  ? show_regs+0x6e/0x80
[   69.799338]  ? __die+0x29/0x70
[   69.800429]  ? page_fault_oops+0x160/0x470
[   69.801788]  ? avc_has_perm+0x47/0xc0
[   69.803020]  ? _raw_spin_unlock+0x12/0x30
[   69.804402]  ? free_unref_page+0x104/0x170
[   69.805713]  ? do_user_addr_fault+0x2f2/0x6c0
[   69.807078]  ? exc_page_fault+0x7d/0x190
[   69.808362]  ? asm_exc_page_fault+0x2b/0x30
[   69.809685]  ? __pfx_selinux_sk_free_security+0x10/0x10
[   69.811200]  ? selinux_netlbl_sk_security_free+0x14/0xe0
[   69.812780]  selinux_sk_free_security+0x29/0x40
[   69.814199]  security_sk_free+0x28/0x40
[   69.815512]  __sk_destruct+0x150/0x240
[   69.816774]  ? sk_filter_trim_cap+0x123/0x260
[   69.818197]  sk_destruct+0x54/0x60
[   69.819358]  __sk_free+0x30/0xd0
[   69.820529]  sk_free+0x2e/0x50
[   69.821657]  sock_put+0x25/0x50
[   69.822797]  tcp_v4_rcv+0xede/0xf20
[   69.823936]  ? raw_local_deliver+0xcd/0x240
[   69.825207]  ip_protocol_deliver_rcu+0x40/0x1e0
[   69.826613]  ip_local_deliver_finish+0x8a/0xb0
[   69.827967]  ip_local_deliver+0x73/0x120
[   69.829209]  ? tcp_write_xmit+0x481/0x1550
[   69.830490]  ? __pfx_ip_local_deliver_finish+0x10/0x10
[   69.831969]  ip_rcv+0x18f/0x1b0
[   69.833032]  ? __pfx_ip_rcv_finish+0x10/0x10
[   69.834340]  __netif_receive_skb_one_core+0x8a/0xa0
[   69.835774]  __netif_receive_skb+0x15/0x60
[   69.836869]  process_backlog+0x9a/0x140
[   69.838155]  __napi_poll+0x30/0x1f0
[   69.839565]  net_rx_action+0x194/0x2f0
[   69.840958]  __do_softirq+0xde/0x320
[   69.842148]  __irq_exit_rcu+0x6b/0x90
[   69.843298]  irq_exit_rcu+0x12/0x20
[   69.844439]  common_interrupt+0x92/0xa0
[   69.845614]  </IRQ>
[   69.846390]  <TASK>
[   69.847171]  asm_common_interrupt+0x2b/0x40
[   69.848402] RIP: 0010:pv_native_safe_halt+0xf/0x20
[   69.849757] Code: 22 d7 c3 cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d b5 d8 3b 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
[   69.854366] RSP: 0018:ffffa359000abe88 EFLAGS: 00000242
[   69.855852] RAX: ffff909f3bd2f7a0 RBX: ffff909ee0830000 RCX: 4000000000000000
[   69.857741] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000000000015d14
[   69.859614] RBP: ffffa359000abe90 R08: 000000103d2f802f R09: 0000000000000001
[   69.861503] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[   69.863398] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   69.865292]  ? ct_kernel_exit.constprop.0+0x7d/0x90
[   69.866709]  ? default_idle+0xd/0x20
[   69.867833]  arch_cpu_idle+0xd/0x20
[   69.868933]  default_idle_call+0x30/0xe0
[   69.870130]  do_idle+0x200/0x240
[   69.871179]  ? complete+0x71/0x80
[   69.872237]  cpu_startup_entry+0x31/0x40
[   69.873435]  start_secondary+0x12d/0x160
[   69.874651]  secondary_startup_64_no_verify+0x184/0x18b
[   69.876220]  </TASK>
[   69.877141] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc overlay intel_rapl_msr intel_rapl_common kvm_intel kvm rapl snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi input_leds serio_raw snd_hda_codec joydev snd_hda_core snd_hwdep snd_pcm snd_timer snd soundcore qemu_fw_cfg mac_hid binfmt_misc sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 hid_generic usbhid hid qxl drm_ttm_helper ttm drm_kms_helper virtio_net net_failover ahci crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sha512_ssse3 drm e1000e psmouse sha256_ssse3 lpc_ich libahci i2c_i801 i2c_smbus virtio_blk
[   69.877229]  failover virtio_rng xhci_pci xhci_pci_renesas aesni_intel crypto_simd cryptd
[   69.901878] CR2: 0000000000000008
[   69.903076] ---[ end trace 0000000000000000 ]---
[   69.904530] RIP: 0010:selinux_netlbl_sk_security_free+0x14/0xe0
[   69.906271] Code: cc 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 41 56 41 55 41 54 53 <4c> 8b 6f 08 4d 85 ed 74 49 41 8b 45 00 49 89 fc a9 00 00 00 01 75
[   69.911196] RSP: 0018:ffffa359000e8b80 EFLAGS: 00010286
[   69.912834] RAX: ffffffff8cc4cde0 RBX: ffffffff8e069930 RCX: 00000000020c2001
[   69.914889] RDX: ffff909ee0830000 RSI: 0000000000000000 RDI: 0000000000000000
[   69.916949] RBP: ffffa359000e8ba0 R08: 0000000000000000 R09: 0000000000000000
[   69.919003] R10: 0000000000000c00 R11: 0000000000000000 R12: 0000000000000000
[   69.921017] R13: ffff909ee2bc2a00 R14: 0000000000000000 R15: ffff909ee3a4c7c0
[   69.923053] FS:  0000000000000000(0000) GS:ffff909f3bd00000(0000) knlGS:0000000000000000
[   69.925289] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   69.927045] CR2: 0000000000000008 CR3: 0000000129c38004 CR4: 0000000000370ef0
[   69.929096] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   69.931183] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   69.933543] Kernel panic - not syncing: Fatal exception in interrupt
[   69.935946] Kernel Offset: 0xb600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   69.939400] Rebooting in 1 seconds..

@const-t
Copy link

const-t commented Jun 27, 2024

I've changed the patch little bit to test the fpu, in my case I can boot the system without any troubles. It would be great if other reviewers also apply this patch and test it. @krizhanovsky @biathlon3 @EvgeniiMekhanik

Patch was applied to commit of the kernel 13f5dc9 and Tempesta: e95951a21ea4c445b37309f1f4d388428854fabb

Files: diff.tar.gz

@kingluo
Copy link
Author

kingluo commented Jun 27, 2024

@biathlon3 I cannot reproduce your errors in my env. What's your backend and did you run curl on the same machine? Did you enable selinux? If so, why? And, did you change the code?

@biathlon3
Copy link

biathlon3 commented Jun 27, 2024

@kingluo My backend Apache. I run curl on the host, Tempesta on VM. I did not enable selinux specially, I will try to disable it. I got this bug with and without changes.

@kingluo
Copy link
Author

kingluo commented Jun 27, 2024

@kingluo My backend Apache. I run curl on the host, Tempesta on VM. I did not enable selinux specially, I will try to disable it. I got this bug with and without changes.

Well, I could not reproduce it. Please make sure the test environment is clean, for example, create a new qemu VM instance and try to install the new kernel of my branch without making any changes. PS, before a final confirmation, I would rather say this is a suspected issue than a bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants