TEST ipq806x: add test patch to improve stability #11173

Ansuel · 2022-11-05T17:45:58Z

Add test patch to improve stability.

One patch is a qsdk backport...
The other is a major rework with the idea of moving everything to safe mux while scaling core clocks... From my limited test scaling looks to work correctly but wonder how it behave under load.

Signed-off-by: Christian Marangi ansuelsmth@gmail.com

Add test patch to improve stability. Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

Ansuel · 2022-11-05T17:46:29Z

@hnyman want to test?

hnyman · 2022-11-05T19:56:51Z

Thanks.
I built r21185-52167feff8 with it, and will report later about the stability.

motolav · 2022-11-06T01:53:01Z

I'm trying out your patch on my C2600 now but off topic/semi related I was getting regular crashes after 1/2 days running master snapshots with the performance governor but I'm not sure how to setup ramoops to see why it was crashing

hnyman · 2022-11-06T09:33:33Z

Well, at least 13 hours uptime already. No problems so far. ;-)

Just wanted to provide feedback that the patch apparently clearly changes the CPU freq idle time behaviour:
During the idle night time the router has had clearly more 1.7 GHz time and more CPU transitions than before.
Earlier the core0 has maybe 2-3% of "high CPU speed" while now it has maybe 13-15%. core1 has increased from 1-2% to 5-8%.
I am not saying that it is bad, I just say that the impact from this PR is visible. Probably an effect from changing more often both cores to the "safe rate".

(Note the short flashing gaps in the graph data: master 5.15 until 20:15, then 22.03 for almost two hours, then at 22:00 master 5.15 including this PR. And then 10 minutes of varying traffic load testing with flent after flashing, after which an idle night.)

Ansuel · 2022-11-06T13:11:12Z

Earlier the core0 has maybe 2-3% of "high CPU speed" while now it has maybe 13-15%. core1 has increased from 1-2% to 5-8%.
I am not saying that it is bad, I just say that the impact from this PR is visible. Probably an effect from changing more often both cores to the "safe rate".

well yes this is expected but I first need to understand if this is a solution to then try to reduce it... Currently we use really expensive function to make a simple clk change.

@motolav that is strange maybe @hnyman can check if ramoops can be introduced and check if there are some interesting crash log

hnyman · 2022-11-06T14:23:24Z

maybe @hnyman can check if ramoops can be introduced and check if there are some interesting crash log

Regarding ramoops, I have no experience with C2600, but as I packaged the pstore kernel modules with 97158fe, the fundamentals are there, and it just needs to be enabled for each device.

For R7800 I then enabled it with two commits:

I defined the reserved memory area with cf346df,
(where I used the same memory area that RT3200 uses in mt7622. I do not think that it matters much as long as it does not conflict with an existing reserved area.)
I enabled kmod-ramoops as a default device package with 549f869

I assume that adding the same for C2600 might work. Likely/possibly the same memory address would work. @motolav might test it.

motolav · 2022-11-06T16:19:46Z

Thanks @hnyman, I got loaded with my new build and if it works I'll send a patch in. I was at 14.5 hours without any issue

[    9.686278] pstore: Using crash dump compression: deflate
[    9.686309] pstore: Registered ramoops as persistent store backend
[    9.690737] ramoops: using 0x40000@0x42100000, ecc: 0

hnyman · 2022-11-06T16:42:12Z

Note that you can trigger it also manually for testing...
You can cause a crash reboot using this command

echo c > /proc/sysrq-trigger

After the reboot you should have the crash log file in /sys/fs/pstore

motolav · 2022-11-06T18:39:08Z

Thanks for the help, the reboot did save a crash log

hnyman · 2022-11-07T16:12:26Z

@Ansuel
I got suddenly a crash related to rrdtool (LuCI statistics).
I experienced it live while I was at my router.
I think that based on these lines, it is about kernel checking the user account of the statistics save process:

<4>[158813.285262] PC is at __get_user_1+0x8/0x20
<4>[158813.289946] LR is at fault_in_readable+0xfc/0x12c

(Never seen that before in the 6 years with R7800, and there has been no change regarding rrdtool for ages.)

full ramoops:

<4>[54838.216712] ath10k_pci 0000:01:00.0: Invalid VHT mcs 15 peer stats
<4>[71913.271549] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 0, skipped old beacon
<1>[158813.066115] 8<--- cut here ---
<1>[158813.066156] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
<1>[158813.068150] pgd = f1c5b082
<1>[158813.076527] [0000000c] *pgd=455a9835, *pte=00000000, *ppte=00000000
<0>[158813.079018] Internal error: Oops: 17 [#1] SMP ARM
<4>[158813.085612] Modules linked in: pppoe ppp_async ath10k_pci ath10k_core ath wireguard pptp pppox ppp_mppe ppp_generic nft_redir nft_nat nft_masq nft_flow_offload nft_fib_inet nft_ct nft_chain_nat nf_nat nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet nf_flow_table nf_conntrack_netlink nf_conntrack mac80211 libchacha20poly1305 ipt_REJECT curve25519_neon cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_esp xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc sch_cake poly1305_arm nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_quota nft_objref nft_numgen nft_log nft_limit nft_hash nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_counter nft_compat nf_tables nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcurve25519_generic libcrc32c iptable_mangle iptable_filter ipt_ah ipt_ECN ip_tables crc_ccitt compat chaoskey chacha_neon fuse cls_bpf act_bpf sch_tbf sch_ingress sch_htb sch_hfsc em_u32
<4>[158813.086408]  cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact ledtrig_usbport xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 msdos ip_gre gre ifb ip6_udp_tunnel udp_tunnel sit tunnel4 ip_tunnel tun hfsplus cdrom cifs oid_registry cifs_md4 cifs_arc4 asn1_decoder dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 sha1_generic seqiv md5 ecb des_generic libdes cmac usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform
<4>[158813.156117]  libahci libata scsi_mod scsi_common ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug vfat fat ext4 mbcache jbd2 exfat crc32c_generic
<4>[158813.264704] CPU: 1 PID: 24076 Comm: rrdtool queue Not tainted 5.15.76 #0
<4>[158813.278405] Hardware name: Generic DT based system
<4>[158813.285262] PC is at __get_user_1+0x8/0x20
<4>[158813.289946] LR is at fault_in_readable+0xfc/0x12c
<4>[158813.294114] pc : [<c0647670>]    lr : [<c0450610>]    psr: 20000013
<4>[158813.298978] sp : c53f5d98  ip : 00000051  fp : 00000638
<4>[158813.305486] r10: c53f5f08  r9 : c257e8a0  r8 : 00000000
<4>[158813.310782] r7 : 00000000  r6 : b6ba80f8  r5 : 00000008  r4 : 00000008
<4>[158813.316078] r3 : 00000fff  r2 : 00000055  r1 : beffffff  r0 : b6ba80f8
<4>[158813.322417] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
<4>[158813.329016] Control: 10c5787d  Table: 4aa0406a  DAC: 00000055
<1>[158813.336304] Register r0 information: non-paged memory
<1>[158813.342119] Register r1 information: 4-page vmalloc region starting at 0xbf000000 allocated at load_module+0x9a0/0x2610
<1>[158813.347249] Register r2 information: non-paged memory
<1>[158813.358264] Register r3 information: non-paged memory
<1>[158813.363212] Register r4 information: non-paged memory
<1>[158813.368334] Register r5 information: non-paged memory
<1>[158813.373455] Register r6 information: non-paged memory
<1>[158813.378576] Register r7 information: NULL pointer
<1>[158813.383698] Register r8 information: NULL pointer
<1>[158813.388472] Register r9 information: slab shmem_inode_cache start c257e760 pointer offset 320
<1>[158813.393252] Register r10 information: non-slab/vmalloc memory
<1>[158813.401841] Register r11 information: non-paged memory
<1>[158813.407656] Register r12 information: non-paged memory
<0>[158813.412777] Process rrdtool queue (pid: 24076, stack limit = 0xb1a155cd)
<0>[158813.417990] Stack: (0xc53f5d98 to 0xc53f6000)
<0>[158813.424934] 5d80:                                                       fefefeff c53f5df0
<0>[158813.429284] 5da0: c257e798 00000000 00000008 c53f5f30 00000000 c05f2080 00000000 00000008
<0>[158813.437531] 5dc0: 00001638 c0ad3408 00000000 c041ba30 00000008 c257e798 c58da000 c53f5f08
<0>[158813.445776] 5de0: c58da000 00000000 00000638 c53f4000 63692b82 00000000 0bcb8ccb 00000000
<0>[158813.454024] 5e00: c53f5e88 00000000 c257e8a0 c58da000 c53f5f08 00000092 b6bbf610 c041ca28
<0>[158813.462268] 5e20: dda92178 c043f018 c53f5f08 c257e818 c53f5f08 c53f5e88 c53f5f80 00000008
<0>[158813.470516] 5e40: c257e818 c53f5f08 c53f5e88 c53f5f80 00000000 00000092 b6bbf610 c041cb30
<0>[158813.478762] 5e60: b6bbf610 b6ce6ba0 00000002 c53f5f20 c53f5f20 00001638 00000000 c53f5f80
<0>[158813.487008] 5e80: 00000000 c0494464 c58da000 00000000 00001638 00000000 00000000 00000000
<0>[158813.495254] 5ea0: 00000000 00000000 00000000 00000000 c58da000 00000008 c53f5f08 00000000
<0>[158813.503502] 5ec0: c58da000 c04954c8 00000000 00001638 00000000 00000092 b6bbf610 00000008
<0>[158813.511747] 5ee0: c58da000 c53f5f08 c53f5f80 00001638 00000000 c04956dc c53f5f04 c53f5f08
<0>[158813.519995] 5f00: 00000000 00000000 00010000 00000000 00000008 c53f5f20 00000002 00000000
<0>[158813.528240] 5f20: b6ba80f8 00000008 00000000 00000000 00001638 c043a040 00000000 e57f1816
<0>[158813.536487] 5f40: fffff000 00000fff 00003238 00000000 b6ba8060 b6ce6ba0 00000002 00000000
<0>[158813.544733] 5f60: b6ce6ba0 c58da003 c58da000 00000002 00000000 c049580c 00000000 00000000
<0>[158813.552981] 5f80: 00001638 00000000 00000000 b6ba8060 b6ce6ba0 00000000 00000092 c03002a4
<0>[158813.561227] 5fa0: c53f4000 c0300060 b6ba8060 b6ce6ba0 00000006 b6ce6ba0 00000002 b6ba80f8
<0>[158813.569474] 5fc0: b6ba8060 b6ce6ba0 00000000 00000092 00000008 00000002 b6bbf680 b6bbf610
<0>[158813.577721] 5fe0: b6ba80ac b6ce6ba0 b6f9c58c b6f9aa7c 20000010 00000006 00000000 00000000
<0>[158813.585971] Code: 31a00001 e12fff1e e2902000 30d22001 (2a00001e)
<4>[158813.594381] ---[ end trace 599f174d2f055913 ]---

motolav · 2022-11-07T16:38:08Z

I just had a hard lockup after like 20 hours of uptime, no logs unfortunately
Update: just had another crash but it rebooted so I have logs this time

<1>[10046.969687] 8<--- cut here ---
<1>[10046.969745] Unable to handle kernel NULL pointer dereference at virtual address 00000000
<1>[10046.971662] pgd = 8449b73e
<1>[10046.979905] [00000000] *pgd=00000000
<0>[10046.982423] Internal error: Oops: 5 [#1] SMP ARM
<4>[10046.986144] Modules linked in: pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 cfg80211 slhc nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc_ccitt compat ledtrig_usbport seqiv cmac leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug crc32c_generic
<4>[10047.045684] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.15.74 #0
<4>[10047.067915] Hardware name: Generic DT based system
<4>[10047.073905] PC is at _raw_spin_unlock_bh+0x10/0x30
<4>[10047.078503] LR is at ath10k_mac_tx_push_txq+0x54/0x318 [ath10k_core]
<4>[10047.083282] pc : [<c09f9584>]    lr : [<bf3b1124>]    psr: 60000113
<4>[10047.089789] sp : c146da64  ip : c146da58  fp : c8466748
<4>[10047.095777] r10: c313ccc8  r9 : c29b8dc0  r8 : c29ba978
<4>[10047.100986] r7 : c29ba060  r6 : c29baa38  r5 : c84650a4  r4 : c29b8dc0
<4>[10047.106196] r3 : 00000000  r2 : 000085c9  r1 : 00000000  r0 : bf3b1124
<4>[10047.112794] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[10047.119306] Control: 10c5787d  Table: 4450806a  DAC: 00000051
<1>[10047.126508] Register r0 information: 90-page vmalloc region starting at 0xbf3a4000 allocated at load_module+0x9a0/0x2610
<1>[10047.132245] Register r1 information: NULL pointer
<1>[10047.143173] Register r2 information: non-paged memory
<1>[10047.147774] Register r3 information: NULL pointer
<1>[10047.152808] Register r4 information: non-slab/vmalloc memory
<1>[10047.157498] Register r5 information: slab kmalloc-4k start c8465000 pointer offset 164 size 4096
<1>[10047.163232] Register r6 information: non-slab/vmalloc memory
<1>[10047.171994] Register r7 information: non-slab/vmalloc memory
<1>[10047.177636] Register r8 information: non-slab/vmalloc memory
<1>[10047.183278] Register r9 information: non-slab/vmalloc memory
<1>[10047.188921] Register r10 information: slab kmalloc-cg-8k start c313c000 pointer offset 3272 size 8192
<1>[10047.194568] Register r11 information: slab kmalloc-4k start c8466000 pointer offset 1864 size 4096
<1>[10047.203682] Register r12 information: non-slab/vmalloc memory
<0>[10047.212533] Process swapper/1 (pid: 0, stack limit = 0x895da590)
<0>[10047.218349] Stack: (0xc146da64 to 0xc146e000)
<0>[10047.224428] da60:          bf3b1124 c2d3a9c0 00000000 c29b8ee4 000005ea c29b8dc0 c29b8dc0
<0>[10047.228688] da80: c84650a4 00000aac 000008c0 000008c0 00000001 00000000 00000000 bf3b15bc
<0>[10047.236846] daa0: c84650a4 c29b8dc0 00000001 bf1fe1a0 00000000 00000000 c77bb088 c313c500
<0>[10047.245006] dac0: 00000000 c29b8e60 c84650a4 c29b8e70 c0d05be8 c313c000 c8466000 c2d3a9c0
<0>[10047.253167] dae0: c313c500 000005ea 00000001 00000000 c2d3aa08 bf204330 80000071 c0355880
<0>[10047.261325] db00: 00000046 c2d3a9d8 c29b8dc0 00000002 efc070a8 c8466000 c2d3a9c0 c313c000
<0>[10047.269486] db20: c2d3a9c0 00000000 c15da800 c0c452e4 c0d05be8 00000000 1cf57000 c084e808
<0>[10047.277645] db40: 00000001 c0d03d00 c0c46e3a c146db80 00000000 c2d3a9c0 c313c000 c0c46e38
<0>[10047.285805] db60: c15da800 c146db78 c77bc700 00000001 1cf57000 c084efb4 00080000 00000008
<0>[10047.293963] db80: fffffff4 00010000 00000000 00000000 00000000 00080000 c2d3a9c0 00000000
<0>[10047.302123] dba0: c224d500 c146dbd8 00000000 c0d05be8 00000001 c09bd168 00000000 c2d3a9c0
<0>[10047.310284] dbc0: 00000000 c2d3a9c0 00000000 c224d500 c146dbd8 c09b9d38 c146dbe5 c146dbf0
<0>[10047.318444] dbe0: 00002007 000003a8 00000000 c224d5c4 00000000 000f51c0 c0dc608c ddcb61c0
<0>[10047.326604] dc00: c6a90b00 00000014 c26519c0 54bd0667 c224d000 c2d3a9c0 c2d3aa80 c15da500
<0>[10047.334762] dc20: c0c452e4 c084e808 00002007 c0d03d00 c0c46e3a c146dc68 00000001 c2d3a9c0
<0>[10047.342923] dc40: c224d000 c0c46e38 c15da500 c146dc60 c6a90b00 00000014 c26519c0 c084efb4
<0>[10047.351082] dc60: c2d33800 00000002 fffffff4 c08a5dc8 c224d000 c085d55c 00000000 0000112c
<0>[10047.359244] dc80: c2d33800 c224d000 c146dce4 c6cf0d80 c0d951cc c6a90514 00000014 c085e154
<0>[10047.367401] dca0: 0000002b c6cf0d80 00000000 00000002 c2651a10 c146dda0 c2503648 c26519c0
<0>[10047.375563] dcc0: c6a90514 bf28e78c 00000001 c03383a4 7f4abd2f 6101a8c0 00000001 00000007
<0>[10047.383722] dce0: 00000000 6101a8c0 5b12084c 00000000 00000000 00000000 7f4abd2f 00000000
<0>[10047.391882] dd00: 00000000 00000000 fc99bb01 00000002 00020602 00000081 00000000 00000000
<0>[10047.400040] dd20: 00000000 00000000 00000000 00000000 00000000 54bd0667 c0c448e0 c6cf0d80
<0>[10047.408200] dd40: 00000000 c378a740 c6cf0d80 c146dda0 c378a740 c0c46e18 c0dd10d4 c08cd354
<0>[10047.416359] dd60: c6cf0d80 00000000 00000001 c1d60000 00000000 c0d05be8 c0c46e18 c08503a4
<0>[10047.424520] dd80: c0353e28 00000000 c0dd10d8 00000000 c146dde4 c146ddec ddb9d440 c6cf0d80
<0>[10047.432680] dda0: ddb90500 c1d60000 00000000 00000000 c0dd09c0 00000000 c0aac2fc c1d60000
<0>[10047.440839] ddc0: 0000001a ddb9de08 00000001 c0dd09a4 00000040 ddb9dea0 ddb9deac c085233c
<0>[10047.448999] dde0: 60000113 c6cf0d80 ddb9d480 00000000 ddb9deb0 c08525e4 00000005 00000001
<0>[10047.457159] de00: ddb9deb0 00000040 c146de47 c146de48 c0d03d00 1cf57000 c146de50 c08518a8
<0>[10047.465319] de20: ddb9deb0 ddb9de00 0000012c c0c46e00 c146de48 c0851bfc c541d480 000edf56
<0>[10047.473479] de40: 00000000 00b9d440 c146de48 c146de48 c146de50 c146de50 ddb9d440 00000000
<0>[10047.481637] de60: 00000003 c0d0308c c0d03080 40000003 c146c000 00000100 c146de80 c03012f8
<0>[10047.489799] de80: 00000000 00000000 ddb9de80 c0d03080 c0c43218 0000000a c0c431a4 c0c46000
<0>[10047.497957] dea0: c0c46000 000edf55 c0d03d00 04200042 de80200c ffffe000 00000000 00000003
<0>[10047.506117] dec0: 00000000 c0c452f8 de80200c c146def8 00000923 c0323538 c0c452ec c0373764
<0>[10047.514278] dee0: c0d0532c c146df18 c0d7e960 de802000 c0c452f8 c065beac c07e0fdc 60000013
<0>[10047.522437] df00: ffffffff c146df4c ddb9c6f0 c146c000 00000000 c0300b7c 00000000 00000923
<0>[10047.530597] df20: 1cf57000 ddb9d440 00000000 3decb800 c1dd9040 00000000 ddb9c6f0 00000923
<0>[10047.538755] df40: 00000000 00000923 0ccbf0a0 c146df68 c07e0fbc c07e0fdc 60000013 ffffffff
<0>[10047.546915] df60: 00000051 c07e0fac ddb9c6f0 3df047e0 ddba0b00 ddb9c6f0 c1dd9040 00000000
<0>[10047.555076] df80: c0d04f54 32c0c700 00000923 00000000 00000000 c07e1228 c146c000 00000001
<0>[10047.563235] dfa0: c0d04f10 ddb9c6f0 c1dd9040 c0351360 4220406a c146dfb0 c0c456e8 c0d0d674
<0>[10047.571394] dfc0: 00000051 01c0387d c146dff8 00000097 00000051 10c0387d c146dff8 4220406a
<0>[10047.579555] dfe0: 512f04d0 00000000 00000000 c0351724 4343c06a 42301530 00000000 00000000
<0>[10047.587716] [<c09f9584>] (_raw_spin_unlock_bh) from [<bf3b1124>] (ath10k_mac_tx_push_txq+0x54/0x318 [ath10k_core])
<0>[10047.595896] [<bf3b1124>] (ath10k_mac_tx_push_txq [ath10k_core]) from [<bf3b15bc>] (ath10k_mac_op_wake_tx_queue+0xc4/0xd0 [ath10k_core])
<0>[10047.606121] [<bf3b15bc>] (ath10k_mac_op_wake_tx_queue [ath10k_core]) from [<bf1fe1a0>] (ieee80211_pspoll_get+0x918/0xad0 [mac80211])
<0>[10047.618195] [<bf1fe1a0>] (ieee80211_pspoll_get [mac80211]) from [<bf204330>] (ieee80211_subif_start_xmit_8023+0x394/0x49c [mac80211])
<0>[10047.630342] [<bf204330>] (ieee80211_subif_start_xmit_8023 [mac80211]) from [<c084e808>] (dev_hard_start_xmit+0xf8/0x1a0)
<0>[10047.642231] [<c084e808>] (dev_hard_start_xmit) from [<c084efb4>] (__dev_queue_xmit+0x624/0xa5c)
<0>[10047.653162] [<c084efb4>] (__dev_queue_xmit) from [<c09bd168>] (br_dev_queue_push_xmit+0x74/0x164)
<0>[10047.661585] [<c09bd168>] (br_dev_queue_push_xmit) from [<c09b9d38>] (br_dev_xmit+0x298/0x4ac)
<0>[10047.670612] [<c09b9d38>] (br_dev_xmit) from [<c084e808>] (dev_hard_start_xmit+0xf8/0x1a0)
<0>[10047.679117] [<c084e808>] (dev_hard_start_xmit) from [<c084efb4>] (__dev_queue_xmit+0x624/0xa5c)
<0>[10047.687277] [<c084efb4>] (__dev_queue_xmit) from [<c085e154>] (neigh_xmit+0xe4/0x240)
<0>[10047.695786] [<c085e154>] (neigh_xmit) from [<bf28e78c>] (nf_flow_offload_ip_hook+0x444/0x638 [nf_flow_table])
<0>[10047.703782] [<bf28e78c>] (nf_flow_offload_ip_hook [nf_flow_table]) from [<c08cd354>] (nf_hook_slow+0x48/0xdc)
<0>[10047.713670] [<c08cd354>] (nf_hook_slow) from [<c08503a4>] (__netif_receive_skb_core.constprop.0+0x8ec/0xbe0)
<0>[10047.723564] [<c08503a4>] (__netif_receive_skb_core.constprop.0) from [<c085233c>] (__netif_receive_skb_one_core+0x28/0x58)
<0>[10047.733462] [<c085233c>] (__netif_receive_skb_one_core) from [<c08525e4>] (process_backlog+0xf4/0x1dc)
<0>[10047.744310] [<c08525e4>] (process_backlog) from [<c08518a8>] (__napi_poll+0x34/0x18c)
<0>[10047.753596] [<c08518a8>] (__napi_poll) from [<c0851bfc>] (net_rx_action+0xdc/0x260)
<0>[10047.761494] [<c0851bfc>] (net_rx_action) from [<c03012f8>] (__do_softirq+0x110/0x2b8)
<0>[10047.768963] [<c03012f8>] (__do_softirq) from [<c0323538>] (irq_exit+0xc8/0x114)
<0>[10047.776946] [<c0323538>] (irq_exit) from [<c0373764>] (handle_domain_irq+0x6c/0x94)
<0>[10047.784065] [<c0373764>] (handle_domain_irq) from [<c065beac>] (gic_handle_irq+0x8c/0xb4)
<0>[10047.791706] [<c065beac>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<0>[10047.800036] Exception stack(0xc146df18 to 0xc146df60)
<0>[10047.807498] df00:                                                       00000000 00000923
<0>[10047.812541] df20: 1cf57000 ddb9d440 00000000 3decb800 c1dd9040 00000000 ddb9c6f0 00000923
<0>[10047.820700] df40: 00000000 00000923 0ccbf0a0 c146df68 c07e0fbc c07e0fdc 60000013 ffffffff
<0>[10047.828858] [<c0300b7c>] (__irq_svc) from [<c07e0fdc>] (cpuidle_enter_state+0x180/0x37c)
<0>[10047.837018] [<c07e0fdc>] (cpuidle_enter_state) from [<c07e1228>] (cpuidle_enter+0x3c/0x5c)
<0>[10047.845176] [<c07e1228>] (cpuidle_enter) from [<c0351360>] (do_idle+0x204/0x2a4)
<0>[10047.853250] [<c0351360>] (do_idle) from [<c0351724>] (cpu_startup_entry+0x1c/0x20)
<0>[10047.860800] [<c0351724>] (cpu_startup_entry) from [<42301530>] (0x42301530)
<0>[10047.868183] Code: e1a03000 e1a0000e e52de004 f57ff05b (e1d320b0) 
<4>[10047.875108] ---[ end trace 0d4628c96625a59a ]---
<0>[10047.895204] Kernel panic - not syncing: Fatal exception in interrupt
<2>[10047.895238] CPU0: stopping
<4>[10047.900616] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D           5.15.74 #0
<4>[10047.903139] Hardware name: Generic DT based system
<4>[10047.910689] [<c030e5b4>] (unwind_backtrace) from [<c030a130>] (show_stack+0x14/0x20)
<4>[10047.915295] [<c030a130>] (show_stack) from [<c0641dbc>] (dump_stack_lvl+0x40/0x4c)
<4>[10047.923194] [<c0641dbc>] (dump_stack_lvl) from [<c030cfdc>] (do_handle_IPI+0x144/0x184)
<4>[10047.930572] [<c030cfdc>] (do_handle_IPI) from [<c030d038>] (ipi_handler+0x1c/0x2c)
<4>[10047.938470] [<c030d038>] (ipi_handler) from [<c0379e9c>] (handle_percpu_devid_irq+0x84/0x16c)
<4>[10047.946113] [<c0379e9c>] (handle_percpu_devid_irq) from [<c0373760>] (handle_domain_irq+0x68/0x94)
<4>[10047.954705] [<c0373760>] (handle_domain_irq) from [<c065beac>] (gic_handle_irq+0x8c/0xb4)
<4>[10047.963559] [<c065beac>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<4>[10047.971802] Exception stack(0xc0d01ae0 to 0xc0d01b28)
<4>[10047.979269] 1ae0: c15da840 00000000 0000410f 0000410e c34fd240 c313c000 c0c46e38 c15da800
<4>[10047.984305] 1b00: 00000000 c2cf0d00 00000000 1cf4b000 000edf5b c0d01b30 c084f00c c09f962c
<4>[10047.992464] 1b20: 80000113 ffffffff
<4>[10048.000616] [<c0300b7c>] (__irq_svc) from [<c09f962c>] (_raw_spin_lock+0x2c/0x40)
<4>[10048.003922] [<c09f962c>] (_raw_spin_lock) from [<c084f00c>] (__dev_queue_xmit+0x67c/0xa5c)
<4>[10048.011561] [<c084f00c>] (__dev_queue_xmit) from [<c09bd168>] (br_dev_queue_push_xmit+0x74/0x164)
<4>[10048.019721] [<c09bd168>] (br_dev_queue_push_xmit) from [<c09b9d38>] (br_dev_xmit+0x298/0x4ac)
<4>[10048.028663] [<c09b9d38>] (br_dev_xmit) from [<c084e808>] (dev_hard_start_xmit+0xf8/0x1a0)
<4>[10048.037170] [<c084e808>] (dev_hard_start_xmit) from [<c084efb4>] (__dev_queue_xmit+0x624/0xa5c)
<4>[10048.045328] [<c084efb4>] (__dev_queue_xmit) from [<c085e154>] (neigh_xmit+0xe4/0x240)
<4>[10048.053836] [<c085e154>] (neigh_xmit) from [<bf28e78c>] (nf_flow_offload_ip_hook+0x444/0x638 [nf_flow_table])
<4>[10048.061827] [<bf28e78c>] (nf_flow_offload_ip_hook [nf_flow_table]) from [<c08cd354>] (nf_hook_slow+0x48/0xdc)
<4>[10048.071720] [<c08cd354>] (nf_hook_slow) from [<c08503a4>] (__netif_receive_skb_core.constprop.0+0x8ec/0xbe0)
<4>[10048.081617] [<c08503a4>] (__netif_receive_skb_core.constprop.0) from [<c085233c>] (__netif_receive_skb_one_core+0x28/0x58)
<4>[10048.091513] [<c085233c>] (__netif_receive_skb_one_core) from [<c08525e4>] (process_backlog+0xf4/0x1dc)
<4>[10048.102362] [<c08525e4>] (process_backlog) from [<c08518a8>] (__napi_poll+0x34/0x18c)
<4>[10048.111647] [<c08518a8>] (__napi_poll) from [<c0851bfc>] (net_rx_action+0xdc/0x260)
<4>[10048.119547] [<c0851bfc>] (net_rx_action) from [<c03012f8>] (__do_softirq+0x110/0x2b8)
<4>[10048.127011] [<c03012f8>] (__do_softirq) from [<c0323538>] (irq_exit+0xc8/0x114)
<4>[10048.134997] [<c0323538>] (irq_exit) from [<c0373764>] (handle_domain_irq+0x6c/0x94)
<4>[10048.142116] [<c0373764>] (handle_domain_irq) from [<c065beac>] (gic_handle_irq+0x8c/0xb4)
<4>[10048.149756] [<c065beac>] (gic_handle_irq) from [<c0300b7c>] (__irq_svc+0x5c/0x78)
<4>[10048.158086] Exception stack(0xc0d01ed0 to 0xc0d01f18)
<4>[10048.165550] 1ec0:                                     00000000 00000923 1cf4b000 ddb91440
<4>[10048.170591] 1ee0: 00000000 425e3a80 c1dd8840 00000000 ddb906f0 00000923 00000000 00000923
<4>[10048.178751] 1f00: 0ccbf0a0 c0d01f20 c07e0fbc c07e0fdc 60000013 ffffffff
<4>[10048.186906] [<c0300b7c>] (__irq_svc) from [<c07e0fdc>] (cpuidle_enter_state+0x180/0x37c)
<4>[10048.193332] [<c07e0fdc>] (cpuidle_enter_state) from [<c07e1228>] (cpuidle_enter+0x3c/0x5c)
<4>[10048.201665] [<c07e1228>] (cpuidle_enter) from [<c0351360>] (do_idle+0x204/0x2a4)
<4>[10048.209737] [<c0351360>] (do_idle) from [<c0351724>] (cpu_startup_entry+0x1c/0x20)
<4>[10048.217290] [<c0351724>] (cpu_startup_entry) from [<c0c0118c>] (start_kernel+0x6ac/0x6c0)

tapper82 · 2022-11-07T19:16:32Z

Hi I had a reboot today on r20893. I have never had one on this build before. Latest master crashes mor for me. Is there a how to on how to get the ramoops logs?

motolav · 2022-11-07T19:23:22Z

@tapper82 Which router are you using? If its not a R7800 then you have to modify your dts to be able to use ramoops.
Follow this comment for how to add ramoops to your device #11173 (comment)

tapper82 · 2022-11-07T19:29:01Z

Oops#1 Part1
<6>[ 5.749740] pstore: Using crash dump compression: deflate
<6>[ 5.749771] pstore: Registered ramoops as persistent store backend
<6>[ 5.754196] ramoops: using 0x40000@0x42100000, ecc: 0
<6>[ 5.761511] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
<5>[ 5.768375] SCSI subsystem initialized
<7>[ 5.775329] libata version 3.00 loaded.
<4>[ 5.777246] ahci 29000000.sata: supply ahci not found, using dummy regulator
<4>[ 5.777387] ahci 29000000.sata: supply phy not found, using dummy regulator
<4>[ 5.783522] ahci 29000000.sata: supply target not found, using dummy regulator
<6>[ 5.790444] ahci 29000000.sata: forcing port_map 0x0 -> 0x1
<6>[ 5.797439] ahci 29000000.sata: AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl platform mode
<6>[ 5.802853] ahci 29000000.sata: flags: ncq sntf pm led clo only pmp pio slum part ccc apst
<6>[ 5.812881] scsi host0: ahci
<6>[ 5.819813] ata1: SATA max UDMA/133 mmio [mem 0x29000000-0x2900017f] port 0x100 irq 32
<6>[ 6.153620] ata1: SATA link down (SStatus 0 SControl 300)
<6>[ 6.160483] ehci-fsl: Freescale EHCI Host controller driver
<6>[ 6.162847] ehci-platform: EHCI generic platform driver
<6>[ 6.179196] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
<6>[ 6.179702] ohci-platform: OHCI generic platform driver
<6>[ 6.185224] uhci_hcd: USB Universal Host Controller Interface driver
<6>[ 6.189905] ohci-pci: OHCI PCI platform driver
<6>[ 6.337099] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
<6>[ 6.337214] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
<6>[ 6.341998] xhci-hcd xhci-hcd.0.auto: hcc params 0x0228f065 hci version 0x100 quirks 0x0000002002010010
<6>[ 6.349177] xhci-hcd xhci-hcd.0.auto: irq 51, io mem 0x10000000
<6>[ 6.359061] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
<6>[ 6.364473] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
<6>[ 6.369916] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
<6>[ 6.379240] hub 1-0:1.0: USB hub found
<6>[ 6.384312] hub 1-0:1.0: 1 port detected
<6>[ 6.388964] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
<6>[ 6.392400] hub 2-0:1.0: USB hub found
<6>[ 6.399946] hub 2-0:1.0: 1 port detected
<6>[ 6.403762] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
<6>[ 6.407571] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 3
<6>[ 6.413058] xhci-hcd xhci-hcd.1.auto: hcc params 0x0228f065 hci version 0x100 quirks 0x0000002002010010
<6>[ 6.420437] xhci-hcd xhci-hcd.1.auto: irq 52, io mem 0x11000000
<6>[ 6.430009] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
<6>[ 6.435757] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 4
<6>[ 6.441246] xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0 SuperSpeed
<6>[ 6.449977] hub 3-0:1.0: USB hub found
<6>[ 6.455492] hub 3-0:1.0: 1 port detected
<6>[ 6.459350] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
<6>[ 6.463584] hub 4-0:1.0: USB hub found
<6>[ 6.471293] hub 4-0:1.0: 1 port detected
<6>[ 6.477989] usbcore: registered new interface driver usb-storage
<6>[ 6.479817] usbcore: registered new interface driver uas
<14>[ 6.485968] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
<14>[ 6.496831] init: - preinit -
<6>[ 6.911652] usb 4-1: new SuperSpeed USB device number 2 using xhci-hcd
<6>[ 7.022524] usb-storage 4-1:1.0: USB Mass Storage device detected
<6>[ 7.049049] scsi host1: usb-storage 4-1:1.0
<5>[ 8.092579] scsi 1:0:0:0: Direct-Access Kingston DataTraveler 3.0 PMAP PQ: 0 ANSI: 6
<5>[ 8.094356] sd 1:0:0:0: [sda] 121208832 512-byte logical blocks: (62.1 GB/57.8 GiB)
<5>[ 8.100170] sd 1:0:0:0: [sda] Write Protect is off
<7>[ 8.107196] sd 1:0:0:0: [sda] Mode Sense: 45 00 00 00
<5>[ 8.107519] sd 1:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
<6>[ 8.938519] ipq806x-gmac-dwmac 37400000.ethernet eth1: Register MEM_TYPE_PAGE_POOL RxQ-0
<6>[ 8.939524] dwmac1000: Master AXI performs any burst length
<6>[ 8.945780] ipq806x-gmac-dwmac 37400000.ethernet eth1: No Safety Features support found
<5>[ 8.947418] sd 1:0:0:0: [sda] Attached SCSI removable disk
<6>[ 8.961002] ipq806x-gmac-dwmac 37400000.ethernet eth1: IEEE 1588-2008 Advanced Timestamp supported
<6>[ 8.964714] ipq806x-gmac-dwmac 37400000.ethernet eth1: registered PTP clock
<6>[ 8.974171] ipq806x-gmac-dwmac 37400000.ethernet eth1: configuring for fixed/sgmii link mode
<6>[ 8.981165] ipq806x-gmac-dwmac 37400000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
<6>[ 8.989168] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
<14>[ 11.178965] mount_root: loading kmods from internal overlay
<14>[ 11.196629] kmodloader: loading kernel modules from //etc/modules-boot.d/*
<14>[ 11.199183] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
<5>[ 11.560551] UBIFS (ubi0:1): Mounting in unauthenticated mode
<5>[ 11.561042] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 191
<5>[ 11.617158] UBIFS (ubi0:1): recovery needed
<5>[ 11.782331] UBIFS (ubi0:1): recovery completed
<5>[ 11.782477] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
<5>[ 11.785743] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
<5>[ 11.793678] UBIFS (ubi0:1): FS size: 84819968 bytes (80 MiB, 668 LEBs), max 679 LEBs, journal size 4190208 bytes (3 MiB, 33 LEBs)
<5>[ 11.803585] UBIFS (ubi0:1): reserved for root: 4006256 bytes (3912 KiB)
<5>[ 11.815212] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID ECE845E2-BE92-4934-85A1-06ECA1AEADF1, small LPT model
<14>[ 11.823846] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
<14>[ 11.838816] block: extroot: not configured
<5>[ 11.839873] UBIFS (ubi0:1): un-mount UBI device 0
<5>[ 11.844139] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" stops
<5>[ 11.852212] UBIFS (ubi0:1): Mounting in unauthenticated mode
<5>[ 11.854894] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 192
<5>[ 11.942161] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
<5>[ 11.942212] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
<5>[ 11.948992] UBIFS (ubi0:1): FS size: 84819968 bytes (80 MiB, 668 LEBs), max 679 LEBs, journal size 4190208 bytes (3 MiB, 33 LEBs)
<5>[ 11.958998] UBIFS (ubi0:1): reserved for root: 4006256 bytes (3912 KiB)
<5>[ 11.970597] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID ECE845E2-BE92-4934-85A1-06ECA1AEADF1, small LPT model
<14>[ 12.063318] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
<14>[ 12.070148] block: extroot: not configured
<14>[ 12.073934] mount_root: switching to ubifs overlay
<12>[ 12.090549] urandom-seed: Seeding with /etc/urandom.seed
<6>[ 12.166328] ipq806x-gmac-dwmac 37400000.ethernet eth1: Link is Down
<14>[ 12.219380] procd: - early -
<14>[ 12.219717] procd: - watchdog -
<14>[ 12.221876] procd: Watchdog has previously reset the system
<14>[ 12.805430] procd: - watchdog -
<14>[ 12.805517] procd: Watchdog has previously reset the system
<14>[ 12.809919] procd: - ubus -
<14>[ 12.948732] procd: - init -
<14>[ 13.342097] urngd: v1.0.2 started.
<14>[ 13.444377] kmodloader: loading kernel modules from /etc/modules.d/*
<6>[ 13.455691] GACT probability on
<6>[ 13.456302] Mirror/redirect action on
<6>[ 13.462290] u32 classifier
<6>[ 13.462321] input device check on
<6>[ 13.464058] Actions configured
<6>[ 13.474375] Loading modules backported from Linux version v5.15.58-0-g7d8048d4e064
<6>[ 13.474416] Backport generated by backports.git v5.15.58-1-0-g42a95ce7
<6>[ 13.486969] usbcore: registered new interface driver ums-alauda
<6>[ 13.487871] usbcore: registered new interface driver ums-cypress
<6>[ 13.494253] usbcore: registered new interface driver ums-datafab
<6>[ 13.500231] usbcore: registered new interface driver ums-freecom
<6>[ 13.506240] usbcore: registered new interface driver ums-isd200
<6>[ 13.512427] usbcore: registered new interface driver ums-jumpshot
<6>[ 13.517750] usbcore: registered new interface driver ums-karma
<6>[ 13.524459] usbcore: registered new interface driver ums-sddr09
<6>[ 13.529863] usbcore: registered new interface driver ums-sddr55
<6>[ 13.536593] usbcore: registered new interface driver ums-usbat
<6>[ 13.553890] xt_time: kernel timezone is -0000
<7>[ 13.636111] ath10k_pci 0000:01:00.0: assign IRQ: got 43
<6>[ 13.636151] ath10k 5.15 driver, optimized for CT firmware, probing pci device: 0x46.
<6>[ 13.636637] ath10k_pci 0000:01:00.0: enabling device (0140 -> 0142)
<7>[ 13.643100] ath10k_pci 0000:01:00.0: enabling bus mastering
<7>[ 13.643501] qcom-pcie 1b500000.pci: msi#1 address_hi 0x0 address_lo 0x43643e98
<6>[ 13.643616] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
<4>[ 13.867310] exFAT-fs (sda): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
<6>[ 15.144935] ath10k_pci 0000:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
<6>[ 15.145029] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
<6>[ 15.160229] ath10k_pci 0000:01:00.0: firmware ver 10.4b-ct-9984-fW-13-5ae337bb1 api 5 features mfp,peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,tx-rc-CT,cust-stats-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT,wmi-bcn-rc-CT crc32 7ea63dc5
<6>[ 17.485707] ath10k_pci 0000:01:00.0: Loading BDF type 0
<6>[ 17.488772] ath10k_pci 0000:01:00.0: board_file api 2 bmi_id 0:1 crc32 826c8c57
<4>[ 21.137171] ath10k_pci 0000:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96
<4>[ 21.137224] ath10k_pci 0000:01:00.0: msdu-desc: 2500 skid: 32
<6>[ 21.219655] ath10k_pci 0000:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0'
<6>[ 21.220520] ath10k_pci 0000:01:00.0: wmi print 'free: 84920 iram: 13156 sram: 11224'
<6>[ 21.508914] ath10k_pci 0000:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-nvmem max-sta 32 raw 0 hwcrypto 1
<7>[ 21.603956] ath: EEPROM regdomain: 0x0
<7>[ 21.603987] ath: EEPROM indicates default country code should be used
<7>[ 21.604002] ath: doing EEPROM country->regdmn map search
<7>[ 21.604022] ath: country maps to regdmn code: 0x3a
<7>[ 21.604040] ath: Country alpha2 being used: US
<7>[ 21.604055] ath: Regpair used: 0x3a
<7>[ 21.614124] ath10k_pci 0001:01:00.0: assign IRQ: got 45
<6>[ 21.614177] ath10k 5.15 driver, optimized for CT firmware, probing pci device: 0x46.
<6>[ 21.616709] ath10k_pci 0001:01:00.0: enabling device (0140 -> 0142)
<7>[ 21.621098] ath10k_pci 0001:01:00.0: enabling bus mastering
<7>[ 21.621760] qcom-pcie 1b700000.pci: msi#1 address_hi 0x0 address_lo 0x43770e98
<6>[ 21.621948] ath10k_pci 0001:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
<6>[ 22.443076] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
<6>[ 22.443134] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
<6>[ 22.454833] ath10k_pci 0001:01:00.0: firmware ver 10.4b-ct-9984-fW-13-5ae337bb1 api 5 features mfp,peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,tx-rc-CT,cust-stats-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT,wmi-bcn-rc-CT crc32 7ea63dc5
<6>[ 24.775680] ath10k_pci 0001:01:00.0: Loading BDF type 0
<6>[ 24.776547] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:2 crc32 826c8c57
<4>[ 28.448861] ath10k_pci 0001:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96
<4>[ 28.448911] ath10k_pci 0001:01:00.0: msdu-desc: 2500 skid: 32
<6>[ 28.533335] ath10k_pci 0001:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0'
<6>[ 28.534236] ath10k_pci 0001:01:00.0: wmi print 'free: 84920 iram: 13156 sram: 11224'
<6>[ 28.840462] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-nvmem max-sta 32 raw 0 hwcrypto 1
<7>[ 28.933996] ath: EEPROM regdomain: 0x0
<7>[ 28.934029] ath: EEPROM indicates default country code should be used
<7>[ 28.934043] ath: doing EEPROM country->regdmn map search
<7>[ 28.934065] ath: country maps to regdmn code: 0x3a
<7>[ 28.934083] ath: Country alpha2 being used: US
<7>[ 28.934100] ath: Regpair used: 0x3a
<14>[ 28.948972] kmodloader: done loading kernel modules from /etc/modules.d/*
<6>[ 33.170898] ipq806x-gmac-dwmac 37400000.ethernet eth1: Register MEM_TYPE_PAGE_POOL RxQ-0
<6>[ 33.171660] dwmac1000: Master AXI performs any burst length
<6>[ 33.178061] ipq806x-gmac-dwmac 37400000.ethernet eth1: No Safety Features support found
<6>[ 33.193530] ipq806x-gmac-dwmac 37400000.ethernet eth1: IEEE 1588-2008 Advanced Timestamp supported
<6>[ 33.193658] ipq806x-gmac-dwmac 37400000.ethernet eth1: registered PTP clock
<6>[ 33.201388] ipq806x-gmac-dwmac 37400000.ethernet eth1: configuring for fixed/sgmii link mode
<6>[ 33.208440] ipq806x-gmac-dwmac 37400000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
<6>[ 33.217327] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
<6>[ 33.228396] br-lan: port 1(eth1.1) entered blocking state
<6>[ 33.231829] br-lan: port 1(eth1.1) entered disabled state
<6>[ 33.237467] device eth1.1 entered promiscuous mode
<6>[ 33.242759] device eth1 entered promiscuous mode
<6>[ 33.248829] br-lan: port 1(eth1.1) entered blocking state
<6>[ 33.252229] br-lan: port 1(eth1.1) entered forwarding state
<6>[ 33.267383] ipq806x-gmac-dwmac 37200000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
<6>[ 33.268069] dwmac1000: Master AXI performs any burst length
<6>[ 33.274639] ipq806x-gmac-dwmac 37200000.ethernet eth0: No Safety Features support found
<6>[ 33.289858] ipq806x-gmac-dwmac 37200000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
<6>[ 33.290005] ipq806x-gmac-dwmac 37200000.ethernet eth0: registered PTP clock
<6>[ 33.304226] ipq806x-gmac-dwmac 37200000.ethernet eth0: configuring for fixed/rgmii link mode
<6>[ 33.304837] ipq806x-gmac-dwmac 37200000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
<6>[ 33.325901] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<7>[ 33.871405] ath: EEPROM regdomain: 0x833a
<7>[ 33.871435] ath: EEPROM indicates we should expect a country code
<7>[ 33.874518] ath: doing EEPROM country->regdmn map search
<7>[ 33.880471] ath: country maps to regdmn code: 0x37
<7>[ 33.885930] ath: Country alpha2 being used: GB
<7>[ 33.890460] ath: Regpair used: 0x37
<7>[ 33.894976] ath: regdomain 0x833a dynamically updated by user
<7>[ 33.898289] ath: EEPROM regdomain: 0x833a
<7>[ 33.904234] ath: EEPROM indicates we should expect a country code
<7>[ 33.908162] ath: doing EEPROM country->regdmn map search
<7>[ 33.914323] ath: country maps to regdmn code: 0x37
<7>[ 33.919619] ath: Country alpha2 being used: GB
<7>[ 33.924289] ath: Regpair used: 0x37
<7>[ 33.928643] ath: regdomain 0x833a dynamically updated by user
<6>[ 34.251700] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
<4>[ 41.366635] ath10k_pci 0000:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96
<4>[ 41.366679] ath10k_pci 0000:01:00.0: msdu-desc: 2500 skid: 32
<6>[ 41.449120] ath10k_pci 0000:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0'
<6>[ 41.449972] ath10k_pci 0000:01:00.0: wmi print 'free: 84920 iram: 13156 sram: 11224'
<6>[ 41.834770] ath10k_pci 0000:01:00.0: rts threshold -1
<4>[ 41.835057] ath10k_pci 0000:01:00.0: Firmware lacks feature flag indicating a retry limit of > 2 is OK, requested limit: 4
<6>[ 41.855265] br-lan: port 2(wlan0) entered blocking state
<6>[ 41.855317] br-lan: port 2(wlan0) entered disabled state
<6>[ 41.860072] device wlan0 entered promiscuous mode
<6>[ 42.030600] ath10k_pci 0000:01:00.0: NOTE: Firmware DBGLOG output disabled in debug_mask: 0x10000000
<4>[ 47.855879] ath10k_pci 0001:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96
<4>[ 47.855920] ath10k_pci 0001:01:00.0: msdu-desc: 2500 skid: 32
<6>[ 47.940389] ath10k_pci 0001:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0'
<6>[ 47.941279] ath10k_pci 0001:01:00.0: wmi print 'free: 84920 iram: 13156 sram: 11224'
<6>[ 48.337004] ath10k_pci 0001:01:00.0: rts threshold -1
<4>[ 48.337329] ath10k_pci 0001:01:00.0: Firmware lacks feature flag indicating a retry limit of > 2 is OK, requested limit: 4
<6>[ 48.351941] br-lan: port 3(wlan1) entered blocking state
<6>[ 48.351996] br-lan: port 3(wlan1) entered disabled state
<6>[ 48.358074] device wlan1 entered promiscuous mode
<6>[ 48.691475] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
<6>[ 48.691725] br-lan: port 2(wlan0) entered blocking state
<6>[ 48.696969] br-lan: port 2(wlan0) entered forwarding state
<6>[ 50.455500] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
<6>[ 50.455755] br-lan: port 3(wlan1) entered blocking state
<6>[ 50.460976] br-lan: port 3(wlan1) entered forwarding state
<4>[ 136.679978] ath10k_pci 0000:01:00.0: Invalid VHT mcs 15 peer stats
<3>[ 2314.768703] ksmbd: Failed to send message: -32
<4>[ 3123.360012] ath10k_pci 0001:01:00.0: wmi: fixing invalid VHT TX rate code 0xff
<3>[ 5012.690360] ksmbd: Failed to send message: -32
<4>[ 5579.632564] ath10k_pci 0001:01:00.0: htt tx: fixing invalid VHT TX rate code 0xff
<4>[ 5969.533422] ath10k_pci 0000:01:00.0: Invalid peer id 5 or peer stats buffer, peer: 00000000 sta: 00000000
<3>[ 6359.091548] ksmbd: Failed to send message: -32
<3>[36122.960808] ksmbd: Failed to send message: -32
<3>[40169.454934] ksmbd: Failed to send message: -32
<3>[46934.568568] ksmbd: Failed to send message: -32
<3>[48288.100501] ksmbd: Failed to send message: -32
<3>[50987.699220] ksmbd: Failed to send message: -32
<3>[53688.504418] ksmbd: Failed to send message: -32
<3>[87902.987077] ksmbd: Failed to send message: -32
<3>[89256.971017] ksmbd: Failed to send message: -32
<3>[93307.685213] ksmbd: Failed to send message: -32
<3>[94651.606363] ksmbd: Failed to send message: -32
<3>[98693.970492] ksmbd: Failed to send message: -32
<3>[109864.544105] ksmbd: Failed to send message: -32
<3>[112565.397536] ksmbd: Failed to send message: -32
<3>[115267.324067] ksmbd: Failed to send message: -32
<3>[122013.140552] ksmbd: Failed to send message: -32
<3>[124705.571359] ksmbd: Failed to send message: -32
<3>[128761.930902] ksmbd: Failed to send message: -32
<3>[130114.391944] ksmbd: Failed to send message: -32
<3>[134171.477499] ksmbd: Failed to send message: -32
<3>[186724.730148] ksmbd: Failed to send message: -32
<1>[189197.304324] 8<--- cut here ---
<1>[189197.304363] Unable to handle kernel NULL pointer dereference at virtual address 00000001
<1>[189197.306285] pgd = cb98b49e
<1>[189197.314614] [00000001] *pgd=00000000
<0>[189197.317222] Internal error: Oops: 17 [#1] SMP ARM
<4>[189197.321040] Modules linked in: ksmbd asn1_decoder nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath sch_fq_pie nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_compat nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 iptable_mangle iptable_filter ipt_REJECT ipt_ECN ip_tables cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY x_tables ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda sch_pie sch_cake nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex
<4>[189197.321834] cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact ledtrig_usbport ifb oid_registry autofs4 nls_utf8 seqiv md5 md4 ecb des_generic libdes cmac uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_pci uhci_hcd ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug exfat crc32c_generic
<4>[189197.435440] CPU: 1 PID: 20218 Comm: kworker/1:1 Not tainted 5.15.72 #0
<4>[189197.457674] Hardware name: Generic DT based system
<4>[189197.464270] Workqueue: events dbs_work_handler
<4>[189197.469126] PC is at __timer_delay+0x30/0x78
<4>[189197.473643] LR is at msm_read_current_timer+0x18/0x24
<4>[189197.478155] pc : [] lr : [] psr: a0000093
<4>[189197.483192] sp : cb243d48 ip : 00000000 fp : c1cf8800
<4>[189197.489701] r10: c179c018 r9 : 00000000 r8 : cb243dac
<4>[189197.494996] r7 : 00000002 r6 : 517cfd47 r5 : 00000006 r4 : 00000001
<4>[189197.500293] r3 : de806024 r2 : c0d9a33c r1 : 00000000 r0 : 517cfd47
<4>[189197.506632] Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
<4>[189197.513231] Control: 10c5787d Table: 44f7806a DAC: 00000051
<1>[189197.520518] Register r0 information: non-paged memory
<1>[189197.526420] Register r1 information: NULL pointer
<1>[189197.531541] Register r2 information: non-slab/vmalloc memory
<1>[189197.536317] Register r3 information: 0-page vmalloc region starting at 0xde806000 allocated at msm_dt_timer_init+0xc4/0x294
<1>[189197.542140] Register r4 information: non-paged memory
<1>[189197.553418] Register r5 information: non-paged memory
<1>[189197.558368] Register r6 information: non-paged memory
<1>[189197.563485] Register r7 information: non-paged memory
<1>[189197.568607] Register r8 information: non-slab/vmalloc memory
<1>[189197.573730] Register r9 information: NULL pointer
<1>[189197.579543] Register r10 information: slab kmalloc-256 start c179c000 pointer offset 24 size 256
<1>[189197.584240] Register r11 information: slab kmalloc-128 start c1cf8800 pointer offset 0 size 128
<1>[189197.593180] Register r12 information: NULL pointer
<0>[189197.602030] Process kworker/1:1 (pid: 20218, stack limit = 0x43738868)
<0>[189197.606631] Stack: (0xcb243d48 to 0xcb244000)
<0>[189197.613232] 3d40: c179bb40 00000001 20000013 c06ad978 c179bb58 00000001
<0>[189197.617752] 3d60: ffffffff c06ad9f8 c179bb64 c0d8082c ffffffff c06af14c 00000000 c0d8082c
<0>[189197.625998] 3d80: ffffffff c0342688 c179c000 c0d8082c c15e0e40 00000002 66d16940 23c34600
<0>[189197.634244] 3da0: 000927c0 c069c430 c158b800 c15aff00 66d16940 23c34600 c158b800 23c34600
<0>[189197.642491] 3dc0: c15e0e40 00000000 c15a5180 c06a0a54 c15e0d28 c15a5180 c06adea4 c14e3300
<0>[189197.650737] 3de0: 66d16940 23c34600 000927c0 c06a0a98 c15e0e40 00000000 23c34600 c15a5180
<0>[189197.658986] 3e00: dd99b010 23c34600 000927c0 c06a0cc4 23c34600 23c34600 00000000 ffffffff
<0>[189197.667231] 3e20: 23c34600 c0d82ad8 c1eeb780 c1eeb3c0 23c34600 00000001 c1cf8b80 dd99b010
<0>[189197.675481] 3e40: 23c34600 c06a0e4 c1610200 c1cf8980 00000001 c1cf8b80 dd99b010 c07d4d24
<0>[189197.683724] 3e60: c15e0e40 c1cf8b80 c1cf8f00 23c34600 00000000 c1610200 dd99b010 00000006
<0>[189197.691969] 3e80: 23c34600 c1cf8980 00000000 000927c0 00000000 c07d50d4 00000000 c1618200
<0>[189197.700216] 3ea0: c0dcff44 23c34600 c1618200 c0dcff44 00000000 00000001 001a524 c07d98ec
<0>[189197.708464] 3ec0: c1618200 001a524 000927c0 00000024 dd99e480 c1618200 c1cf8c80 c1cf8780
<0>[189197.716712] 3ee0: c1cf8c80 c1d32600 c1cf8780 dd9a1305 c2d01cc0 c07dcb54 c1cf8cb8 00000000
<0>[189197.724958] 3f00: c1cf8c84 c0d90710 00000000 00000040 dd9a1305 c07dd830 c1cf8cb8 c2d01c80
<0>[189197.733202] 3f20: dd99e040 dd9a1300 00000000 c0338d78 cb242000 dd99e040 00000008 c2d01c80
<0>[189197.741449] 3f40: c2d01c98 dd99e040 00000008 dd99e058 c0d03d00 dd99e200 cb242000 c0339078
<0>[189197.749698] 3f60: c0d0c36c c0d9aadc c56b7ecc c5f943c0 c2d42dc0 c0339004 c2d01c80 cb242000
<0>[189197.757942] 3f80: c56b7ecc c2d42de0 00000000 c0340d44 c5f943c0 c0340bec 00000000 00000000
<0>[189197.766188] 3fa0: 00000000 00000000 00000000 c0300130 00000000 00000000 00000000 00000000
<0>[189197.774435] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[189197.782681] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[189197.790927] [] (__timer_delay) from [] (__krait_mux_set_sel+0x7c/0x9c)
<0>[189197.799172] [] (__krait_mux_set_sel) from [] (krait_mux_set_parent+0x60/0x64)
<0>[189197.807416] [] (krait_mux_set_parent) from [] (krait_notifier_cb+0x58/0xb8)
<0>[189197.816445] [] (krait_notifier_cb) from [] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[189197.825388] [] (srcu_notifier_call_chain) from [] (__clk_notify+0x70/0x94)
<0>[189197.834501] [] (__clk_notify) from [] (clk_change_rate+0xfc/0x2b8)
<0>[189197.842919] [] (clk_change_rate) from [] (clk_change_rate+0x140/0x2b8)
<0>[189197.850905] [] (clk_change_rate) from [] (clk_core_set_rate_nolock+0xb4/0x1f8)
<0>[189197.859240] [] (clk_core_set_rate_nolock) from [] (clk_set_rate+0x3c/0x170)
<0>[189197.868266] [] (clk_set_rate) from [] (_set_opp+0x250/0x518)
<0>[189197.877297] [] (_set_opp) from [] (dev_pm_opp_set_rate+0xe8/0x218)
<0>[189197.884673] [] (dev_pm_opp_set_rate) from [] (__cpufreq_driver_target+0x194/0x238)
<0>[189197.892487] [] (__cpufreq_driver_target) from [] (od_dbs_update+0xcc/0x1a0)
<0>[189197.901863] [] (od_dbs_update) from [] (dbs_work_handler+0x38/0x74)
<0>[189197.910889] [] (dbs_work_handler) from [] (process_one_work+0x230/0x4bc)
<0>[189197.918962] [] (process_one_work) from [] (worker_thread+0x74/0x5d4)
<0>[189197.927380] [] (worker_thread) from [] (kthread+0x158/0x174)
<0>[189197.935540] [] (kthread) from [] (ret_from_fork+0x14/0x24)
<0>[189197.943004] Exception stack(0xcb243fb0 to 0xcb243ff8)
<0>[189197.950122] 3fa0: 00000000 00000000 00000000 00000000
<0>[189197.955338] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[189197.963581] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[189197.971832] Code: 0a000012 e5933000 e12fff33 e1a06000 (e5943000)
<4>[189197.978687] ---[ end trace 5961ab30030b334a ]---

tapper82 · 2022-11-07T19:29:26Z

Panic#2 Part1
<6>[ 6.359061] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
<6>[ 6.364473] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
<6>[ 6.369916] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
<6>[ 6.379240] hub 1-0:1.0: USB hub found
<6>[ 6.384312] hub 1-0:1.0: 1 port detected
<6>[ 6.388964] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
<6>[ 6.392400] hub 2-0:1.0: USB hub found
<6>[ 6.399946] hub 2-0:1.0: 1 port detected
<6>[ 6.403762] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
<6>[ 6.407571] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 3
<6>[ 6.413058] xhci-hcd xhci-hcd.1.auto: hcc params 0x0228f065 hci version 0x100 quirks 0x0000002002010010
<6>[ 6.420437] xhci-hcd xhci-hcd.1.auto: irq 52, io mem 0x11000000
<6>[ 6.430009] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
<6>[ 6.435757] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 4
<6>[ 6.441246] xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0 SuperSpeed
<6>[ 6.449977] hub 3-0:1.0: USB hub found
<6>[ 6.455492] hub 3-0:1.0: 1 port detected
<6>[ 6.459350] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
<6>[ 6.463584] hub 4-0:1.0: USB hub found
<6>[ 6.471293] hub 4-0:1.0: 1 port detected
<6>[ 6.477989] usbcore: registered new interface driver usb-storage
<6>[ 6.479817] usbcore: registered new interface driver uas
<14>[ 6.485968] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
<14>[ 6.496831] init: - preinit -
<6>[ 6.911652] usb 4-1: new SuperSpeed USB device number 2 using xhci-hcd
<6>[ 7.022524] usb-storage 4-1:1.0: USB Mass Storage device detected
<6>[ 7.049049] scsi host1: usb-storage 4-1:1.0
<5>[ 8.092579] scsi 1:0:0:0: Direct-Access Kingston DataTraveler 3.0 PMAP PQ: 0 ANSI: 6
<5>[ 8.094356] sd 1:0:0:0: [sda] 121208832 512-byte logical blocks: (62.1 GB/57.8 GiB)
<5>[ 8.100170] sd 1:0:0:0: [sda] Write Protect is off
<7>[ 8.107196] sd 1:0:0:0: [sda] Mode Sense: 45 00 00 00
<5>[ 8.107519] sd 1:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
<6>[ 8.938519] ipq806x-gmac-dwmac 37400000.ethernet eth1: Register MEM_TYPE_PAGE_POOL RxQ-0
<6>[ 8.939524] dwmac1000: Master AXI performs any burst length
<6>[ 8.945780] ipq806x-gmac-dwmac 37400000.ethernet eth1: No Safety Features support found
<5>[ 8.947418] sd 1:0:0:0: [sda] Attached SCSI removable disk
<6>[ 8.961002] ipq806x-gmac-dwmac 37400000.ethernet eth1: IEEE 1588-2008 Advanced Timestamp supported
<6>[ 8.964714] ipq806x-gmac-dwmac 37400000.ethernet eth1: registered PTP clock
<6>[ 8.974171] ipq806x-gmac-dwmac 37400000.ethernet eth1: configuring for fixed/sgmii link mode
<6>[ 8.981165] ipq806x-gmac-dwmac 37400000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
<6>[ 8.989168] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
<14>[ 11.178965] mount_root: loading kmods from internal overlay
<14>[ 11.196629] kmodloader: loading kernel modules from //etc/modules-boot.d/*
<14>[ 11.199183] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
<5>[ 11.560551] UBIFS (ubi0:1): Mounting in unauthenticated mode
<5>[ 11.561042] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 191
<5>[ 11.617158] UBIFS (ubi0:1): recovery needed
<5>[ 11.782331] UBIFS (ubi0:1): recovery completed
<5>[ 11.782477] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
<5>[ 11.785743] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
<5>[ 11.793678] UBIFS (ubi0:1): FS size: 84819968 bytes (80 MiB, 668 LEBs), max 679 LEBs, journal size 4190208 bytes (3 MiB, 33 LEBs)
<5>[ 11.803585] UBIFS (ubi0:1): reserved for root: 4006256 bytes (3912 KiB)
<5>[ 11.815212] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID ECE845E2-BE92-4934-85A1-06ECA1AEADF1, small LPT model
<14>[ 11.823846] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
<14>[ 11.838816] block: extroot: not configured
<5>[ 11.839873] UBIFS (ubi0:1): un-mount UBI device 0
<5>[ 11.844139] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" stops
<5>[ 11.852212] UBIFS (ubi0:1): Mounting in unauthenticated mode
<5>[ 11.854894] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 192
<5>[ 11.942161] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
<5>[ 11.942212] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
<5>[ 11.948992] UBIFS (ubi0:1): FS size: 84819968 bytes (80 MiB, 668 LEBs), max 679 LEBs, journal size 4190208 bytes (3 MiB, 33 LEBs)
<5>[ 11.958998] UBIFS (ubi0:1): reserved for root: 4006256 bytes (3912 KiB)
<5>[ 11.970597] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID ECE845E2-BE92-4934-85A1-06ECA1AEADF1, small LPT model
<14>[ 12.063318] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
<14>[ 12.070148] block: extroot: not configured
<14>[ 12.073934] mount_root: switching to ubifs overlay
<12>[ 12.090549] urandom-seed: Seeding with /etc/urandom.seed
<6>[ 12.166328] ipq806x-gmac-dwmac 37400000.ethernet eth1: Link is Down
<14>[ 12.219380] procd: - early -
<14>[ 12.219717] procd: - watchdog -
<14>[ 12.221876] procd: Watchdog has previously reset the system
<14>[ 12.805430] procd: - watchdog -
<14>[ 12.805517] procd: Watchdog has previously reset the system
<14>[ 12.809919] procd: - ubus -
<14>[ 12.948732] procd: - init -
<14>[ 13.342097] urngd: v1.0.2 started.
<14>[ 13.444377] kmodloader: loading kernel modules from /etc/modules.d/*
<6>[ 13.455691] GACT probability on
<6>[ 13.456302] Mirror/redirect action on
<6>[ 13.462290] u32 classifier
<6>[ 13.462321] input device check on
<6>[ 13.464058] Actions configured
<6>[ 13.474375] Loading modules backported from Linux version v5.15.58-0-g7d8048d4e064
<6>[ 13.474416] Backport generated by backports.git v5.15.58-1-0-g42a95ce7
<6>[ 13.486969] usbcore: registered new interface driver ums-alauda
<6>[ 13.487871] usbcore: registered new interface driver ums-cypress
<6>[ 13.494253] usbcore: registered new interface driver ums-datafab
<6>[ 13.500231] usbcore: registered new interface driver ums-freecom
<6>[ 13.506240] usbcore: registered new interface driver ums-isd200
<6>[ 13.512427] usbcore: registered new interface driver ums-jumpshot
<6>[ 13.517750] usbcore: registered new interface driver ums-karma
<6>[ 13.524459] usbcore: registered new interface driver ums-sddr09
<6>[ 13.529863] usbcore: registered new interface driver ums-sddr55
<6>[ 13.536593] usbcore: registered new interface driver ums-usbat
<6>[ 13.553890] xt_time: kernel timezone is -0000
<7>[ 13.636111] ath10k_pci 0000:01:00.0: assign IRQ: got 43
<6>[ 13.636151] ath10k 5.15 driver, optimized for CT firmware, probing pci device: 0x46.
<6>[ 13.636637] ath10k_pci 0000:01:00.0: enabling device (0140 -> 0142)
<7>[ 13.643100] ath10k_pci 0000:01:00.0: enabling bus mastering
<7>[ 13.643501] qcom-pcie 1b500000.pci: msi#1 address_hi 0x0 address_lo 0x43643e98
<6>[ 13.643616] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
<4>[ 13.867310] exFAT-fs (sda): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
<6>[ 15.144935] ath10k_pci 0000:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
<6>[ 15.145029] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
<6>[ 15.160229] ath10k_pci 0000:01:00.0: firmware ver 10.4b-ct-9984-fW-13-5ae337bb1 api 5 features mfp,peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,tx-rc-CT,cust-stats-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT,wmi-bcn-rc-CT crc32 7ea63dc5
<6>[ 17.485707] ath10k_pci 0000:01:00.0: Loading BDF type 0
<6>[ 17.488772] ath10k_pci 0000:01:00.0: board_file api 2 bmi_id 0:1 crc32 826c8c57
<4>[ 21.137171] ath10k_pci 0000:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96
<4>[ 21.137224] ath10k_pci 0000:01:00.0: msdu-desc: 2500 skid: 32
<6>[ 21.219655] ath10k_pci 0000:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0'
<6>[ 21.220520] ath10k_pci 0000:01:00.0: wmi print 'free: 84920 iram: 13156 sram: 11224'
<6>[ 21.508914] ath10k_pci 0000:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-nvmem max-sta 32 raw 0 hwcrypto 1
<7>[ 21.603956] ath: EEPROM regdomain: 0x0
<7>[ 21.603987] ath: EEPROM indicates default country code should be used
<7>[ 21.604002] ath: doing EEPROM country->regdmn map search
<7>[ 21.604022] ath: country maps to regdmn code: 0x3a
<7>[ 21.604040] ath: Country alpha2 being used: US
<7>[ 21.604055] ath: Regpair used: 0x3a
<7>[ 21.614124] ath10k_pci 0001:01:00.0: assign IRQ: got 45
<6>[ 21.614177] ath10k 5.15 driver, optimized for CT firmware, probing pci device: 0x46.
<6>[ 21.616709] ath10k_pci 0001:01:00.0: enabling device (0140 -> 0142)
<7>[ 21.621098] ath10k_pci 0001:01:00.0: enabling bus mastering
<7>[ 21.621760] qcom-pcie 1b700000.pci: msi#1 address_hi 0x0 address_lo 0x43770e98
<6>[ 21.621948] ath10k_pci 0001:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
<6>[ 22.443076] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
<6>[ 22.443134] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
<6>[ 22.454833] ath10k_pci 0001:01:00.0: firmware ver 10.4b-ct-9984-fW-13-5ae337bb1 api 5 features mfp,peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,tx-rc-CT,cust-stats-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT,wmi-bcn-rc-CT crc32 7ea63dc5
<6>[ 24.775680] ath10k_pci 0001:01:00.0: Loading BDF type 0
<6>[ 24.776547] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:2 crc32 826c8c57
<4>[ 28.448861] ath10k_pci 0001:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96
<4>[ 28.448911] ath10k_pci 0001:01:00.0: msdu-desc: 2500 skid: 32
<6>[ 28.533335] ath10k_pci 0001:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0'
<6>[ 28.534236] ath10k_pci 0001:01:00.0: wmi print 'free: 84920 iram: 13156 sram: 11224'
<6>[ 28.840462] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-nvmem max-sta 32 raw 0 hwcrypto 1
<7>[ 28.933996] ath: EEPROM regdomain: 0x0
<7>[ 28.934029] ath: EEPROM indicates default country code should be used
<7>[ 28.934043] ath: doing EEPROM country->regdmn map search
<7>[ 28.934065] ath: country maps to regdmn code: 0x3a
<7>[ 28.934083] ath: Country alpha2 being used: US
<7>[ 28.934100] ath: Regpair used: 0x3a
<14>[ 28.948972] kmodloader: done loading kernel modules from /etc/modules.d/*
<6>[ 33.170898] ipq806x-gmac-dwmac 37400000.ethernet eth1: Register MEM_TYPE_PAGE_POOL RxQ-0
<6>[ 33.171660] dwmac1000: Master AXI performs any burst length
<6>[ 33.178061] ipq806x-gmac-dwmac 37400000.ethernet eth1: No Safety Features support found
<6>[ 33.193530] ipq806x-gmac-dwmac 37400000.ethernet eth1: IEEE 1588-2008 Advanced Timestamp supported
<6>[ 33.193658] ipq806x-gmac-dwmac 37400000.ethernet eth1: registered PTP clock
<6>[ 33.201388] ipq806x-gmac-dwmac 37400000.ethernet eth1: configuring for fixed/sgmii link mode
<6>[ 33.208440] ipq806x-gmac-dwmac 37400000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
<6>[ 33.217327] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
<6>[ 33.228396] br-lan: port 1(eth1.1) entered blocking state
<6>[ 33.231829] br-lan: port 1(eth1.1) entered disabled state
<6>[ 33.237467] device eth1.1 entered promiscuous mode
<6>[ 33.242759] device eth1 entered promiscuous mode
<6>[ 33.248829] br-lan: port 1(eth1.1) entered blocking state
<6>[ 33.252229] br-lan: port 1(eth1.1) entered forwarding state
<6>[ 33.267383] ipq806x-gmac-dwmac 37200000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
<6>[ 33.268069] dwmac1000: Master AXI performs any burst length
<6>[ 33.274639] ipq806x-gmac-dwmac 37200000.ethernet eth0: No Safety Features support found
<6>[ 33.289858] ipq806x-gmac-dwmac 37200000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
<6>[ 33.290005] ipq806x-gmac-dwmac 37200000.ethernet eth0: registered PTP clock
<6>[ 33.304226] ipq806x-gmac-dwmac 37200000.ethernet eth0: configuring for fixed/rgmii link mode
<6>[ 33.304837] ipq806x-gmac-dwmac 37200000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
<6>[ 33.325901] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<7>[ 33.871405] ath: EEPROM regdomain: 0x833a
<7>[ 33.871435] ath: EEPROM indicates we should expect a country code
<7>[ 33.874518] ath: doing EEPROM country->regdmn map search
<7>[ 33.880471] ath: country maps to regdmn code: 0x37
<7>[ 33.885930] ath: Country alpha2 being used: GB
<7>[ 33.890460] ath: Regpair used: 0x37
<7>[ 33.894976] ath: regdomain 0x833a dynamically updated by user
<7>[ 33.898289] ath: EEPROM regdomain: 0x833a
<7>[ 33.904234] ath: EEPROM indicates we should expect a country code
<7>[ 33.908162] ath: doing EEPROM country->regdmn map search
<7>[ 33.914323] ath: country maps to regdmn code: 0x37
<7>[ 33.919619] ath: Country alpha2 being used: GB
<7>[ 33.924289] ath: Regpair used: 0x37
<7>[ 33.928643] ath: regdomain 0x833a dynamically updated by user
<6>[ 34.251700] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
<4>[ 41.366635] ath10k_pci 0000:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96
<4>[ 41.366679] ath10k_pci 0000:01:00.0: msdu-desc: 2500 skid: 32
<6>[ 41.449120] ath10k_pci 0000:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0'
<6>[ 41.449972] ath10k_pci 0000:01:00.0: wmi print 'free: 84920 iram: 13156 sram: 11224'
<6>[ 41.834770] ath10k_pci 0000:01:00.0: rts threshold -1
<4>[ 41.835057] ath10k_pci 0000:01:00.0: Firmware lacks feature flag indicating a retry limit of > 2 is OK, requested limit: 4
<6>[ 41.855265] br-lan: port 2(wlan0) entered blocking state
<6>[ 41.855317] br-lan: port 2(wlan0) entered disabled state
<6>[ 41.860072] device wlan0 entered promiscuous mode
<6>[ 42.030600] ath10k_pci 0000:01:00.0: NOTE: Firmware DBGLOG output disabled in debug_mask: 0x10000000
<4>[ 47.855879] ath10k_pci 0001:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96
<4>[ 47.855920] ath10k_pci 0001:01:00.0: msdu-desc: 2500 skid: 32
<6>[ 47.940389] ath10k_pci 0001:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0'
<6>[ 47.941279] ath10k_pci 0001:01:00.0: wmi print 'free: 84920 iram: 13156 sram: 11224'
<6>[ 48.337004] ath10k_pci 0001:01:00.0: rts threshold -1
<4>[ 48.337329] ath10k_pci 0001:01:00.0: Firmware lacks feature flag indicating a retry limit of > 2 is OK, requested limit: 4
<6>[ 48.351941] br-lan: port 3(wlan1) entered blocking state
<6>[ 48.351996] br-lan: port 3(wlan1) entered disabled state
<6>[ 48.358074] device wlan1 entered promiscuous mode
<6>[ 48.691475] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
<6>[ 48.691725] br-lan: port 2(wlan0) entered blocking state
<6>[ 48.696969] br-lan: port 2(wlan0) entered forwarding state
<6>[ 50.455500] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
<6>[ 50.455755] br-lan: port 3(wlan1) entered blocking state
<6>[ 50.460976] br-lan: port 3(wlan1) entered forwarding state
<4>[ 136.679978] ath10k_pci 0000:01:00.0: Invalid VHT mcs 15 peer stats
<3>[ 2314.768703] ksmbd: Failed to send message: -32
<4>[ 3123.360012] ath10k_pci 0001:01:00.0: wmi: fixing invalid VHT TX rate code 0xff
<3>[ 5012.690360] ksmbd: Failed to send message: -32
<4>[ 5579.632564] ath10k_pci 0001:01:00.0: htt tx: fixing invalid VHT TX rate code 0xff
<4>[ 5969.533422] ath10k_pci 0000:01:00.0: Invalid peer id 5 or peer stats buffer, peer: 00000000 sta: 00000000
<3>[ 6359.091548] ksmbd: Failed to send message: -32
<3>[36122.960808] ksmbd: Failed to send message: -32
<3>[40169.454934] ksmbd: Failed to send message: -32
<3>[46934.568568] ksmbd: Failed to send message: -32
<3>[48288.100501] ksmbd: Failed to send message: -32
<3>[50987.699220] ksmbd: Failed to send message: -32
<3>[53688.504418] ksmbd: Failed to send message: -32
<3>[87902.987077] ksmbd: Failed to send message: -32
<3>[89256.971017] ksmbd: Failed to send message: -32
<3>[93307.685213] ksmbd: Failed to send message: -32
<3>[94651.606363] ksmbd: Failed to send message: -32
<3>[98693.970492] ksmbd: Failed to send message: -32
<3>[109864.544105] ksmbd: Failed to send message: -32
<3>[112565.397536] ksmbd: Failed to send message: -32
<3>[115267.324067] ksmbd: Failed to send message: -32
<3>[122013.140552] ksmbd: Failed to send message: -32
<3>[124705.571359] ksmbd: Failed to send message: -32
<3>[128761.930902] ksmbd: Failed to send message: -32
<3>[130114.391944] ksmbd: Failed to send message: -32
<3>[134171.477499] ksmbd: Failed to send message: -32
<3>[186724.730148] ksmbd: Failed to send message: -32
<1>[189197.304324] 8<--- cut here ---
<1>[189197.304363] Unable to handle kernel NULL pointer dereference at virtual address 00000001
<1>[189197.306285] pgd = cb98b49e
<1>[189197.314614] [00000001] *pgd=00000000
<0>[189197.317222] Internal error: Oops: 17 [#1] SMP ARM
<4>[189197.321040] Modules linked in: ksmbd asn1_decoder nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath sch_fq_pie nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_compat nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 iptable_mangle iptable_filter ipt_REJECT ipt_ECN ip_tables cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY x_tables ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda sch_pie sch_cake nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex
<4>[189197.321834] cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact ledtrig_usbport ifb oid_registry autofs4 nls_utf8 seqiv md5 md4 ecb des_generic libdes cmac uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_pci uhci_hcd ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug exfat crc32c_generic
<4>[189197.435440] CPU: 1 PID: 20218 Comm: kworker/1:1 Not tainted 5.15.72 #0
<4>[189197.457674] Hardware name: Generic DT based system
<4>[189197.464270] Workqueue: events dbs_work_handler
<4>[189197.469126] PC is at __timer_delay+0x30/0x78
<4>[189197.473643] LR is at msm_read_current_timer+0x18/0x24
<4>[189197.478155] pc : [] lr : [] psr: a0000093
<4>[189197.483192] sp : cb243d48 ip : 00000000 fp : c1cf8800
<4>[189197.489701] r10: c179c018 r9 : 00000000 r8 : cb243dac
<4>[189197.494996] r7 : 00000002 r6 : 517cfd47 r5 : 00000006 r4 : 00000001
<4>[189197.500293] r3 : de806024 r2 : c0d9a33c r1 : 00000000 r0 : 517cfd47
<4>[189197.506632] Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
<4>[189197.513231] Control: 10c5787d Table: 44f7806a DAC: 00000051
<1>[189197.520518] Register r0 information: non-paged memory
<1>[189197.526420] Register r1 information: NULL pointer
<1>[189197.531541] Register r2 information: non-slab/vmalloc memory
<1>[189197.536317] Register r3 information: 0-page vmalloc region starting at 0xde806000 allocated at msm_dt_timer_init+0xc4/0x294
<1>[189197.542140] Register r4 information: non-paged memory
<1>[189197.553418] Register r5 information: non-paged memory
<1>[189197.558368] Register r6 information: non-paged memory
<1>[189197.563485] Register r7 information: non-paged memory
<1>[189197.568607] Register r8 information: non-slab/vmalloc memory
<1>[189197.573730] Register r9 information: NULL pointer
<1>[189197.579543] Register r10 information: slab kmalloc-256 start c179c000 pointer offset 24 size 256
<1>[189197.584240] Register r11 information: slab kmalloc-128 start c1cf8800 pointer offset 0 size 128
<1>[189197.593180] Register r12 information: NULL pointer
<0>[189197.602030] Process kworker/1:1 (pid: 20218, stack limit = 0x43738868)
<0>[189197.606631] Stack: (0xcb243d48 to 0xcb244000)
<0>[189197.613232] 3d40: c179bb40 00000001 20000013 c06ad978 c179bb58 00000001
<0>[189197.617752] 3d60: ffffffff c06ad9f8 c179bb64 c0d8082c ffffffff c06af14c 00000000 c0d8082c
<0>[189197.625998] 3d80: ffffffff c0342688 c179c000 c0d8082c c15e0e40 00000002 66d16940 23c34600
<0>[189197.634244] 3da0: 000927c0 c069c430 c158b800 c15aff00 66d16940 23c34600 c158b800 23c34600
<0>[189197.642491] 3dc0: c15e0e40 00000000 c15a5180 c06a0a54 c15e0d28 c15a5180 c06adea4 c14e3300
<0>[189197.650737] 3de0: 66d16940 23c34600 000927c0 c06a0a98 c15e0e40 00000000 23c34600 c15a5180
<0>[189197.658986] 3e00: dd99b010 23c34600 000927c0 c06a0cc4 23c34600 23c34600 00000000 ffffffff
<0>[189197.667231] 3e20: 23c34600 c0d82ad8 c1eeb780 c1eeb3c0 23c34600 00000001 c1cf8b80 dd99b010
<0>[189197.675481] 3e40: 23c34600 c06a0e4 c1610200 c1cf8980 00000001 c1cf8b80 dd99b010 c07d4d24
<0>[189197.683724] 3e60: c15e0e40 c1cf8b80 c1cf8f00 23c34600 00000000 c1610200 dd99b010 00000006
<0>[189197.691969] 3e80: 23c34600 c1cf8980 00000000 000927c0 00000000 c07d50d4 00000000 c1618200
<0>[189197.700216] 3ea0: c0dcff44 23c34600 c1618200 c0dcff44 00000000 00000001 001a524 c07d98ec
<0>[189197.708464] 3ec0: c1618200 001a524 000927c0 00000024 dd99e480 c1618200 c1cf8c80 c1cf8780
<0>[189197.716712] 3ee0: c1cf8c80 c1d32600 c1cf8780 dd9a1305 c2d01cc0 c07dcb54 c1cf8cb8 00000000
<0>[189197.724958] 3f00: c1cf8c84 c0d90710 00000000 00000040 dd9a1305 c07dd830 c1cf8cb8 c2d01c80
<0>[189197.733202] 3f20: dd99e040 dd9a1300 00000000 c0338d78 cb242000 dd99e040 00000008 c2d01c80
<0>[189197.741449] 3f40: c2d01c98 dd99e040 00000008 dd99e058 c0d03d00 dd99e200 cb242000 c0339078
<0>[189197.749698] 3f60: c0d0c36c c0d9aadc c56b7ecc c5f943c0 c2d42dc0 c0339004 c2d01c80 cb242000
<0>[189197.757942] 3f80: c56b7ecc c2d42de0 00000000 c0340d44 c5f943c0 c0340bec 00000000 00000000
<0>[189197.766188] 3fa0: 00000000 00000000 00000000 c0300130 00000000 00000000 00000000 00000000
<0>[189197.774435] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[189197.782681] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[189197.790927] [] (__timer_delay) from [] (__krait_mux_set_sel+0x7c/0x9c)
<0>[189197.799172] [] (__krait_mux_set_sel) from [] (krait_mux_set_parent+0x60/0x64)
<0>[189197.807416] [] (krait_mux_set_parent) from [] (krait_notifier_cb+0x58/0xb8)
<0>[189197.816445] [] (krait_notifier_cb) from [] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[189197.825388] [] (srcu_notifier_call_chain) from [] (__clk_notify+0x70/0x94)
<0>[189197.834501] [] (__clk_notify) from [] (clk_change_rate+0xfc/0x2b8)
<0>[189197.842919] [] (clk_change_rate) from [] (clk_change_rate+0x140/0x2b8)
<0>[189197.850905] [] (clk_change_rate) from [] (clk_core_set_rate_nolock+0xb4/0x1f8)
<0>[189197.859240] [] (clk_core_set_rate_nolock) from [] (clk_set_rate+0x3c/0x170)
<0>[189197.868266] [] (clk_set_rate) from [] (_set_opp+0x250/0x518)
<0>[189197.877297] [] (_set_opp) from [] (dev_pm_opp_set_rate+0xe8/0x218)
<0>[189197.884673] [] (dev_pm_opp_set_rate) from [] (__cpufreq_driver_target+0x194/0x238)
<0>[189197.892487] [] (__cpufreq_driver_target) from [] (od_dbs_update+0xcc/0x1a0)
<0>[189197.901863] [] (od_dbs_update) from [] (dbs_work_handler+0x38/0x74)
<0>[189197.910889] [] (dbs_work_handler) from [] (process_one_work+0x230/0x4bc)
<0>[189197.918962] [] (process_one_work) from [] (worker_thread+0x74/0x5d4)
<0>[189197.927380] [] (worker_thread) from [] (kthread+0x158/0x174)
<0>[189197.935540] [] (kthread) from [] (ret_from_fork+0x14/0x24)
<0>[189197.943004] Exception stack(0xcb243fb0 to 0xcb243ff8)
<0>[189197.950122] 3fa0: 00000000 00000000 00000000 00000000
<0>[189197.955338] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[189197.963581] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[189197.971832] Code: 0a000012 e5933000 e12fff33 e1a06000 (e5943000)
<4>[189197.978687] ---[ end trace 5961ab30030b334a ]---
<0>[189198.005521] Kernel panic - not syncing: Fatal exception
<2>[189198.005565] CPU0: stopping
<4>[189198.009903] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D 5.15.72 #0
<4>[189198.012423] Hardware name: Generic DT based system
<4>[189198.020063] [] (unwind_backtrace) from [] (show_stack+0x14/0x20)
<4>[189198.024753] [] (show_stack) from [] (dump_stack_lvl+0x40/0x4c)
<4>[189198.032738] [] (dump_stack_lvl) from [] (do_handle_IPI+0x144/0x184)
<4>[189198.040203] [] (do_handle_IPI) from [] (ipi_handler+0x1c/0x2c)
<4>[189198.048535] [] (ipi_handler) from [] (handle_percpu_devid_irq+0x84/0x16c)
<4>[189198.055917] [] (handle_percpu_devid_irq) from [] (handle_domain_irq+0x68/0x94)
<4>[189198.064598] [] (handle_domain_irq) from [] (gic_handle_irq+0x8c/0xb4)
<4>[189198.073536] [] (gic_handle_irq) from [] (__irq_svc+0x5c/0x78)
<4>[189198.081866] Exception stack(0xc0d01ed0 to 0xc0d01f18)
<4>[189198.089420] 1ec0: 00000000 0000ac13 1cd4c000 dd992440
<4>[189198.094548] 1ee0: 00000000 17bb7980 c1d09040 00000000 dd9916f0 0000ac13 00000000 0000ac13
<4>[189198.102793] 1f00: d2e01ba0 c0d01f20 c07e08b0 c07e08d0 60000013 ffffffff
<4>[189198.111034] [] (__irq_svc) from [] (cpuidle_enter_state+0x180/0x37c)
<4>[189198.117896] [] (cpuidle_enter_state) from [] (cpuidle_enter+0x3c/0x5c)
<4>[189198.125967] [] (cpuidle_enter) from [] (do_idle+0x204/0x2a4)
<4>[189198.134128] [] (do_idle) from [] (cpu_startup_entry+0x1c/0x20)
<4>[189198.141764] [] (cpu_startup_entry) from [] (start_kernel+0x6ac/0x6c0)

tapper82 · 2022-11-07T19:30:30Z

Thanks @motolav commented

tapper82 · 2022-11-07T19:33:05Z

BTW I am not running this patch on my r7800.

Ansuel · 2022-11-07T19:34:42Z

@hnyman random question can you disable devfreq and the krait cache driver in the config and try an image?
(the device should run slow but i'm curious to understand if the problem is in the notifier or in the cache scaling specifically)

hnyman · 2022-11-07T19:54:19Z

Can you give exact changes to be made? Dts or kernel config?

Ansuel · 2022-11-07T19:56:24Z

just kernel config...

setting the 2 flag as not set.

# CONFIG_ARM_KRAIT_CACHE_DEVFREQ  is not set
# CONFIG_ARM_IPQ806X_FAB_DEVFREQ  is not set

motolav · 2022-11-09T19:03:07Z

My last crash might be relevant to these changes?

<1>[ 9641.304557] 8<--- cut here ---
<1>[ 9641.304598] Unable to handle kernel NULL pointer dereference at virtual address 000000c0
<1>[ 9641.306522] pgd = 11c41bae
<1>[ 9641.314759] [000000c0] *pgd=00000000
<0>[ 9641.317282] Internal error: Oops: 805 [#1] SMP ARM
<4>[ 9641.321014] Modules linked in: pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath10k_pci ath10k_core ath pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 cfg80211 slhc nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc_ccitt compat ledtrig_usbport seqiv cmac leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod scsi_common ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug crc32c_generic
<4>[ 9641.380726] CPU: 0 PID: 9408 Comm: kworker/0:1 Not tainted 5.15.74 #0
<4>[ 9641.402959] Hardware name: Generic DT based system
<4>[ 9641.409295] Workqueue: events dbs_work_handler
<4>[ 9641.413980] PC is at krait_mux_set_parent+0xb0/0x11c
<4>[ 9641.418408] LR is at krait_mux_set_parent+0xac/0x11c
<4>[ 9641.423529] pc : [<c06ada10>]    lr : [<c06ada0c>]    psr: 60000093
<4>[ 9641.428479] sp : c7fafd38  ip : 00000000  fp : c17b8a54
<4>[ 9641.434467] r10: 23c34600  r9 : 00000000  r8 : 16e36000
<4>[ 9641.439674] r7 : 20000093  r6 : 00000101  r5 : 00000001  r4 : c158005c
<4>[ 9641.444886] r3 : 000000c0  r2 : c0d9a33c  r1 : 20000093  r0 : c0d9a33c
<4>[ 9641.451485] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[ 9641.457996] Control: 10c5787d  Table: 451e406a  DAC: 00000051
<1>[ 9641.465197] Register r0 information: non-slab/vmalloc memory
<1>[ 9641.471013] Register r1 information: non-paged memory
<1>[ 9641.476742] Register r2 information: non-slab/vmalloc memory
<1>[ 9641.481691] Register r3 information: non-paged memory
<1>[ 9641.487418] Register r4 information: slab kmalloc-128 start c1580000 pointer offset 92 size 128
<1>[ 9641.492371] Register r5 information: non-paged memory
<1>[ 9641.500872] Register r6 information: non-paged memory
<1>[ 9641.506082] Register r7 information: non-paged memory
<1>[ 9641.511117] Register r8 information: non-paged memory
<1>[ 9641.516151] Register r9 information: NULL pointer
<1>[ 9641.521185] Register r10 information: non-paged memory
<1>[ 9641.525873] Register r11 information: slab kmalloc-128 start c17b8a00 pointer offset 84 size 128
<1>[ 9641.530914] Register r12 information: NULL pointer
<0>[ 9641.539848] Process kworker/0:1 (pid: 9408, stack limit = 0xa636cda8)
<0>[ 9641.544452] Stack: (0xc7fafd38 to 0xc7fb0000)
<0>[ 9641.550963] fd20:                                                       23c34600 c15e1f00
<0>[ 9641.555311] fd40: c15e1e40 c15a7000 16e36000 c06a0bf8 c15e1f00 00000000 23c34600 c15a7000
<0>[ 9641.563471] fd60: c17b8a40 c0d04f1c 23c34600 c06a0cf0 c0b6e248 23c34600 00000000 ffffffff
<0>[ 9641.571632] fd80: 23c34600 c0d829b0 c31dc200 c1530cc0 23c34600 c1580140 c0d04fe0 c17b8a40
<0>[ 9641.579792] fda0: c0d04f1c c06a0e70 00000002 c17bd1c0 c1580140 c0d04fe0 c17b8a40 c06aef78
<0>[ 9641.587951] fdc0: c17bb000 a0000013 00000000 00000000 c17bb000 c06aeea8 00000000 53724e00
<0>[ 9641.596112] fde0: 23c34600 000927c0 c1515a00 c06a0bdc c17bb000 00000000 23c34600 c17bb000
<0>[ 9641.604272] fe00: ddb8e010 23c34600 000927c0 c06a0cf0 bd97c5c7 23c34600 00000000 ffffffff
<0>[ 9641.612429] fe20: 00000000 00000000 00000000 c1517240 23c34600 00000001 c1515d80 ddb8e010
<0>[ 9641.620590] fe40: 23c34600 c06a0e70 c1d28400 c1515b80 00000001 c1515d80 ddb8e010 c07d5f1c
<0>[ 9641.628749] fe60: c17bb000 c1515d80 c7fafec8 23c34600 00000000 c1d28400 ddb8e010 00000006
<0>[ 9641.636908] fe80: 23c34600 c1515b80 00000000 000927c0 00000000 c07d62cc c0d903c8 c0342688
<0>[ 9641.645068] fea0: c1d28e00 23c34600 c1d28e00 c0dcff84 00000000 00000001 00155cc0 c07daae4
<0>[ 9641.653228] fec0: c1d28e00 00155cc0 000927c0 00000024 ddb91440 c1d28e00 c1518180 c1518200
<0>[ 9641.661388] fee0: c1518180 c1517640 c1518200 ddb94305 c482a240 c07ddd4c c15181b8 00000000
<0>[ 9641.669548] ff00: c1518184 c0d906ac 00000000 00000000 ddb94305 c07dea28 c15181b8 c482a200
<0>[ 9641.677708] ff20: ddb91040 ddb94300 00000000 c0338d78 c7fae000 ddb91040 00000008 c482a200
<0>[ 9641.685867] ff40: c482a218 ddb91040 00000008 ddb91058 c0d03d00 ddb91200 c7fae000 c0339078
<0>[ 9641.694027] ff60: c0d0c36c c0d9aadc c7333ecc c15deb80 c369ca40 c0339004 c482a200 c7fae000
<0>[ 9641.702189] ff80: c7333ecc c369ca60 00000000 c0340d44 c15deb80 c0340bec 00000000 00000000
<0>[ 9641.710346] ffa0: 00000000 00000000 00000000 c0300130 00000000 00000000 00000000 00000000
<0>[ 9641.718505] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[ 9641.726664] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[ 9641.734825] [<c06ada10>] (krait_mux_set_parent) from [<c06a0bf8>] (clk_change_rate+0x274/0x2b8)
<0>[ 9641.742983] [<c06a0bf8>] (clk_change_rate) from [<c06a0cf0>] (clk_core_set_rate_nolock+0xb4/0x1f8)
<0>[ 9641.751489] [<c06a0cf0>] (clk_core_set_rate_nolock) from [<c06a0e70>] (clk_set_rate+0x3c/0x170)
<0>[ 9641.760516] [<c06a0e70>] (clk_set_rate) from [<c06aef78>] (mux_cc_set_rate+0xd0/0x164)
<0>[ 9641.769110] [<c06aef78>] (mux_cc_set_rate) from [<c06a0bdc>] (clk_change_rate+0x258/0x2b8)
<0>[ 9641.777096] [<c06a0bdc>] (clk_change_rate) from [<c06a0cf0>] (clk_core_set_rate_nolock+0xb4/0x1f8)
<0>[ 9641.785343] [<c06a0cf0>] (clk_core_set_rate_nolock) from [<c06a0e70>] (clk_set_rate+0x3c/0x170)
<0>[ 9641.794283] [<c06a0e70>] (clk_set_rate) from [<c07d5f1c>] (_set_opp+0x250/0x518)
<0>[ 9641.802877] [<c07d5f1c>] (_set_opp) from [<c07d62cc>] (dev_pm_opp_set_rate+0xe8/0x218)
<0>[ 9641.810516] [<c07d62cc>] (dev_pm_opp_set_rate) from [<c07daae4>] (__cpufreq_driver_target+0x194/0x238)
<0>[ 9641.818244] [<c07daae4>] (__cpufreq_driver_target) from [<c07ddd4c>] (od_dbs_update+0xcc/0x1a0)
<0>[ 9641.827531] [<c07ddd4c>] (od_dbs_update) from [<c07dea28>] (dbs_work_handler+0x38/0x74)
<0>[ 9641.836122] [<c07dea28>] (dbs_work_handler) from [<c0338d78>] (process_one_work+0x230/0x4bc)
<0>[ 9641.844113] [<c0338d78>] (process_one_work) from [<c0339078>] (worker_thread+0x74/0x5d4)
<0>[ 9641.852791] [<c0339078>] (worker_thread) from [<c0340d44>] (kthread+0x158/0x174)
<0>[ 9641.860863] [<c0340d44>] (kthread) from [<c0300130>] (ret_from_fork+0x14/0x24)
<0>[ 9641.868238] Exception stack(0xc7faffb0 to 0xc7fafff8)
<0>[ 9641.875270] ffa0:                                     00000000 00000000 00000000 00000000
<0>[ 9641.880397] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[ 9641.888557] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[ 9641.896719] Code: e1a01006 e5140018 ebf19d45 e5543004 (e3530000) 
<4>[ 9641.903137] ---[ end trace 774be9aac936b193 ]---

motolav · 2022-11-11T16:57:06Z

Had another crash overnight but with the same PC and LR as tapper82

<4>[84015.328503] PC is at __timer_delay+0x38/0x78
<4>[84015.332928] LR is at msm_read_current_timer+0x18/0x24

hnyman · 2022-11-25T14:36:58Z

@Ansuel
Any roadmap?

I have not seen a crash since I applied the "devfreq" disablement patch we discussed privately

--- a/target/linux/ipq806x/config-5.15
+++ b/target/linux/ipq806x/config-5.15
@@ -127,8 +127,8 @@ CONFIG_DEVFREQ_GOV_PASSIVE=y
 # CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND is not set
 # CONFIG_DEVFREQ_GOV_USERSPACE is not set
 # CONFIG_DEVFREQ_THERMAL is not set
-CONFIG_ARM_KRAIT_CACHE_DEVFREQ=y
-CONFIG_ARM_IPQ806X_FAB_DEVFREQ=y
+# CONFIG_ARM_KRAIT_CACHE_DEVFREQ is not set
+# CONFIG_ARM_IPQ806X_FAB_DEVFREQ is not set
 CONFIG_DMADEVICES=y
 CONFIG_DMA_ENGINE=y
 CONFIG_DMA_OF=y

Ansuel · 2022-11-25T14:40:02Z

@hnyman i wonder if temporarily while i still search the cause of the wrong freq with messing with the original fw we can consider disabling the devfreq drivers? Will result in worse perf but at least better stability.

Just to make sure you only have the devfreq patch right? not the changes to krait-cc right ?

hnyman · 2022-11-25T14:52:06Z

I would restore stability first, and think about performance afterwards.

Just to make sure you only have the devfreq patch right? not the changes to krait-cc right ?

Yes.
I first tested

with the patch from this PR: some crashes,
then this PR + devfreq disablement: no crashes,
finally/currently just the devfreq disablement (above lines): no crashes since 11.11.2022

At least with my personal internet usage with a 200/100 connection, I see no actual performance hit in speedtests with SQM limits at 190/85:
https://www.waveform.com/tools/bufferbloat?test-id=9d4689b2-2acc-452a-bbd9-5c290e72e596
Download 178.4 Mbps / ping 4.0 ms
Upload 83.9 Mbps / ping 4.3 ms

Ansuel · 2022-11-25T14:56:04Z

Considering the minimal change it looks sane to just disable the config. And this would confirm that the main problem on this platform is something badly configured around the cache clk

Ansuel · 2022-11-25T23:46:34Z

@hnyman well i disabled the driver for now hoping i finally find the real culprit of all this mess...

TEST ipq806x: add test patch to improve stability

c243d3d

Add test patch to improve stability. Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

Ansuel added the RFT pull request ready for testing label Nov 5, 2022

github-actions bot added the target/ipq806x pull request/issue for ipq806x target label Nov 5, 2022

Ansuel closed this Nov 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEST ipq806x: add test patch to improve stability #11173

TEST ipq806x: add test patch to improve stability #11173

Ansuel commented Nov 5, 2022 •

edited

Ansuel commented Nov 5, 2022

hnyman commented Nov 5, 2022

motolav commented Nov 6, 2022

hnyman commented Nov 6, 2022

Ansuel commented Nov 6, 2022

hnyman commented Nov 6, 2022 •

edited

motolav commented Nov 6, 2022

hnyman commented Nov 6, 2022

motolav commented Nov 6, 2022

hnyman commented Nov 7, 2022

motolav commented Nov 7, 2022 •

edited

tapper82 commented Nov 7, 2022

motolav commented Nov 7, 2022

tapper82 commented Nov 7, 2022

tapper82 commented Nov 7, 2022

tapper82 commented Nov 7, 2022

tapper82 commented Nov 7, 2022

Ansuel commented Nov 7, 2022

hnyman commented Nov 7, 2022

Ansuel commented Nov 7, 2022

motolav commented Nov 9, 2022 •

edited

motolav commented Nov 11, 2022

hnyman commented Nov 25, 2022 •

edited

Ansuel commented Nov 25, 2022

hnyman commented Nov 25, 2022

Ansuel commented Nov 25, 2022

Ansuel commented Nov 25, 2022

TEST ipq806x: add test patch to improve stability #11173

TEST ipq806x: add test patch to improve stability #11173

Conversation

Ansuel commented Nov 5, 2022 • edited

Ansuel commented Nov 5, 2022

hnyman commented Nov 5, 2022

motolav commented Nov 6, 2022

hnyman commented Nov 6, 2022

Ansuel commented Nov 6, 2022

hnyman commented Nov 6, 2022 • edited

motolav commented Nov 6, 2022

hnyman commented Nov 6, 2022

motolav commented Nov 6, 2022

hnyman commented Nov 7, 2022

motolav commented Nov 7, 2022 • edited

tapper82 commented Nov 7, 2022

motolav commented Nov 7, 2022

tapper82 commented Nov 7, 2022

tapper82 commented Nov 7, 2022

tapper82 commented Nov 7, 2022

tapper82 commented Nov 7, 2022

Ansuel commented Nov 7, 2022

hnyman commented Nov 7, 2022

Ansuel commented Nov 7, 2022

motolav commented Nov 9, 2022 • edited

motolav commented Nov 11, 2022

hnyman commented Nov 25, 2022 • edited

Ansuel commented Nov 25, 2022

hnyman commented Nov 25, 2022

Ansuel commented Nov 25, 2022

Ansuel commented Nov 25, 2022

Ansuel commented Nov 5, 2022 •

edited

hnyman commented Nov 6, 2022 •

edited

motolav commented Nov 7, 2022 •

edited

motolav commented Nov 9, 2022 •

edited

hnyman commented Nov 25, 2022 •

edited