Receiving spinlocks KP randomly

[stevefan1999-personal]

Just a few minutes ago I saw this from my byobu remote session:

e>/-1, .owner_cpu: 152518656                                               [0/0]                                                                               
Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66559.853169] BUG: spinlock lockup suspected on CPU#3, swapper/3/0    
                                                                               
Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66559.857273]  lock: 0xffffffc0f5e27e10, .magic: 00000040, .owner: <non
e>/-1, .owner_cpu: 152518656                                                   
Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66552.808519] BUG: spinlock bad magic on CPU#3, swapper/3/0           
                                                                               
Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66552.809119]  lock: 0xffffffc0f5e27e10, .magic: 00000040, .owner: <non
e>/-1, .owner_cpu: 152518656                                                   

Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66559.853169] BUG: spinlock lockup suspected on CPU#3, swapper/3/0    
                                                                               
Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66559.857273]  lock: 0xffffffc0f5e27e10, .magic: 00000040, .owner: <non
e>/-1, .owner_cpu: 152518656                                                   

And then my R64 just reached a complete lockdown. Have no idea what happened, what caused this.

[GuyNoIRQ]

I've been hitting similar on a RockPro64 running Open Media Vault 0.8.3-1141. Every time I try to initialize a raid1 volume I hit this after just a couple minutes.

RockPro_Raid_Init_kernlog_Crash.log

                _                     __   _  _   
 _ __ ___   ___| | ___ __  _ __ ___  / /_ | || |  
| '__/ _ \ / __| |/ / '_ \| '__/ _ \| '_ \| || |_ 
| | | (_) | (__|   <| |_) | | | (_) | (_) |__   _|
|_|  \___/ \___|_|\_\ .__/|_|  \___/ \___/   |_|  
                    |_|                           
Linux rockpro64 4.4.167-1213-rockchip-ayufan-g34ae07687fce #1 SMP Tue Jun 18 20:44:49 UTC 2019 aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@rockpro64:~# mdadm --readwrite /dev/md0
root@rockpro64:~# tail -f /var/log/kern.log 
Jul  6 23:17:08 rockpro64 kernel: [   18.870817] cdn-dp fec00000.dp: Direct firmware load for rockchip/dptx.bin failed with error -2
Jul  6 23:17:14 rockpro64 kernel: [   25.226971] rk_gmac-dwmac fe300000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
Jul  6 23:17:18 rockpro64 kernel: [   29.324141] NFSD: starting 90-second grace period (net ffffff80093a2f40)
Jul  6 23:17:24 rockpro64 kernel: [   34.886722] cdn-dp fec00000.dp: Direct firmware load for rockchip/dptx.bin failed with error -2
Jul  7 00:24:13 rockpro64 kernel: [   62.621049] md: md0 switched to read-write mode.
Jul  7 00:24:13 rockpro64 kernel: [   62.639974] md: resync of RAID array md0
Jul  7 00:24:13 rockpro64 kernel: [   62.640460] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jul  7 00:24:13 rockpro64 kernel: [   62.641129] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
Jul  7 00:24:13 rockpro64 kernel: [   62.642201] md: using 128k window, over a total of 7813895488k.
Jul  7 00:24:17 rockpro64 kernel: [   66.891480] cdn-dp fec00000.dp: Direct firmware load for rockchip/dptx.bin failed with error -2
Jul  7 00:25:21 rockpro64 kernel: [  130.896639] cdn-dp fec00000.dp: [drm:cdn_dp_request_firmware] *ERROR* Timed out trying to load firmware

Message from syslogd@rockpro64 at Jul  7 00:27:08 ...
 kernel:[  233.001921] Internal error: : 96000210 [#1] SMP
Jul  7 00:27:08 rockpro64 kernel: [  233.001042] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80096c61b0
Jul  7 00:27:08 rockpro64 kernel: [  233.001921] Internal error: : 96000210 [#1] SMP
Jul  7 00:27:08 rockpro64 kernel: [  233.002397] Modules linked in: midgard_kbase dw_hdmi_i2s_audio rockchip_saradc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid0 multipath linear raid1 md_mod
Jul  7 00:27:08 rockpro64 kernel: [  233.004571] CPU: 0 PID: 143 Comm: scsi_eh_1 Not tainted 4.4.167-1213-rockchip-ayufan-g34ae07687fce #1
Jul  7 00:27:08 rockpro64 kernel: [  233.005524] Hardware name: Pine64 RockPro64 (DT)
Jul  7 00:27:08 rockpro64 kernel: [  233.006007] task: ffffffc0efcb8000 task.stack: ffffffc0efcb4000
Jul  7 00:27:08 rockpro64 kernel: [  233.006631] PC is at ahci_scr_read+0x70/0x98
Jul  7 00:27:08 rockpro64 kernel: [  233.007081] LR is at sata_scr_read+0x88/0x94
Jul  7 00:27:08 rockpro64 kernel: [  233.007528] pc : [<ffffff800877cb8c>] lr : [<ffffff80087634a4>] pstate: 80000145
Jul  7 00:27:08 rockpro64 kernel: [  233.008292] sp : ffffffc0efcb7a50
Jul  7 00:27:08 rockpro64 kernel: [  233.008640] x29: ffffffc0efcb7a50 x28: 0000000000000000 
Jul  7 00:27:08 rockpro64 kernel: [  233.009211] x27: ffffffc0efc49e80 x26: 0000000000000001 
Jul  7 00:27:08 rockpro64 kernel: [  233.009781] x25: ffffffc0efc4a258 x24: ffffffc0efc48000 
Jul  7 00:27:08 rockpro64 kernel: [  233.010352] x23: 0000000000000000 x22: ffffffc0efcb7b28 
Jul  7 00:27:08 rockpro64 kernel: [  233.010923] x21: ffffffc0efc49e80 x20: ffffffc0efcb7b28 
Jul  7 00:27:08 rockpro64 kernel: [  233.011493] x19: 0000000000000001 x18: 0000000030d00800 
Jul  7 00:27:08 rockpro64 kernel: [  233.012063] x17: 0000000000000000 x16: 0000000000000000 
Jul  7 00:27:08 rockpro64 kernel: [  233.012633] x15: 0000000000000000 x14: 0000000000000000 
Jul  7 00:27:08 rockpro64 kernel: [  233.013203] x13: 0000000000000000 x12: 0000000000000000 
Jul  7 00:27:08 rockpro64 kernel: [  233.013772] x11: 0000000000000000 x10: 0000000000000000 
Jul  7 00:27:08 rockpro64 kernel: [  233.014343] x9 : 0000000000000000 x8 : ffffffc0efc4a258 
Jul  7 00:27:08 rockpro64 kernel: [  233.014914] x7 : 0000000000000000 x6 : ffffffc0efc4a2d0 
Jul  7 00:27:08 rockpro64 kernel: [  233.018973] x5 : 0000000000000000 x4 : ffffff8008d9af94 
Jul  7 00:27:08 rockpro64 kernel: [  233.022971] x3 : 00000000000001b0 x2 : ffffffc0efcb7b28 
Jul  7 00:27:08 rockpro64 kernel: [  233.026914] x1 : 0000000000000030 x0 : ffffff80096c61b0 
Jul  7 00:27:08 rockpro64 kernel: [  233.030806] 
Jul  7 00:27:08 rockpro64 kernel: [  233.030806] PC: 0xffffff800877cb0c:
Jul  7 00:27:08 rockpro64 kernel: [  233.037877] cb0c  52800013 912c1821 97fe23cd 17ffffe2 a9bd7bfd 910003fd a90153f3 f90013f5
Jul  7 00:27:08 rockpro64 kernel: [  233.042148] cb2c  aa0003f5 2a0103f3 aa1e03e0 aa0203f4 d503201f f94002a3 7100127f f959dc60
Jul  7 00:27:08 rockpro64 kernel: [  233.046485] cb4c  b9402c63 f9401801 53196063 91040063 f9400820 54000268 54000061 b9401821
Jul  7 00:27:08 rockpro64 kernel: [  233.050838] cb6c  36e80201 d00030e4 913de084 91007084 b8735881 34000161 8b21c063 8b030000
Jul  7 00:27:08 rockpro64 kernel: [  233.055199] cb8c  b9400000 d5033d9f b9000280 52800000 a94153f3 f94013f5 a8c37bfd d65f03c0
Jul  7 00:27:08 rockpro64 kernel: [  233.059567] cbac  128002a0 17fffffb a9be7bfd 910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f
Jul  7 00:27:08 rockpro64 kernel: [  233.063939] cbcc  f9400260 f9400000 f959dc00 f9401800 b9400000 36400120 52801fe0 b9038660
Jul  7 00:27:08 rockpro64 kernel: [  233.068322] cbec  f0004482 90004061 912c8842 911dbc21 aa1303e0 97ff9d72 f9400bf3 a8c27bfd
Jul  7 00:27:08 rockpro64 kernel: [  233.072786] 
Jul  7 00:27:08 rockpro64 kernel: [  233.072786] LR: 0xffffff8008763424:
Jul  7 00:27:08 rockpro64 kernel: [  233.080343] 3424  a90153f3 a9025bf5 aa0003f3 2a0103f5 aa1e03e0 aa0203f6 d503201f f9400274
Jul  7 00:27:08 rockpro64 kernel: [  233.084836] 3444  d283d000 8b000280 eb00027f 54000180 f959c280 eb00027f 54000120 aa1603e2
Jul  7 00:27:08 rockpro64 kernel: [  233.089268] 3464  2a1503e1 aa1303e0 9400572a a94153f3 a9425bf5 a8c37bfd d65f03c0 aa1303e0
Jul  7 00:27:08 rockpro64 kernel: [  233.093647] 3484  97fffb19 34000100 f9400680 aa1603e2 2a1503e1 f9406c03 aa1303e0 d63f0060
Jul  7 00:27:08 rockpro64 kernel: [  233.098065] 34a4  17fffff3 12800bc0 17fffff1 a9ba7bfd 910003fd a90153f3 a9025bf5 a90363f7
Jul  7 00:27:08 rockpro64 kernel: [  233.102501] 34c4  a9046bf9 aa0103f5 aa0203f6 aa0003f7 aa1e03e0 d503201f d0005813 b94012a0
Jul  7 00:27:08 rockpro64 kernel: [  233.106912] 34e4  f940e274 a94066b8 97e6e656 8b000280 cb160001 910173a2 f100003f 52800001
Jul  7 00:27:08 rockpro64 kernel: [  233.111301] 3504  9a80a2d6 aa1703e0 97ffffc4 2a0003f5 350003c0 b9405fb4 f940e27a 12000e94
Jul  7 00:27:08 rockpro64 kernel: [  233.115658] 
Jul  7 00:27:08 rockpro64 kernel: [  233.115658] SP: 0xffffffc0efcb79d0:
Jul  7 00:27:08 rockpro64 kernel: [  233.122908] 79d0  efcb7b28 ffffffc0 00000000 00000000 efc48000 ffffffc0 efc4a258 ffffffc0
Jul  7 00:27:08 rockpro64 kernel: [  233.127166] 79f0  00000001 00000000 efc49e80 ffffffc0 00000000 00000000 efcb7a50 ffffffc0
Jul  7 00:27:08 rockpro64 kernel: [  233.131340] 7a10  087634a4 ffffff80 efcb7a50 ffffffc0 0877cb8c ffffff80 80000145 00000000
Jul  7 00:27:08 rockpro64 kernel: [  233.135428] 7a30  08cbe644 ffffff80 f2bb60a0 ffffffc0 ffffffff ffffffff 00000000 00000000
Jul  7 00:27:08 rockpro64 kernel: [  233.139426] 7a50  efcb7a80 ffffffc0 087634a4 ffffff80 efc49e80 ffffffc0 efc48000 ffffffc0
Jul  7 00:27:08 rockpro64 kernel: [  233.143368] 7a70  00000001 00000000 00000000 00000000 efcb7ab0 ffffffc0 08770374 ffffff80
Jul  7 00:27:08 rockpro64 kernel: [  233.147307] 7a90  efc4a258 ffffffc0 efc49e80 ffffffc0 efc4b858 ffffffc0 00003008 00000000
Jul  7 00:27:08 rockpro64 kernel: [  233.151252] 7ab0  efcb7b60 ffffffc0 08770b34 ffffff80 efc48000 ffffffc0 efc49e80 ffffffc0
Jul  7 00:27:08 rockpro64 kernel: [  233.155230] 
Jul  7 00:27:08 rockpro64 kernel: [  233.155230] X0: 0xffffff80096c6130:
Jul  7 00:27:08 rockpro64 kernel: [  233.161788] 6130 
Jul  7 00:27:08 rockpro64 kernel: [  233.211980] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80096c6130
Jul  7 00:27:08 rockpro64 kernel: [  237.987191] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

Message from syslogd@rockpro64 at Jul  7 00:27:47 ...
 kernel:[  240.218948] BUG: spinlock lockup suspected on CPU#0, scsi_eh_1/143

Message from syslogd@rockpro64 at Jul  7 00:27:47 ...
 kernel:[  240.222849]  lock: 0xffffff800926fdf0, .magic: dead4ead, .owner: scsi_eh_1/143, .owner_cpu: 0
Jul  7 00:27:47 rockpro64 kernel: [  240.218948] BUG: spinlock lockup suspected on CPU#0, scsi_eh_1/143
Jul  7 00:27:47 rockpro64 kernel: [  240.222849]  lock: 0xffffff800926fdf0, .magic: dead4ead, .owner: scsi_eh_1/143, .owner_cpu: 0
Jul  7 00:27:47 rockpro64 kernel: [  240.227026] CPU: 0 PID: 143 Comm: scsi_eh_1 Not tainted 4.4.167-1213-rockchip-ayufan-g34ae07687fce #1
Jul  7 00:27:47 rockpro64 kernel: [  240.231336] Hardware name: Pine64 RockPro64 (DT)
Jul  7 00:27:47 rockpro64 kernel: [  240.235282] Call trace:
Jul  7 00:27:47 rockpro64 kernel: [  240.239056] [<ffffff80080889b0>] dump_backtrace+0x0/0x220
Jul  7 00:27:47 rockpro64 kernel: [  240.243183] [<ffffff8008088bf4>] show_stack+0x24/0x30
Jul  7 00:27:47 rockpro64 kernel: [  240.247311] [<ffffff80085a2b2c>] dump_stack+0x98/0xc0
Jul  7 00:27:47 rockpro64 kernel: [  240.251429] [<ffffff8008104a24>] spin_dump+0x84/0xa4
Jul  7 00:27:47 rockpro64 kernel: [  240.255566] [<ffffff8008104bc0>] do_raw_spin_lock+0xdc/0x164
Jul  7 00:27:47 rockpro64 kernel: [  240.259802] [<ffffff8008c9862c>] _raw_spin_lock_irq+0x24/0x30
Jul  7 00:27:47 rockpro64 kernel: [  240.264071] [<ffffff8008088c38>] die+0x38/0x1ac
Jul  7 00:27:47 rockpro64 kernel: [  240.268241] [<ffffff8008088ea0>] arm64_notify_die+0x6c/0x70
Jul  7 00:27:47 rockpro64 kernel: [  240.272502] [<ffffff8008080b50>] do_mem_abort+0x9c/0xac
Jul  7 00:27:47 rockpro64 kernel: [  240.276743] Exception stack(0xffffffc0efcb7560 to 0xffffffc0efcb7690)
Jul  7 00:27:47 rockpro64 kernel: [  240.281086] 7560: ffffffc0efcb8000 0000008000000000 ffffffc0efcb7730 ffffff80085a0dc4
Jul  7 00:27:47 rockpro64 kernel: [  240.285566] 7580: 0000000000000007 0000000000000000 ffffff80096c6130 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.290013] 75a0: ffffff800941f2fc 0000000000000000 ffffffc0efcb75d0 ffffff800810b7f8
Jul  7 00:27:47 rockpro64 kernel: [  240.294437] 75c0: 0000000000000005 0000000000000000 ffffffc0efcb7670 ffffff800810b958
Jul  7 00:27:47 rockpro64 kernel: [  240.298807] 75e0: ffffffc0efcb76f0 ffffff8008f8816c ffffff8008f8816c ffffff8008fed582
Jul  7 00:27:47 rockpro64 kernel: [  240.303163] 7600: ffffffc0efcb77bc ffffff80096c6130 0000000000000004 0000000000000010
Jul  7 00:27:47 rockpro64 kernel: [  240.307501] 7620: 0000000000000000 ffffffc0efcb77c0 ffffffc0efcb77bc 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.311810] 7640: 0000000000001000 0000000000aaaaaa 0000000000000008 00000000ffffffff
Jul  7 00:27:47 rockpro64 kernel: [  240.316081] 7660: ffffffc0f29dbf10 ffffff800ba002a0 ffffff800ba002a0 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.320398] 7680: 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.324397] [<ffffff80080825cc>] el1_da+0x24/0x84
Jul  7 00:27:47 rockpro64 kernel: [  240.328343] [<ffffff8008085024>] show_data.constprop.3+0xa4/0xf8
Jul  7 00:27:47 rockpro64 kernel: [  240.332391] [<ffffff80080853b0>] __show_regs+0x154/0x1a8
Jul  7 00:27:47 rockpro64 kernel: [  240.336332] [<ffffff8008088cb4>] die+0xb4/0x1ac
Jul  7 00:27:47 rockpro64 kernel: [  240.340152] [<ffffff8008088ea0>] arm64_notify_die+0x6c/0x70
Jul  7 00:27:47 rockpro64 kernel: [  240.344030] [<ffffff8008080b50>] do_mem_abort+0x9c/0xac
Jul  7 00:27:47 rockpro64 kernel: [  240.347831] Exception stack(0xffffffc0efcb7880 to 0xffffffc0efcb79b0)
Jul  7 00:27:47 rockpro64 kernel: [  240.351752] 7880: 0000000000000001 0000008000000000 ffffffc0efcb7a50 ffffff800877cb8c
Jul  7 00:27:47 rockpro64 kernel: [  240.355798] 78a0: 0000000000000007 0000000000000000 ffffff80096c61b0 ffffffc0efc4a258
Jul  7 00:27:47 rockpro64 kernel: [  240.359850] 78c0: 0000000000000001 ffffffc0efc49e80 ffffffc0efcb8000 ffffffc0efcb7a50
Jul  7 00:27:47 rockpro64 kernel: [  240.363846] 78e0: ffffff80087634a4 ffffffc0efcb7920 ffffff80080825b8 0000000060000145
Jul  7 00:27:47 rockpro64 kernel: [  240.367838] 7900: 0000000000000000 ffffffc0f298eb80 0000008000000000 ffffff80080e92b8
Jul  7 00:27:47 rockpro64 kernel: [  240.371828] 7920: ffffff80096c61b0 0000000000000030 ffffffc0efcb7b28 00000000000001b0
Jul  7 00:27:47 rockpro64 kernel: [  240.375860] 7940: ffffff8008d9af94 0000000000000000 ffffffc0efc4a2d0 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.379858] 7960: ffffffc0efc4a258 0000000000000000 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.383830] 7980: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.387754] 79a0: 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.391371] [<ffffff80080825cc>] el1_da+0x24/0x84
Jul  7 00:27:47 rockpro64 kernel: [  240.394929] [<ffffff80087634a4>] sata_scr_read+0x88/0x94
Jul  7 00:27:47 rockpro64 kernel: [  240.398563] [<ffffff8008770374>] ata_eh_link_autopsy+0x4c/0x740
Jul  7 00:27:47 rockpro64 kernel: [  240.402271] [<ffffff8008770b34>] ata_eh_autopsy+0xcc/0xdc
Jul  7 00:27:47 rockpro64 kernel: [  240.405862] [<ffffff8008778914>] sata_pmp_error_handler+0x34/0x7b8
Jul  7 00:27:47 rockpro64 kernel: [  240.409476] [<ffffff800877c4f8>] ahci_error_handler+0x44/0x70
Jul  7 00:27:47 rockpro64 kernel: [  240.413018] [<ffffff8008772f54>] ata_scsi_port_error_handler+0xe0/0x550
Jul  7 00:27:47 rockpro64 kernel: [  240.416597] [<ffffff8008773448>] ata_scsi_error+0x84/0xa4
Jul  7 00:27:47 rockpro64 kernel: [  240.419986] [<ffffff8008744610>] scsi_error_handler+0x130/0x6d4
Jul  7 00:27:47 rockpro64 kernel: [  240.423359] [<ffffff80080d54ac>] kthread+0xe0/0xf0
Jul  7 00:27:47 rockpro64 kernel: [  240.426620] [<ffffff8008082ef0>] ret_from_fork+0x10/0x20
Jul  7 00:27:47 rockpro64 kernel: [  276.141323] Watchdog detected hard LOCKUP on cpu 0
Jul  7 00:27:47 rockpro64 kernel: [  276.141898] ------------[ cut here ]------------
Jul  7 00:27:47 rockpro64 kernel: [  276.151042] WARNING: at kernel/watchdog.c:352
Jul  7 00:27:47 rockpro64 kernel: [  276.155862] Modules linked in: midgard_kbase dw_hdmi_i2s_audio rockchip_saradc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid0 multipath linear raid1 md_mod
Jul  7 00:27:47 rockpro64 kernel: [  276.167535] 
Jul  7 00:27:47 rockpro64 kernel: [  276.172480] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 4.4.167-1213-rockchip-ayufan-g34ae07687fce #1
Jul  7 00:27:47 rockpro64 kernel: [  276.178353] Hardware name: Pine64 RockPro64 (DT)
Jul  7 00:27:47 rockpro64 kernel: [  276.183844] task: ffffffc0f2af4600 task.stack: ffffffc0f2b0c000
Jul  7 00:27:47 rockpro64 kernel: [  276.189486] PC is at watchdog_timer_fn+0x190/0x2e8
Jul  7 00:27:47 rockpro64 kernel: [  276.194978] LR is at watchdog_timer_fn+0x190/0x2e8
Jul  7 00:27:47 rockpro64 kernel: [  276.200389] pc : [<ffffff800815631c>] lr : [<ffffff800815631c>] pstate: 600001c5
Jul  7 00:27:47 rockpro64 kernel: [  276.206136] sp : ffffffc0f7f43d00
Jul  7 00:27:47 rockpro64 kernel: [  276.211514] x29: ffffffc0f7f43d00 x28: 0000000000000000 
Jul  7 00:27:47 rockpro64 kernel: [  276.217097] x27: ffffff80092678a0 x26: ffffff800922b238 
Jul  7 00:27:47 rockpro64 kernel: [  276.222643] x25: ffffffc0f7f46250 x24: ffffffc0f2b0fda0 
Jul  7 00:27:47 rockpro64 kernel: [  276.228150] x23: 0000000000000000 x22: 00000000000000fd 
Jul  7 00:27:47 rockpro64 kernel: [  276.233638] x21: ffffff8009266000 x20: ffffff800922b2b0 
Jul  7 00:27:47 rockpro64 kernel: [  276.239072] x19: ffffff800922b000 x18: 0000000000000005 
Jul  7 00:27:47 rockpro64 kernel: [  276.244427] x17: 0000000000000000 x16: 0000000000000000 
Jul  7 00:27:47 rockpro64 kernel: [  276.249745] x15: 0000000000000000 x14: ffffff800ba006c0 
Jul  7 00:27:47 rockpro64 kernel: [  276.255025] x13: ffffff800ba006c0 x12: ffffffc0f29dac20 
Jul  7 00:27:47 rockpro64 kernel: [  276.260287] x11: 00000000ffffffff x10: 0000000000000008 
Jul  7 00:27:47 rockpro64 kernel: [  276.265567] x9 : 0000000000aaaaaa x8 : 0000000000001000 
Jul  7 00:27:47 rockpro64 kernel: [  276.270820] x7 : 0000000000000000 x6 : 0000000000000001 
Jul  7 00:27:47 rockpro64 kernel: [  276.276040] x5 : 0000000000000000 x4 : 0000000000000000 
Jul  7 00:27:47 rockpro64 kernel: [  276.281157] x3 : 00000040eed1b000 x2 : 00000040eed1b000 
Jul  7 00:27:47 rockpro64 kernel: [  276.286200] x1 : 0000000000000005 x0 : 0000000000000026 
Jul  7 00:27:47 rockpro64 kernel: [  276.291240] 
Jul  7 00:27:47 rockpro64 kernel: [  276.291240] PC: 0xffffff800815629c:
Jul  7 00:27:47 rockpro64 kernel: [  276.300567] 629c  91022694 d538d080 38746801 34000041 3834681f 97ffff7a a94153f3 52800020
Jul  7 00:27:47 rockpro64 kernel: [  276.306060] 62bc  a9425bf5 a94363f7 a9446bf9 a94573fb a8c77bfd d65f03c0 91020282 f87a6860
Jul  7 00:27:47 rockpro64 kernel: [  276.311564] 62dc  f8626865 91022294 eb05001f 54000101 38746860 35fffb80 b9403082 90007200
Jul  7 00:27:47 rockpro64 kernel: [  276.317155] 62fc  910de000 340000c2 9400dfb4 f8226860 f87c7b60 3834681f 17ffffd3 9400e03e
Jul  7 00:27:47 rockpro64 kernel: [  276.322821] 631c  d4210000 f87c7b60 52800021 38346801 17ffffcd 97fe4698 f9444ea1 36080941
Jul  7 00:27:47 rockpro64 kernel: [  276.328584] 633c  b9401341 910003e2 340008e1 531f7821 d35efc00 8b21c2c1 cb000021 b6f80841
Jul  7 00:27:47 rockpro64 kernel: [  276.334401] 635c  6b160016 54000800 91022a99 d538d080 38606b21 34000101 91024294 f8746802
Jul  7 00:27:47 rockpro64 kernel: [  276.340277] 637c  d5384101 eb01005f 54fff980 3839681f 17ffffc9 34000157 90009981 52800000
Jul  7 00:27:47 rockpro64 kernel: [  276.346226] 
Jul  7 00:27:47 rockpro64 kernel: [  276.346226] LR: 0xffffff800815629c:
Jul  7 00:27:47 rockpro64 kernel: [  276.356609] 629c  91022694 d538d080 38746801 34000041 3834681f 97ffff7a a94153f3 52800020
Jul  7 00:27:47 rockpro64 kernel: [  276.362696] 62bc  a9425bf5 a94363f7 a9446bf9 a94573fb a8c77bfd d65f03c0 91020282 f87a6860
Jul  7 00:27:47 rockpro64 kernel: [  276.368843] 62dc  f8626865 91022294 eb05001f 54000101 38746860 35fffb80 b9403082 90007200
Jul  7 00:27:47 rockpro64 kernel: [  276.375029] 62fc  910de000 340000c2 9400dfb4 f8226860 f87c7b60 3834681f 17ffffd3 9400e03e
Jul  7 00:27:47 rockpro64 kernel: [  276.381277] 631c  d4210000 f87c7b60 52800021 38346801 17ffffcd 97fe4698 f9444ea1 36080941
Jul  7 00:27:47 rockpro64 kernel: [  276.387587] 633c  b9401341 910003e2 340008e1 531f7821 d35efc00 8b21c2c1 cb000021 b6f80841
Jul  7 00:27:47 rockpro64 kernel: [  276.393943] 635c  6b160016 54000800 91022a99 d538d080 38606b21 34000101 91024294 f8746802
Jul  7 00:27:47 rockpro64 kernel: [  276.400375] 637c  d5384101 eb01005f 54fff980 3839681f 17ffffc9 34000157 90009981 52800000
Jul  7 00:27:47 rockpro64 kernel: [  276.406846] 
Jul  7 00:27:47 rockpro64 kernel: [  276.406846] SP: 0xffffffc0f7f43c80:
Jul  7 00:27:47 rockpro64 kernel: [  276.418255] 3c80  000000fd 00000000 00000000 00000000 f2b0fda0 ffffffc0 f7f46250 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.424700] 3ca0  0922b238 ffffff80 092678a0 ffffff80 00000000 00000000 f7f43d00 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.430999] 3cc0  0815631c ffffff80 f7f43d00 ffffffc0 0815631c ffffff80 600001c5 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.437238] 3ce0  09266898 ffffff80 0000003a 00000000 00000000 00000080 f2400028 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.443416] 3d00  f7f43d70 ffffffc0 08120dbc ffffff80 f7f46250 ffffffc0 f7f45c00 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.449523] 3d20  4639b2b9 00000040 f7f45c80 ffffffc0 09225000 ffffff80 00000003 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.455638] 3d40  093d9290 ffffff80 4639b2b9 00000040 0815618c ffffff80 4639a1a2 00000040
Jul  7 00:27:47 rockpro64 kernel: [  276.461755] 3d60  f7f46250 ffffffc0 09266898 ffffff80 f7f43de0 ffffffc0 081213a0 ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  276.467984] 
Jul  7 00:27:47 rockpro64 kernel: [  276.467984] X12: 0xffffffc0f29daba0:
Jul  7 00:27:47 rockpro64 kernel: [  276.478731] aba0  30303030 3030c630 30303030 787878c6 7c787878 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.484976] abc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.491248] abe0  06600606 06c00606 c6c6c60c 7cfe7c78 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.497530] ac00  c6060006 c606c606 c6067c06 c606c606 7c0cc60c 7c0c7c0c c618c018 c618c618
Jul  7 00:27:47 rockpro64 kernel: [  276.503868] ac20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 38380000
Jul  7 00:27:47 rockpro64 kernel: [  276.510270] ac40  d6d6d6d6 d6d6d6d6 d6d6d6d6 d6d6d6d6 c6c6c6c6 c6c6c6c6 c6c6c6c6 c6c6c6c6
Jul  7 00:27:47 rockpro64 kernel: [  276.516765] ac60  d6d6d6cc c6ccc6cc cc00c0c0 ccc6ccc6 00c6c6c6 7c767c76 7c7c7c76 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.523335] ac80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.529992] 
Jul  7 00:27:47 rockpro64 kernel: [  276.529992] X13: 0xffffff800ba00640:
Jul  7 00:27:47 rockpro64 kernel: [  276.541821] 0640  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.548650] 0660  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.555451] 0680  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.562248] 06a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.569016] 06c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.575741] 06e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.582395] 0700  00000000 00000000 00000000 00000000 00aaaaaa 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.589027] 0720  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.595526] 
Jul  7 00:27:47 rockpro64 kernel: [  276.595526] X14: 0xffffff800ba00640:
Jul  7 00:27:47 rockpro64 kernel: [  276.606832] 0640  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.613276] 0660  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.619592] 0680  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.625788] 06a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.631856] 06c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.637803] 06e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.643607] 0700  00000000 00000000 00000000 00000000 00aaaaaa 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.649395] 0720  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.655058] 
Jul  7 00:27:47 rockpro64 kernel: [  276.655058] X19: 0xffffff800922af80:
Jul  7 00:27:47 rockpro64 kernel: [  276.664674] af80  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.670280] afa0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.675762] afc0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.681108] afe0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.686326] b000  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.691420] b020  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.696375] b040  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.701298] b060  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.706079] 
Jul  7 00:27:47 rockpro64 kernel: [  276.706079] X20: 0xffffff800922b230:
Jul  7 00:27:47 rockpro64 kernel: [  276.714064] b230  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.718868] b250  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.723572] b270  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.728173] b290  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.732673] b2b0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.737087] b2d0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.741361] b2f0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.745578] b310  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.749673] 
Jul  7 00:27:47 rockpro64 kernel: [  276.749673] X21: 0xffffff8009265f80:
Jul  7 00:27:47 rockpro64 kernel: [  276.756322] 5f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.760484] 5fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.764520] 5fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.768417] 5fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.772209] 6000  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.775895] 6020  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.779436] 6040  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.782944] 6060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.786312] 
Jul  7 00:27:47 rockpro64 kernel: [  276.786312] X24: 0xffffffc0f2b0fd20:
Jul  7 00:27:47 rockpro64 kernel: [  276.791734] fd20  00000000 00000000 f2af4600 ffffffc0 000001c0 00000000 089708c0 ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  276.795422] fd40  00000002 00000000 08092188 ffffff80 f2b0fd60 ffffffc0 0809227c ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  276.799179] fd60  f2b0fda0 ffffffc0 089efbbc ffffff80 f1d32210 ffffffc0 f1d31e10 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.803012] fd80  f2b0fda0 ffffffc0 089efc0c ffffff80 f1d32000 ffffffc0 085d0a14 ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  276.806863] fda0  00000000 00000000 00000005 00000000 eed1b000 00000040 00000020 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.810814] fdc0  ffffffff 00ffffff 5d6e6ac3 00000001 00013240 00000000 00000005 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.814696] fde0  b5593519 00000034 08081800 ffffff80 00001000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.818622] fe00  34d5d91d 00000000 eed1b000 00000040 f2af4600 ffffffc0 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.822484] 
Jul  7 00:27:47 rockpro64 kernel: [  276.822484] X25: 0xffffffc0f7f461d0:
Jul  7 00:27:47 rockpro64 kernel: [  276.828883] 61d0  ffffffff 00000000 ffffffff ffffffff 00000001 00000000 f7f461e8 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.833044] 61f0  f7f461e8 ffffffc0 f7f461f8 ffffffc0 f7f461f8 ffffffc0 0814a210 ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  276.837304] 6210  f2aa7cf8 ffffffc0 f2aa7c50 ffffffc0 00020002 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.841659] 6230  00000001 00000000 00000045 00000000 00000044 00000000 f2b87000 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.846093] 6250  f7f46250 ffffffc0 00000000 00000000 00000000 00000000 46384600 00000040
Jul  7 00:27:47 rockpro64 kernel: [  276.850589] 6270  46384600 00000040 0815618c ffffff80 f7f45c80 ffffffc0 00000000 0000001f
Jul  7 00:27:47 rockpro64 kernel: [  276.855051] 6290  08156160 ffffff80 63746177 676f6468 0000352f 00000000 00000044 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.859579] 62b0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.864041] 
Jul  7 00:27:47 rockpro64 kernel: [  276.864041] X26: 0xffffff800922b1b8:
Jul  7 00:27:47 rockpro64 kernel: [  276.871664] b1b8  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.876458] b1d8  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.881255] b1f8  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.886048] b218  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.890843] b238  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.895635] b258  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.900283] b278  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.904883] b298  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.909336] 
Jul  7 00:27:47 rockpro64 kernel: [  276.909336] X27: 0xffffff8009267820:
Jul  7 00:27:47 rockpro64 kernel: [  276.916932] 7820  00000001 00000000 00000001 00000000 000000fa 00000000 f7f55480 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.921706] 7840  00000015 00000000 f7f55440 ffffffc0 00000001 00000006 00000060 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.926579] 7860  00015000 00001000 00000001 00000000 f7f554c0 ffffffc0 f7f578c0 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.931567] 7880  00000000 00000005 00000010 00000000 f7f55340 ffffffc0 f7ed7000 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.936570] 78a0  eecb2000 00000040 eecc7000 00000040 eecdc000 00000040 eecf1000 00000040
Jul  7 00:27:47 rockpro64 kernel: [  276.941666] 78c0  eed06000 00000040 eed1b000 00000040 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.946663] 78e0  00000002 00000000 0000341a 00000000 00010000 00000000 000f7fff 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.951672] 7900  0001e26f 00000000 00002000 00000000 00000000 0000fffa 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.956584] 
Jul  7 00:27:47 rockpro64 kernel: [  276.956584] X29: 0xffffffc0f7f43c80:
Jul  7 00:27:47 rockpro64 kernel: [  276.965091] 3c80  000000fd 00000000 00000000 00000000 f2b0fda0 ffffffc0 f7f46250 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.970327] 3ca0  0922b238 ffffff80 092678a0 ffffff80 00000000 00000000 f7f43d00 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.975652] 3cc0  0815631c ffffff80 f7f43d00 ffffffc0 0815631c ffffff80 600001c5 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.981076] 3ce0  09266898 ffffff80 0000003a 00000000 00000000 00000080 f2400028 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.986605] 3d00  f7f43d70 ffffffc0 08120dbc ffffff80 f7f46250 ffffffc0 f7f45c00 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.992229] 3d20  4639b2b9 00000040 f7f45c80 ffffffc0 09225000 ffffff80 00000003 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.997814] 3d40  093d9290 ffffff80 4639b2b9 00000040 0815618c ffffff80 4639a1a2 00000040
Jul  7 00:27:47 rockpro64 kernel: [  277.003464] 3d60  f7f46250 ffffffc0 09266898 ffffff80 f7f43de0 ffffffc0 081213a0 ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  277.009073] 
Jul  7 00:27:47 rockpro64 kernel: [  277.013844] ---[ end trace 8ad4400a27e2532b ]---
Jul  7 00:27:47 rockpro64 kernel: [  277.018940] Call trace:
Jul  7 00:27:47 rockpro64 kernel: [  277.023832] Exception stack(0xffffffc0f7f43b30 to 0xffffffc0f7f43c60)
Jul  7 00:27:47 rockpro64 kernel: [  277.029162] 3b20:                                   ffffff800922b000 0000008000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.034695] 3b40: ffffffc0f7f43d00 ffffff800815631c 0000000000000026 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.040252] 3b60: ffffffc0f7f43c00 ffffff800810b958 ffffffc0f7f43c80 ffffff8008f96378
Jul  7 00:27:47 rockpro64 kernel: [  277.045809] 3b80: ffffff8009266000 00000000000000fd 0000000000000000 ffffffc0f2b0fda0
Jul  7 00:27:47 rockpro64 kernel: [  277.051316] 3ba0: ffffffc0f7f46250 ffffff800922b238 ffffff80092678a0 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.056895] 3bc0: ffffffc0f7f43be0 ffffff8009225000 0000000000000026 0000000000000005
Jul  7 00:27:47 rockpro64 kernel: [  277.062391] 3be0: 00000040eed1b000 00000040eed1b000 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.067850] 3c00: 0000000000000001 0000000000000000 0000000000001000 0000000000aaaaaa
Jul  7 00:27:47 rockpro64 kernel: [  277.073271] 3c20: 0000000000000008 00000000ffffffff ffffffc0f29dac20 ffffff800ba006c0
Jul  7 00:27:47 rockpro64 kernel: [  277.078672] 3c40: ffffff800ba006c0 0000000000000000 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.084108] [<ffffff800815631c>] watchdog_timer_fn+0x190/0x2e8
Jul  7 00:27:47 rockpro64 kernel: [  277.089350] [<ffffff8008120dbc>] __hrtimer_run_queues+0x1bc/0x284
Jul  7 00:27:47 rockpro64 kernel: [  277.094585] [<ffffff80081213a0>] hrtimer_interrupt+0xac/0x1bc
Jul  7 00:27:47 rockpro64 kernel: [  277.099778] [<ffffff800812cdfc>] tick_handle_oneshot_broadcast+0x144/0x1c0
Jul  7 00:27:47 rockpro64 kernel: [  277.105045] [<ffffff80089719a4>] rk_timer_interrupt+0x4c/0x5c
Jul  7 00:27:47 rockpro64 kernel: [  277.110241] [<ffffff800810d0c8>] handle_irq_event_percpu+0xf4/0x1f0
Jul  7 00:27:47 rockpro64 kernel: [  277.115419] [<ffffff800810d214>] handle_irq_event+0x50/0x80
Jul  7 00:27:47 rockpro64 kernel: [  277.120565] [<ffffff8008110a50>] handle_fasteoi_irq+0xcc/0x134
Jul  7 00:27:47 rockpro64 kernel: [  277.125794] [<ffffff800810c5c4>] generic_handle_irq+0x2c/0x44
Jul  7 00:27:47 rockpro64 kernel: [  277.131064] [<ffffff800810c940>] __handle_domain_irq+0xb4/0xb8
Jul  7 00:27:47 rockpro64 kernel: [  277.136404] [<ffffff8008080e70>] gic_handle_irq+0xc8/0x180
Jul  7 00:27:47 rockpro64 kernel: [  277.141769] Exception stack(0xffffffc0f2b0fda0 to 0xffffffc0f2b0fed0)
Jul  7 00:27:47 rockpro64 kernel: [  277.147335] fda0: 0000000000000000 0000000000000005 00000040eed1b000 0000000000000020
Jul  7 00:27:47 rockpro64 kernel: [  277.153195] fdc0: 00ffffffffffffff 000000015d6e6ac3 0000000000013240 0000000000000005
Jul  7 00:27:47 rockpro64 kernel: [  277.158984] fde0: 00000034b5593519 ffffff8008081800 0000000000001000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.164811] fe00: 0000000034d5d91d 00000040eed1b000 ffffffc0f2af4600 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.170576] fe20: 0000000000000000 0000000000000000 0000000030d00800 00000040463979c2
Jul  7 00:27:47 rockpro64 kernel: [  277.176342] fe40: 0000000000000002 ffffffc0efd9b400 0000000000000002 ffffff80093d97b8
Jul  7 00:27:47 rockpro64 kernel: [  277.182075] fe60: 0000000000000001 0000004045fd1bb7 ffffff80093720c0 0000000002c9d234
Jul  7 00:27:47 rockpro64 kernel: [  277.187875] fe80: 0000000000000000 ffffffc0f2b0fed0 ffffff8008948f34 ffffffc0f2b0fed0
Jul  7 00:27:47 rockpro64 kernel: [  277.193661] fea0: ffffff8008948f38 0000000060000145 ffffffc0f2b0fed0 ffffff8008948f34
Jul  7 00:27:47 rockpro64 kernel: [  277.199506] fec0: ffffffffffffffff 0000000000000002
Jul  7 00:27:47 rockpro64 kernel: [  277.205075] [<ffffff80080827b4>] el1_irq+0xb4/0x140
Jul  7 00:27:47 rockpro64 kernel: [  277.210676] [<ffffff8008948f38>] cpuidle_enter_state+0x1fc/0x254
Jul  7 00:27:47 rockpro64 kernel: [  277.216358] [<ffffff8008949004>] cpuidle_enter+0x34/0x44
Jul  7 00:27:47 rockpro64 kernel: [  277.221947] [<ffffff80080fe400>] call_cpuidle+0x70/0x78
Jul  7 00:27:47 rockpro64 kernel: [  277.227432] [<ffffff80080fe6a4>] cpu_startup_entry+0x29c/0x2ac
Jul  7 00:27:47 rockpro64 kernel: [  277.232944] [<ffffff800808d550>] secondary_start_kernel+0x190/0x1bc
Jul  7 00:27:47 rockpro64 kernel: [  277.238509] [<0000000002c9d178>] 0x2c9d178

[ayufan]

Are you sure that you don't have underpower issue? This clearly seems to be the one. Maybe too weak power adapter?

[stevefan1999-personal]

Are you sure that you don't have underpower issue? This clearly seems to be the one. Maybe too weak power adapter?

I don't think 3A 5V with my R64 4GB is underpowered

[GuyNoIRQ]

I'm running the 12v 5A adapter on the RockPro64.

Also doing some testing I can dd straight from one disk to the other without hitting this. I'm seeing some of the below errors in kern.log sporadically, but if I md5sum large portions of the disks mirrored via dd I don't see any discrepancies.

errors

root@RockProNAS:~# tail -f /var/log/kern.log 
Jul  9 22:48:08 RockProNAS kernel: [  109.146034] zram: Decompression failed! err=-22, page=34
Jul  9 22:48:08 RockProNAS kernel: [  109.146119] zram: Decompression failed! err=-22, page=34
Jul  9 22:48:08 RockProNAS kernel: [  109.146129] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 22:48:11 RockProNAS kernel: [  112.218832] md127: detected capacity change from 8001428979712 to 0
Jul  9 22:48:11 RockProNAS kernel: [  112.218864] md: md127 stopped.
Jul  9 22:48:11 RockProNAS kernel: [  112.218935] md: unbind<sdb>
Jul  9 22:48:11 RockProNAS kernel: [  112.239895] md: export_rdev(sdb)
Jul  9 22:48:11 RockProNAS kernel: [  112.240083] md: unbind<sda>
Jul  9 22:48:11 RockProNAS kernel: [  112.252111] md: export_rdev(sda)
Jul  9 22:48:29 RockProNAS kernel: [  129.772559] cdn-dp fec00000.dp: [drm:cdn_dp_request_firmware] *ERROR* Timed out trying to load firmware
Jul  9 22:52:19 RockProNAS kernel: [  359.473686] zram: Decompression failed! err=-22, page=619
Jul  9 22:52:19 RockProNAS kernel: [  359.495814] zram: Decompression failed! err=-22, page=619
Jul  9 22:52:19 RockProNAS kernel: [  359.497577] zram: Decompression failed! err=-22, page=619
Jul  9 22:52:20 RockProNAS kernel: [  361.193502] zram: Decompression failed! err=-22, page=34
Jul  9 22:52:20 RockProNAS kernel: [  361.193543] zram: Decompression failed! err=-22, page=34
Jul  9 22:52:20 RockProNAS kernel: [  361.193633] zram: Decompression failed! err=-22, page=34
Jul  9 22:52:20 RockProNAS kernel: [  361.193643] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 22:53:09 RockProNAS kernel: [  410.166900] zram: Decompression failed! err=-22, page=34
Jul  9 22:53:09 RockProNAS kernel: [  410.166943] zram: Decompression failed! err=-22, page=34
Jul  9 22:53:09 RockProNAS kernel: [  410.167034] zram: Decompression failed! err=-22, page=34
Jul  9 22:53:09 RockProNAS kernel: [  410.167044] Buffer I/O error on dev zram0, logical block 34, async page read
.....truncated.....

Jul  9 23:25:49 RockProNAS kernel: [ 2369.682516] zram: Decompression failed! err=-22, page=34
Jul  9 23:25:49 RockProNAS kernel: [ 2369.682529] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 23:26:47 RockProNAS kernel: [ 2428.151934] zram: Decompression failed! err=-22, page=34
Jul  9 23:26:47 RockProNAS kernel: [ 2428.151953] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 23:26:47 RockProNAS kernel: [ 2428.207522] zram: Decompression failed! err=-22, page=34
Jul  9 23:26:47 RockProNAS kernel: [ 2428.207536] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 23:27:50 RockProNAS kernel: [ 2490.648482] zram: Decompression failed! err=-22, page=34
Jul  9 23:27:50 RockProNAS kernel: [ 2490.648527] zram: Decompression failed! err=-22, page=34
Jul  9 23:27:50 RockProNAS kernel: [ 2490.648619] zram: Decompression failed! err=-22, page=34
Jul  9 23:27:50 RockProNAS kernel: [ 2490.648630] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 23:27:50 RockProNAS kernel: [ 2490.692936] zram: Decompression failed! err=-22, page=34
Jul  9 23:27:50 RockProNAS kernel: [ 2490.692952] Buffer I/O error on dev zram0, logical block 34, async page read

the dd command I used to copy from sda to sdb

root@RockProNAS:~# dd if=/dev/sda of=/dev/sdb bs=4M status=progress
391034961920 bytes (391 GB, 364 GiB) copied, 2059.02 s, 190 MB/s^C^C

A coupld md5sum checks of what should be identical blocks.

root@RockProNAS:~# sudo dd if=/dev/sdb bs=4M skip=1000 count=10000 | md5sum
10000+0 records in
10000+0 records out
41943040000 bytes (42 GB, 39 GiB) copied, 436.12 s, 96.2 MB/s
d8c26ac026147737e73aee59eb37766a  -
root@RockProNAS:~# sudo dd if=/dev/sda bs=4M skip=1000 count=10000 | md5sum
10000+0 records in
10000+0 records out
41943040000 bytes (42 GB, 39 GiB) copied, 429.064 s, 97.8 MB/s
d8c26ac026147737e73aee59eb37766a  -

EDIT: Maybe dd doesn't work. I let it run overnight and came back to the box powered down and not respondig. Unfortunately I wasn't able to gather any logs from this. I'm going to run it again to see if I can repro the issue and tail -f kern.log out to a separate file so I can see if I can glean any more info into what happened.

EDIT 2: dd isn't stable. I hit this when dding /dev/zero to one of the disks.

EDIT 3: I hit the spinlock error when I simply formatted just one of the disks with ext4.

[GuyNoIRQ]

I actually figured out that I had a bad PCI card. I picked up the one at the link below, and added the "pci=nomsi" boot parameter as outlined in the following forum post and was able to initialize raid1 on 2x white label HGST as well as 2x older intel SSDs I had laying around at the same time while also running "stress -c4 -m16" then further could run dd of /dev/urandom to both raid devices and all is stable.

https://www.amazon.com/gp/product/B07KNXZFRH/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1

https://forum.pine64.org/showthread.php?tid=6496

[jwindhager]

I have the same issue.

My configuration:
ROCKPro64 4GB Single Board Computer
ROCKPro64 PCI-e to Dual SATA-II Interface Card
ROCKPro64 Metal Desktop/NAS Casing with fan
ROCKPro64 12V 5A EU Power Supply
2x Seagate IronWolf NAS HDD, 4TB each

Installed 0.9.14 (tried both armhf and arm64) first, updated all packages, activated SMART monitoring, quick-wiped both disks, created RAID1 via the OMV web interface. Server crashes at random times, web interface not reachable, SSH doesn't work. No dmesg output (via SSH) before the crash. Sometimes still responds to iCMP pings, sometimes doesn't. After a power cycle, RAID is in "sync pending" state, nothing suspicious in kern.log.

Then tried a vanilla armbian (stretch), updated all packages, installed OMV via armbian-config, created RAID1 via the OMV web interface as before. This time received a kernel panic while connected via SSH:

Message from syslogd@localhost at Nov 18 10:01:47 ...
kernel:[ 75.976355] Internal error: : 96000210 [#1] SMP

Message from syslogd@localhost at Nov 18 10:01:47 ...
kernel:[ 83.247622] BUG: spinlock lockup suspected on CPU#2, scsi_eh_1/284

Message from syslogd@localhost at Nov 18 10:01:47 ...
kernel:[ 83.251260] lock: 0xffffff8009121870, .magic: dead4ead, .owner: scsi_eh_1/284, .owner_cpu: 2

Message from syslogd@localhost at Nov 18 10:03:59 ...
kernel:[ 240.428491] Kernel panic - not syncing: hung_task: blocked tasks

Finally tried booting the armbian-setup with pci=nomsi, resulting in the same behavior as with the 0.9.14 image (no error message, just crashed).

Any updates on this issue?

[ayufan]

Can you try to force downclock RAM to 400MHz? I would assume that this is due to this (this was usually thing in the past). Is not that RAM is bad, but maybe some setting is off somewhere.

…

On Mon, Nov 18, 2019 at 12:22 PM Jonas Windhager ***@***.***> wrote: I have the same issue. My configuration: ROCKPro64 4GB Single Board Computer ROCKPro64 PCI-e to Dual SATA-II Interface Card ROCKPro64 Metal Desktop/NAS Casing with fan ROCKPro64 12V 5A EU Power Supply 2x Seagate IronWolf NAS HDD, 4TB each Installed 0.9.14 (tried both armhf and arm64) first, updated all packages, activated SMART monitoring, quick-wiped both disks, created RAID1 via the OMV web interface. Server crashes at random times, web interface not reachable, SSH doesn't work. No dmesg output (via SSH) before the crash. Sometimes still responds to iCMP pings, sometimes doesn't. After a power cycle, RAID is in "sync pending" state, nothing suspicious in kern.log. Then tried a vanilla armbian (stretch), updated all packages, installed OMV via armbian-config, created RAID1 via the OMV web interface as before. This time received a kernel panic while connected via SSH: Message from ***@***.*** at Nov 18 10:01:47 ... kernel:[ 75.976355] Internal error: : 96000210 [#1] SMP Message from ***@***.*** at Nov 18 10:01:47 ... kernel:[ 83.247622] BUG: spinlock lockup suspected on CPU#2, scsi_eh_1/284 Message from ***@***.*** at Nov 18 10:01:47 ... kernel:[ 83.251260] lock: 0xffffff8009121870, .magic: dead4ead, .owner: scsi_eh_1/284, .owner_cpu: 2 Message from ***@***.*** at Nov 18 10:03:59 ... kernel:[ 240.428491] Kernel panic - not syncing: hung_task: blocked tasks Finally tried booting the armbian-setup with pci=nomsi, resulting in the same behavior as with the 0.9.14 image (no error message, just crashed). Any updates on this issue? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#299?email_source=notifications&email_token=AASOSQNZAVHYAOKOC2HOG63QUJ3IBA5CNFSM4GBUZLP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEKDR3I#issuecomment-554973421>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASOSQP57VJPCZJV63UYAY3QUJ3IBANCNFSM4GBUZLPQ> .

[jwindhager]

Hi ayufan! Thanks for your reply. How can I force the RAM clock down to 400MHz?

[ayufan]

Look in /sys/class/devfreq/dmc

[jwindhager]

Unfortunately, setting /sys/class/devfreq/dmc/max_freq to 400000000 didn't help. This time, however, dmesg recorded the problem "on time":

[Nov18 23:22] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80093e6130
[  +0.008153] Internal error: : 96000210 [#1] SMP

Message from syslogd@localhost at Nov 18 23:22:09 ...
 kernel:[ 2234.999648] Internal error: : 96000210 [#1] SMP
[  +0.007787] Modules linked in: softdog af_packet lz4hc lz4hc_compress zlib snd_soc_rockchip_hdmi_dp rk_vcodec lzo zram ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod phy_rockchip_pcie
[  +0.026297] CPU: 0 PID: 275 Comm: scsi_eh_0 Not tainted 4.4.192-rockchip64 #1
[  +0.008800] Hardware name: Pine64 RockPro64 (DT)
[  +0.008527] task: ffffffc0dfb2f000 task.stack: ffffffc0f13c8000
[  +0.008705] PC is at ahci_scr_read+0x70/0x98
[  +0.008534] LR is at sata_scr_read+0x88/0x94
[  +0.008413] pc : [<ffffff800875bf08>] lr : [<ffffff80087427cc>] pstate: 80000145
[  +0.008847] sp : ffffffc0f13cba50
[  +0.008375] x29: ffffffc0f13cba50 x28: 0000000000000000 
[  +0.008557] x27: ffffffc0f111de80 x26: 0000000000000001 
[  +0.008501] x25: ffffffc0f111e258 x24: ffffffc0f111c000 
[  +0.008404] x23: 0000000000000000 x22: ffffffc0f13cbb28 
[  +0.008362] x21: ffffffc0f111de80 x20: ffffffc0f13cbb28 
[  +0.008370] x19: 0000000000000001 x18: 0000000030d00800 
[  +0.008406] x17: 0000000000000000 x16: 0000000000000000 
[  +0.008354] x15: 0000000000000000 x14: 0000000000000000 
[  +0.008249] x13: 0000000000000000 x12: 0000000000000000 
[  +0.008137] x11: 0000000000000000 x10: 0000000000000000 
[  +0.007989] x9 : 0000000000000000 x8 : ffffffc0f111e258 
[  +0.007940] x7 : 0000000000000000 x6 : ffffffc0f111e2d0 
[  +0.007889] x5 : 0000000000000000 x4 : ffffff8008caa2fc 
[  +0.007881] x3 : 0000000000000130 x2 : ffffffc0f13cbb28 
[  +0.007871] x1 : 0000000000000030 x0 : ffffff80093e6130 
[  +0.007827] 
              PC: 0xffffff800875be88:
[  +0.014995] be88  52800013 91392821 97fe24e0 17ffffe2 a9bd7bfd 910003fd a90153f3 f90013f5
[  +0.008527] bea8  aa0003f5 2a0103f3 aa1e03e0 aa0203f4 d503201f f94002a3 7100127f f959dc60
[  +0.008603] bec8  b9402c63 f9401801 53196063 91040063 f9400820 54000268 54000061 b9401821
[  +0.008668] bee8  36e80201 f0002a64 910b8084 91007084 b8735881 34000161 8b21c063 8b030000
[  +0.008674] bf08  b9400000 d5033d9f b9000280 52800000 a94153f3 f94013f5 a8c37bfd d65f03c0
[  +0.008708] bf28  128002a0 17fffffb a9be7bfd 910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f
[  +0.008725] bf48  f9400260 f9400000 f959dc00 f9401800 b9400000 36400120 52801fe0 b9038660
[  +0.008736] bf68  f0003c22 900038a1 91399842 912ff421 aa1303e0 97ff9d5d f9400bf3 a8c27bfd
[  +0.008855] 
              LR: 0xffffff800874274c:
[  +0.016111] 274c  a90153f3 a9025bf5 aa0003f3 2a0103f5 aa1e03e0 aa0203f6 d503201f f9400274
[  +0.008943] 276c  d283d000 8b000280 eb00027f 54000180 f959c280 eb00027f 54000120 aa1603e2
[  +0.008966] 278c  2a1503e1 aa1303e0 9400573d a94153f3 a9425bf5 a8c37bfd d65f03c0 aa1303e0
[  +0.008985] 27ac  97fffb19 34000100 f9400680 aa1603e2 2a1503e1 f9406c03 aa1303e0 d63f0060
[  +0.009029] 27cc  17fffff3 12800bc0 17fffff1 a9ba7bfd 910003fd a90153f3 a9025bf5 a90363f7
[  +0.009210] 27ec  a9046bf9 aa0103f5 aa0203f6 aa0003f7 aa1e03e0 d503201f f0004e93 b94012a0
[  +0.009226] 280c  f940e274 a94066b8 97e76ebe 8b000280 cb160001 910173a2 f100003f 52800001
[  +0.009244] 282c  9a80a2d6 aa1703e0 97ffffc4 2a0003f5 350003c0 b9405fb4 f940e27a 12000e94
[  +0.009336] 
              SP: 0xffffffc0f13cb9d0:
[  +0.016472] b9d0  f13cbb28 ffffffc0 00000000 00000000 f111c000 ffffffc0 f111e258 ffffffc0
[  +0.008879] b9f0  00000001 00000000 f111de80 ffffffc0 00000000 00000000 f13cba50 ffffffc0
[  +0.008658] ba10  087427cc ffffff80 f13cba50 ffffffc0 0875bf08 ffffff80 80000145 00000000
[  +0.008461] ba30  091178a0 ffffff80 f7ede200 ffffffc0 ffffffff ffffffff 080f0094 ffffff80
[  +0.008373] ba50  f13cba80 ffffffc0 087427cc ffffff80 f111de80 ffffffc0 f111c000 ffffffc0
[  +0.008306] ba70  00000001 00000000 f2aa4680 ffffffc0 f13cbab0 ffffffc0 0874f6ac ffffff80
[  +0.008282] ba90  f111e258 ffffffc0 f111de80 ffffffc0 f111f858 ffffffc0 00003008 00000000
[  +0.008266] bab0  f13cbb60 ffffffc0 0874fe6c ffffff80 f111c000 ffffffc0 f111de80 ffffffc0
[  +0.008353] 
              X0: 0xffffff80093e60b0:
[  +0.014831] 60b0 
[  +0.050245] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80093e60b0
[  +0.961736] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[romanczukiewiczs]

Hi,

Is there some person who has this setup with HDD disk and this works correctly?

ROCKPro64 4GB Single Board Computer
ROCKPro64 PCI-e to Dual SATA-II Interface Card
ROCKPro64 Metal Desktop/NAS Casing with fan
ROCKPro64 12V 5A EU Power Supply

Thanks in advance.

[bullet64]

@romanczukiewiczs Please ask in https://forum.pine64.org/index.php

I have massiv problems with pine64 pcie-to-sata adapter. Now i use Marvell 88SE9230 without problems.

[romanczukiewiczs]

@bullet64 do You use this with pine64rockpro? You did on this setup something with mdadm?

[bullet64]

Yes. Please read about it in my forum. My forum profile https://forum.pine64.org/member.php?action=profile&uid=8601

[abdel-unxp]

Hi
Any news on that ?
I had the same issue on manjaro and debian buster.
But the issue is not only on PCIe to sata card:

I can get, like you, the issue in 5 minutes with PCIe and SATA disk if I use syncthing and let it scan my files or if I use dd command.
But I have the same issue if I remove the PCIe card and if I do hexdump /dev/eMMC disk or sdcard and wait 10 to 40 minutes. I would have never noticed that if I didn't decided to do more test on my eMMC and sd card driver.

When I use the sdcard I can see that when I have the issue, it is always after 10 seconds of DMA failure (DMA is started but we dont get the DMA done interruption).
So I first though it was an issue on sdcard driver but no:
If I remove the sdcard and use eMMC only, I will also get the same (a)synchronous external abort oops

Note: When adding debug messages in the kernel driver, I was taking more time to get the oops, but you will get it

It is unlikely that we have similar BUG in eMMC, sdcard and PCIe so I think it is a hardware issue.
But why some people dont have this issue ?
Maybe only some devices have this BUG ?

I am talking about this issue here:
https://forum.pine64.org/showthread.php?tid=8372