Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pi 3+ buster pxe nfsboot - kernel hangs / "Fatal exception in interrupt" panics #3067

Open
maxnet opened this issue Jul 12, 2019 · 12 comments

Comments

@maxnet
Copy link
Contributor

commented Jul 12, 2019

Describe the bug

PXE/nfs booting Buster on a Pi 3+ does not quite work

Does boot the kernel and starts to execute the first systemd units, but then the system hangs some seconds after on random moments (text cursor stops blinking, no activity, numlock does not respond)

Happens both with the kernel version you ship by default in your June 2019 Buster image, and 4.19.57 you get after running "apt-get upgrade"

In most cases kernels hangs without outputing any message, but seen the following kernel panic as well.

         Starting Helper to synchronize boot up for ifupdown...
[  OK  ] Mounted /tmp.
[  OK  ] Mounted /media.
[  OK  ] Mounted /var/lib/sudo.
[  OK  ] Mounted /var/lib/dhcp.
[  OK  ] Mounted /var/tmp.
[  OK  ] Mounted /home.
[  OK  ] Mounted /var/log.
[  OK  ] Mounted /var/cache.
[  OK  ] Mounted /var/spool.
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Mounted /var/lib/systemd/rfkill.
[  OK  ] Mounted /var/lib/lightdm.
[  OK  ] Mounted /var/lib/systemd/coredump.
[  OK  ] Started Helper to synchronize boot up for ifupdown.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Reached target Local File Systems.
         Starting Preprocess NFS configuration...
         Starting Set console font and keymap...
         Starting Raise network interfaces...
[  OK  ] Started Preprocess NFS configuration.
[  OK  ] Reached target NFS client services.
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Remote File Systems.
[   14.481793] Unable to handle kernel paging request at virtual address f74193c8
[   14.500886] pgd = 30af461a
[   14.515245] [f74193c8] *pgd=00000000
[   14.530421] Internal error: Oops: 80000005 [#1] SMP ARM
[   14.547260] Modules linked in: bcm2835_v4l2(C+) v4l2_mem2mem snd_timer v4l2_common bcm2835_mmal_vchiq(C) snd videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media vc_sm_cma(C) uio_pdrv_genirq uio fixed i2c_dev ip_tables x_tables ipv6
[   14.597249] CPU: 0 PID: 9 Comm: ksoftirqd/0 Tainted: G         C        4.19.57-v7+ #1244
[   14.617972] Hardware name: BCM2835
[   14.633734] PC is at 0xf74193c8
[   14.649171] LR is at rcu_process_callbacks+0x2cc/0x674
[   14.666604] pc : [<f74193c8>]    lr : [<8018e0b8>]    psr: a0000013
[   14.685285] sp : b9d3de58  ip : b9d3de58  fp : b9d3decc
[   14.702969] r10: b9d3c018  r9 : 0000000a  r8 : 80d10c80
[   14.720594] r7 : b9d3de8c  r6 : ba352660  r5 : 80d10780  r4 : ba352640
[   14.739565] r3 : f74193c8  r2 : f74193c8  r1 : b9d3de08  r0 : b7419654
[   14.758469] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   14.778024] Control: 10c5383d  Table: 376fc06a  DAC: 00000055
[   14.796173] Process ksoftirqd/0 (pid: 9, stack limit = 0x1631a8d3)
[   14.814825] Stack: (0xb9d3de58 to 0xb9d3e000)
[   14.831548] de40:                                                       00000000 80d04d48
[   14.852230] de60: ba351e40 00000001 80d10780 80d97784 80d04d48 80d04d70 80d8ed74 80c903f0
[   14.872765] de80: 00000001 80d04e10 80102374 f74193c8 b8350000 fffffffa ffffffff a271c001
[   14.893156] dea0: b9d3decc 80d030a4 00000009 b9d3c010 00000100 80d0be48 00000024 00000200
[   14.913429] dec0: b9d3df34 b9d3ded0 80102410 8018ddf8 802018e8 80202aec 00000000 04208040
[   14.933588] dee0: 80d03d00 ffff907a 00000009 80c94a00 80d8f278 80d04d70 80c8c308 80d03080
[   14.953722] df00: b9d3ded0 0000000a b9d3df34 b9d3c000 b9ccb380 00000000 00000001 80d0be48
[   14.973701] df20: 80d04d48 00000000 b9d3df44 b9d3df38 8012664c 8010228c b9d3df7c b9d3df48
[   14.993483] df40: 80146b1c 80126614 b9d40000 a271c001 b9d3df7c b9ccb580 b9ccb540 00000000
[   15.013156] df60: b9ccb380 801469e4 b9ccb59c b9d0ddd4 b9d3dfac b9d3df80 8014251c 801469f0
[   15.032816] df80: ffffffff b9ccb540 801423e4 00000000 00000000 00000000 00000000 00000000
[   15.052490] dfa0: 00000000 b9d3dfb0 801010ac 801423f0 00000000 00000000 00000000 00000000
[   15.072096] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   15.091679] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[   15.111014] [<8018e0b8>] (rcu_process_callbacks) from [<80102410>] (__do_softirq+0x190/0x3f0)
[   15.130616] [<80102410>] (__do_softirq) from [<8012664c>] (run_ksoftirqd+0x44/0x54)
[   15.149185] [<8012664c>] (run_ksoftirqd) from [<80146b1c>] (smpboot_thread_fn+0x138/0x1d0)
[   15.168335] [<80146b1c>] (smpboot_thread_fn) from [<8014251c>] (kthread+0x138/0x168)
[   15.186934] [<8014251c>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[   15.204883] Exception stack(0xb9d3dfb0 to 0xb9d3dff8)
[   15.220500] dfa0:                                     00000000 00000000 00000000 00000000
[   15.239418] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   15.258293] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   15.275462] Code: bad PC value
[   15.288934] ---[ end trace 3e6a1f362873c12f ]---
[   15.304036] Kernel panic - not syncing: Fatal exception in interrupt
[   15.321004] CPU1: stopping
[   15.334358] CPU: 1 PID: 144 Comm: systemd-udevd Tainted: G      D  C        4.19.57-v7+ #1244
[   15.353750] Hardware name: BCM2835
[   15.367836] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   15.386527] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   15.404422] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   15.422021] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   15.441271] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<80101d0c>] (__irq_usr+0x4c/0x60)
[   15.460543] Exception stack(0xb8269fb0 to 0xb8269ff8)
[   15.475907] 9fa0:                                     00000001 004f4910 2d4eb800 2d4eb800
[   15.494608] 9fc0: 0000002f 00000000 7ed3897c 76ffd968 004f4910 004e8680 0000002f 7ed38924
[   15.513331] 9fe0: 00515cbc 7ed38758 004c1998 76ee1134 60000010 ffffffff
[   15.530213] CPU3: stopping
[   15.542679] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D  C        4.19.57-v7+ #1244
[   15.560547] Hardware name: BCM2835
[   15.573622] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   15.591260] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   15.608438] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   15.625665] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   15.644624] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
[   15.663548] Exception stack(0xb9d51f38 to 0xb9d51f80)
[   15.678202] 1f20:                                                       80109a84 00000000
[   15.695751] 1f40: 40000193 40000193 80d04d70 b9d50000 80d04db8 00000008 80d8ed3e 410fd034
[   15.713290] 1f60: 00000000 b9d51f94 80d0517c b9d51f88 00000000 80109a88 40000113 ffffffff
[   15.730839] [<801019bc>] (__irq_svc) from [<80109a88>] (arch_cpu_idle+0x34/0x4c)
[   15.747679] [<80109a88>] (arch_cpu_idle) from [<80831eb4>] (default_idle_call+0x34/0x48)
[   15.765310] [<80831eb4>] (default_idle_call) from [<8015265c>] (do_idle+0xec/0x17c)
[   15.782510] [<8015265c>] (do_idle) from [<801529ac>] (cpu_startup_entry+0x28/0x2c)
[   15.799636] [<801529ac>] (cpu_startup_entry) from [<8010fd54>] (secondary_start_kernel+0x134/0x140)
[   15.818384] [<8010fd54>] (secondary_start_kernel) from [<0010270c>] (0x10270c)
[   15.835311] CPU2: stopping
[   15.847637] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D  C        4.19.57-v7+ #1244
[   15.865445] Hardware name: BCM2835
[   15.878460] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   15.896045] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   15.913200] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   15.930420] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   15.949258] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
[   15.967977] Exception stack(0xb9d4ff38 to 0xb9d4ff80)
[   15.982797] ff20:                                                       80109a84 00000000
[   16.000859] ff40: 40000093 40000093 80d04d70 b9d4e000 80d04db8 00000004 80d8ed3e 410fd034
[   16.018940] ff60: 00000000 b9d4ff94 80d0517c b9d4ff88 00000000 80109a88 40000013 ffffffff
[   16.037026] [<801019bc>] (__irq_svc) from [<80109a88>] (arch_cpu_idle+0x34/0x4c)
[   16.054360] [<80109a88>] (arch_cpu_idle) from [<80831eb4>] (default_idle_call+0x34/0x48)
[   16.072432] [<80831eb4>] (default_idle_call) from [<8015265c>] (do_idle+0xec/0x17c)
[   16.090080] [<8015265c>] (do_idle) from [<801529ac>] (cpu_startup_entry+0x28/0x2c)
[   16.107627] [<801529ac>] (cpu_startup_entry) from [<8010fd54>] (secondary_start_kernel+0x134/0x140)
[   16.126830] [<8010fd54>] (secondary_start_kernel) from [<0010270c>] (0x10270c)
[   16.144261] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
@justinh-rahb

This comment has been minimized.

Copy link

commented Jul 12, 2019

Confirmed. It's broken. How can I revert to previous version?

@maxnet

This comment has been minimized.

Copy link
Contributor Author

commented Jul 12, 2019

Some more panics, which seems to be different each time.
But all happen pretty soon after boot.

[   14.212069] Unable to handle kernel paging request at virtual address 726f7774
[   14.231232] pgd = f02b8ead
[   14.245697] [726f7774] *pgd=00000000
[   14.261056] Internal error: Oops: 5 [#1] SMP ARM
[   14.277303] Modules linked in: bcm2835_v4l2(C+) bcm2835_codec(C+) snd_timer v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig v4l2_common snd videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common vc_sm_cma(C) videodev media uio_pdrv_genirq fixed uio i2c_dev ip_tables x_tables ipv6
[   14.329028] CPU: 0 PID: 187 Comm: setupcon Tainted: G         C        4.19.57-v7+ #1244
[   14.349746] Hardware name: BCM2835
[   14.365579] PC is at __kmalloc_track_caller+0xac/0x288
[   14.383203] LR is at __kmalloc_track_caller+0x60/0x288
[   14.400693] pc : [<80297b6c>]    lr : [<80297b20>]    psr: 20000113
[   14.419399] sp : b6e9d4e0  ip : b6e9d4e0  fp : b6e9d524
[   14.437112] r10: ffffffff  r9 : b9c01b40  r8 : 806e63fc
[   14.454771] r7 : 00490220  r6 : b9c01b40  r5 : 00000180  r4 : 726f7774
[   14.473775] r3 : 00000000  r2 : 80c97aa0  r1 : 396bd000  r0 : b6e9d4e0
[   14.492706] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   14.512303] Control: 10c5383d  Table: 36ed006a  DAC: 00000055
[   14.530456] Process setupcon (pid: 187, stack limit = 0xbd2c0117)
[   14.549055] Stack: (0xb6e9d4e0 to 0xb6e9e000)
[   14.565815] d4e0: b815d480 80d04d48 b9d16000 00001cb8 b6e9d50c b6e9d500 80126b88 b6e9d563
[   14.586555] d500: 00480020 80d04d48 806e63fc 00000180 00480020 ffffffff b6e9d54c b6e9d528
[   14.607118] d520: 806e6320 80297acc 8060763c b815d3c0 00000092 80d04d48 8060763c b9e42240
[   14.627511] d540: b6e9d594 b6e9d550 806e63fc 806e62f8 b6e9d563 b6e9d560 802018e8 00000000
[   14.647742] d560: 20000113 6950c633 b9d16558 00000092 00000001 b9d16558 00000001 b9d16500
[   14.667857] d580: b9d16564 b9d16558 b6e9d5dc b6e9d598 8060763c 806e6378 80d04d48 b9d16538
[   14.687894] d5a0: ba349298 00000000 00000006 6950c633 b6e9d5dc b9d16568 b9d1656c 00000000
[   14.707793] d5c0: ba349298 00000000 00000006 00000040 b6e9d604 b6e9d5e0 80126cf4 8060749c
[   14.727524] d5e0: 80d03098 00000006 b6e9c018 00000101 80d04d48 00000018 b6e9d614 b6e9d608
[   14.747198] d600: 80126dd4 80126c9c b6e9d67c b6e9d618 80102410 80126db8 b6e9d69c b6e9d628
[   14.766851] d620: 00000000 00400040 80d03d00 ffff905f 0000000a 80c94a00 80d8f278 80d04d70
[   14.786507] d640: 80c8c308 80d03080 b6e9d618 00000007 b815d480 60000193 b815d480 b969e400
[   14.806177] d660: 0000008a 80d04d48 00000010 00000000 b6e9d694 b6e9d680 801266d0 8010228c
[   14.825923] d680: b6e9c000 00000200 b6e9d6ac b6e9d698 801267fc 80126668 6950c633 00000000
[   14.845529] d6a0: b6e9d6e4 b6e9d6b0 8075d7f8 80126734 8db2a8c0 6950c633 0000000c b815d480
[   14.864988] d6c0: 80d04d48 b97c49c0 00000000 80d7fc40 000005dc b9d16000 b6e9d71c b6e9d6e8
[   14.884304] d6e0: 8075f4a4 8075d5d4 00000200 80d04d48 0000048a b815d480 80d04d48 0000048a
[   14.903562] d700: 00000000 80d7fc40 b97c49c0 b9d16000 b6e9d76c b6e9d720 80760114 8075f36c
[   14.922822] d720: 8075f658 8075d544 807609c4 8070e998 b9f46700 00000000 00000000 6950c633
[   14.941989] d740: 8075e61c b815d480 b97c49c0 80d7fc40 00000068 00000000 b6e9d868 00000060
[   14.961057] d760: b6e9d78c b6e9d770 8075f78c 8076004c 00003ae7 b815d480 80d7fc40 b80be434
[   14.980123] d780: b6e9d7a4 b6e9d790 80760a8c 8075f750 b815d480 b97c49c0 b6e9d7cc b6e9d7a8
[   14.999212] d7a0: 8078e638 80760a74 b97c49c0 b815d480 b6e9d868 80d04d48 b6e9d840 00000000
[   15.018292] d7c0: b6e9d914 b6e9d7d0 80791388 8078e518 00000068 00000008 b6e9d818 b6e9d814
[   15.037381] d7e0: b6e9d840 00004040 80d7fc40 8db2a8c0 00000000 00000000 00000108 8075f0c8
[   15.056488] d800: 00000000 00000000 b97c4c20 00000000 805b2158 00000000 00000000 00000000
[   15.075604] d820: 00000000 00000000 8db2a8c0 00000000 00000000 ffff0000 00000000 00000000
[   15.094710] d840: 00000000 00000000 00000000 000005dc 00000068 00000000 ffff0000 00000000
[   15.113754] d860: 00000000 00000000 00000002 00000001 00000000 00110000 00000000 00000040
[   15.132743] d880: 00000000 00000000 00000000 801c4cac e2b2a8c0 8db2a8c0 3d030108 80154f84
[   15.151744] d8a0: 00000001 00000000 000001d9 00000000 00000000 00000000 8081f0d4 ba351ec0
[   15.170359] d8c0: b9422d80 80d04d48 00000000 4f1765da 00000003 00000000 b6e9d94c b6e9d8e8
[   15.188606] d8e0: 80154f84 6950c633 b9422d80 b97c49c0 b9ab0000 00000000 6950c633 00000000
[   15.206876] d900: b6e9da38 00000000 b6e9d934 b6e9d918 8079c1c0 80790e74 b6e9d94c 6950c633
[   15.225220] d920: b6e9d97c b9ab0000 b6e9d944 b6e9d938 806ddab8 8079c184 b6e9d964 b6e9d948
[   15.243527] d940: 806ddbf4 806ddaa0 00000060 00000402 80d04d48 b97d5004 b6e9d9d4 b6e9d968
[   15.261857] d960: 807e78fc 806ddbc0 00000060 ba351e40 b6e9d9c4 b97d5004 00000060 b97b100c
[   15.280188] d980: 00000010 00000003 00000000 00000000 b6e9d97c 00000000 00000000 00000000
[   15.298259] d9a0: 00000000 00004040 00000000 6950c633 b6eb9e00 00000001 00000060 b970b804
[   15.316069] d9c0: b9ab0000 b9f46e00 b6e9da24 b6e9d9d8 807e799c 807e785c 00000000 00000000
[   15.333752] d9e0: b6e9da24 b6e9d9f0 8019b964 806a7b48 45ed5e2c 00000003 b9b89958 b970b800
[   15.351427] da00: 80d04d48 b97b1000 b970b804 b9f46e00 00000000 00000000 b6e9da64 b6e9da28
[   15.369060] da20: 807e7e54 807e7924 00000000 00000001 b6e9da38 b6e9da40 00000000 6950c633
[   15.386700] da40: b97b1000 b970b800 00000880 b970b874 02000000 5d8f07f8 b6e9daa4 b6e9da68
[   15.404330] da60: 807e4a70 807e7de4 b970b804 00000000 00000000 b9f46e00 b970b800 b9f46e00
[   15.422034] da80: b970b800 00000880 b970b800 02000000 5d8f07f8 807e172c b6e9dad4 b6e9daa8
[   15.439775] daa0: 807e18a0 807e4a28 b9b89958 807ef8c8 b9f46e00 b97b10f8 00000000 b6e9c018
[   15.457159] dac0: 80d04d70 80d04d48 b6e9db1c b6e9dad8 807ef8c8 807e1738 00000080 80d932b0
[   15.474167] dae0: b6e9dafc b6e9daf0 80168bb4 6950c633 b6e9db1c b9f46e00 b9f46e00 00000000
[   15.491169] db00: b96c3200 b9b23b28 fffffff6 00000000 b6e9db3c b6e9db20 807f0234 807ef848
[   15.508151] db20: 00000002 b9f46e00 b96c3200 b6e9db64 b6e9db5c b6e9db40 807e0360 807f01bc
[   15.525169] db40: 00000000 b6e9c000 80d04d48 b6e9dbc4 b6e9db9c b6e9db60 807e04bc 807e021c
[   15.542235] db60: 807f12a0 00000000 b96c3200 00000000 b6e9dbc4 809567a8 00000000 00000000
[   15.559303] db80: 00000000 6950c633 b6e9c000 80dc40c0 b6e9dbbc b6e9dba0 803e8b94 807e0460
[   15.576377] dba0: 80d04d48 b9de0c00 b9de0c00 b9b23bb0 b6e9dbec b6e9dbc0 803e9b20 803e8b4c
[   15.593463] dbc0: fffffff6 80914c64 b9b89958 b81a4900 00000000 6950c633 b9b89aa0 b81a4900
[   15.610619] dbe0: b6e9dc14 b6e9dbf0 803d22e0 803e9ad4 b9b89aa0 803d1e78 b9b89aa0 b9b89aa0
[   15.627841] dc00: 00000101 b9b23bb0 b6e9dc2c b6e9dc18 803ca118 803d2224 00410444 b9b897d0
[   15.645053] dc20: b6e9dc74 b6e9dc30 803ccefc 803ca0d8 b6e9dc54 b6e9dc40 802b62e8 804f6af0
[   15.662285] dc40: 00000202 00000000 b6e9dd00 b6e9dd48 b9b23bb0 b6e9dd00 b6e9dcfc 80d04d48
[   15.679521] dc60: fffffff6 00000002 b6e9dcb4 b6e9dc78 802b6e54 803ccc08 0040080c b9d95310
[   15.696773] dc80: 00000002 6950c633 802b7008 00020020 b9b23b28 00000000 b6e9de08 b838c6c0
[   15.714000] dca0: b6e9dd48 b6e9dd00 b6e9dd44 b6e9dcb8 802ba390 802b6b80 802ca0bc 802c9f20
[   15.731256] dcc0: b6e9dd04 b6e9dcd0 804fd7dc 804fc0a8 80238e6c b6e9c000 80d04d48 00000eac
[   15.748516] dce0: 80d04d48 00000041 80d05688 00000000 b8f787ec 00000001 00000002 b9b89aa0
[   15.765766] dd00: b9d95310 b9b23aa0 00000120 803d1ea4 00080001 6950c633 00000000 80d04d48
[   15.783011] dd20: 00000000 b6e9de08 b6e9dd48 00000001 b80be000 00000000 b6e9ddfc b6e9dd48
[   15.800270] dd40: 802bc48c 802ba188 b9d95310 b9b23b28 fcfb5025 0000000a b9746014 b6e9dd68
[   15.817514] dd60: b9d95310 b9b23990 b9b897d0 00000101 00000002 00000202 00000000 00000001
[   15.834793] dd80: 00000001 b6e9dd88 b9d95310 b9b23aa0 802b723c babb45d8 00000000 00000002
[   15.852075] dda0: b6c29b80 802aaca0 00000002 802aac84 00000000 b6e9dda8 b9fc1000 00000000
[   15.869355] ddc0: b9b89500 00000000 ffffff9c b6e9ddd8 8082d004 6950c633 b6e9de24 80d04d48
[   15.886611] dde0: 00000000 b9fc1000 b6c29b99 b6e9de60 b6e9de44 b6e9de00 802b148c 802bc418
[   15.903859] de00: 80d056a8 00000019 00020020 00000000 00000001 00000100 00000001 6950c633
[   15.921130] de20: b9fc1000 b815c880 b970d000 b6c29b99 b6e9de60 b80be000 b6e9de5c b6e9de48
[   15.938399] de40: 802b15e8 802b1414 80d04d48 b815c880 b6e9defc b6e9de60 8030a2c8 802b15c0
[   15.955662] de60: 803cbb64 803d1e78 b6e9deb4 b6e9de78 807f0d34 807f25fc 00000000 00000000
[   15.972937] de80: 00000000 b9d94c80 00000021 00000000 00000000 6950c633 00000000 b8f786f0
[   15.990235] dea0: b8f786f0 b6c29b80 b80be000 b6e9de60 0000016d 00000000 b6e9dee4 b6e9dec8
[   16.007510] dec0: 803cbfd0 803cbaf0 b8f786f0 6950c633 00000000 80d26d74 b970d000 80e16ae4
[   16.024789] dee0: 80e16ae4 fffffff8 80d23ed0 00000001 b6e9df2c b6e9df00 802b1e80 80309c78
[   16.042078] df00: b6cfa180 80d04d48 b6eb9e00 b970d000 00000000 000000bb 01986bf4 b9ddd000
[   16.059391] df20: b6e9df74 b6e9df30 802b31cc 802b1ddc 01986c0c b6cbe03c 000000bb b970d000
[   16.076731] df40: 00000000 6950c633 802bb53c 01986bf4 01986bd0 0002334c 0000000b 801011c4
[   16.094048] df60: b6e9c000 0000000b b6e9df8c b6e9df78 802b34ec 802b2ca8 00000000 00000000
[   16.111358] df80: b6e9dfa4 b6e9df90 802b3724 802b34bc 01986c0c 01986bd0 00000000 b6e9dfa8
[   16.128709] dfa0: 80101000 802b3704 01986c0c 01986bd0 01986c0c 01986bd0 01986bf4 239be800
[   16.146074] dfc0: 01986c0c 01986bd0 0002334c 0000000b 00038568 01986bf4 0002334c 00024414
[   16.163515] dfe0: 00035f80 7e94499c 00014b90 76eb59ac 60000010 01986c0c 00000000 00000000
[   16.181023] [<80297b6c>] (__kmalloc_track_caller) from [<806e6320>] (__kmalloc_reserve+0x34/0x80)
[   16.199421] [<806e6320>] (__kmalloc_reserve) from [<806e63fc>] (__alloc_skb+0x90/0x178)
[   16.216962] [<806e63fc>] (__alloc_skb) from [<8060763c>] (lan78xx_bh+0x1ac/0x81c)
[   16.233993] [<8060763c>] (lan78xx_bh) from [<80126cf4>] (tasklet_action_common+0x64/0xec)
[   16.251788] [<80126cf4>] (tasklet_action_common) from [<80126dd4>] (tasklet_action+0x28/0x30)
[   16.269954] [<80126dd4>] (tasklet_action) from [<80102410>] (__do_softirq+0x190/0x3f0)
[   16.287472] [<80102410>] (__do_softirq) from [<801266d0>] (do_softirq+0x74/0xcc)
[   16.304513] [<801266d0>] (do_softirq) from [<801267fc>] (__local_bh_enable_ip+0xd4/0xe4)
[   16.322304] [<801267fc>] (__local_bh_enable_ip) from [<8075d7f8>] (ip_finish_output2+0x230/0x470)
[   16.340991] [<8075d7f8>] (ip_finish_output2) from [<8075f4a4>] (ip_finish_output+0x144/0x20c)
[   16.359361] [<8075f4a4>] (ip_finish_output) from [<80760114>] (ip_output+0xd4/0x154)
[   16.376987] [<80760114>] (ip_output) from [<8075f78c>] (ip_local_out+0x48/0x4c)
[   16.394167] [<8075f78c>] (ip_local_out) from [<80760a8c>] (ip_send_skb+0x24/0xa4)
[   16.411509] [<80760a8c>] (ip_send_skb) from [<8078e638>] (udp_send_skb+0x12c/0x37c)
[   16.429006] [<8078e638>] (udp_send_skb) from [<80791388>] (udp_sendmsg+0x520/0x98c)
[   16.446507] [<80791388>] (udp_sendmsg) from [<8079c1c0>] (inet_sendmsg+0x48/0xe4)
[   16.463777] [<8079c1c0>] (inet_sendmsg) from [<806ddab8>] (sock_sendmsg+0x24/0x34)
[   16.481144] [<806ddab8>] (sock_sendmsg) from [<806ddbf4>] (kernel_sendmsg+0x40/0x48)
[   16.498663] [<806ddbf4>] (kernel_sendmsg) from [<807e78fc>] (xs_send_kvec+0xac/0xc8)
[   16.516172] [<807e78fc>] (xs_send_kvec) from [<807e799c>] (xs_sendpages+0x84/0x264)
[   16.533640] [<807e799c>] (xs_sendpages) from [<807e7e54>] (xs_udp_send_request+0x7c/0x140)
[   16.551653] [<807e7e54>] (xs_udp_send_request) from [<807e4a70>] (xprt_transmit+0x54/0x32c)
[   16.569744] [<807e4a70>] (xprt_transmit) from [<807e18a0>] (call_transmit+0x174/0x208)
[   16.587424] [<807e18a0>] (call_transmit) from [<807ef8c8>] (__rpc_execute+0x8c/0x3f4)
[   16.605029] [<807ef8c8>] (__rpc_execute) from [<807f0234>] (rpc_execute+0x84/0xfc)
[   16.622346] [<807f0234>] (rpc_execute) from [<807e0360>] (rpc_run_task+0x150/0x168)
[   16.639766] [<807e0360>] (rpc_run_task) from [<807e04bc>] (rpc_call_sync+0x68/0xf0)
[   16.657218] [<807e04bc>] (rpc_call_sync) from [<803e8b94>] (nfs3_rpc_wrapper.constprop.6+0x54/0xe4)
[   16.676141] [<803e8b94>] (nfs3_rpc_wrapper.constprop.6) from [<803e9b20>] (nfs3_proc_getattr+0x58/0x78)
[   16.695477] [<803e9b20>] (nfs3_proc_getattr) from [<803d22e0>] (__nfs_revalidate_inode+0xc8/0x29c)
[   16.714419] [<803d22e0>] (__nfs_revalidate_inode) from [<803ca118>] (nfs_lookup_verify_inode+0x4c/0xb0)
[   16.733836] [<803ca118>] (nfs_lookup_verify_inode) from [<803ccefc>] (nfs_lookup_revalidate+0x300/0x488)
[   16.753359] [<803ccefc>] (nfs_lookup_revalidate) from [<802b6e54>] (lookup_fast+0x2e0/0x2ec)
[   16.771862] [<802b6e54>] (lookup_fast) from [<802ba390>] (path_openat+0x214/0x1024)
[   16.789550] [<802ba390>] (path_openat) from [<802bc48c>] (do_filp_open+0x80/0xec)
[   16.807006] [<802bc48c>] (do_filp_open) from [<802b148c>] (do_open_execat+0x84/0x1ac)
[   16.824848] [<802b148c>] (do_open_execat) from [<802b15e8>] (open_exec+0x34/0x54)
[   16.842338] [<802b15e8>] (open_exec) from [<8030a2c8>] (load_elf_binary+0x65c/0x127c)
[   16.860172] [<8030a2c8>] (load_elf_binary) from [<802b1e80>] (search_binary_handler+0xb0/0x24c)
[   16.878953] [<802b1e80>] (search_binary_handler) from [<802b31cc>] (__do_execve_file+0x530/0x7d4)
[   16.897976] [<802b31cc>] (__do_execve_file) from [<802b34ec>] (do_execve+0x3c/0x44)
[   16.915757] [<802b34ec>] (do_execve) from [<802b3724>] (sys_execve+0x2c/0x30)
[   16.933028] [<802b3724>] (sys_execve) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[   16.950819] Exception stack(0xb6e9dfa8 to 0xb6e9dff0)
[   16.965933] dfa0:                   01986c0c 01986bd0 01986c0c 01986bd0 01986bf4 239be800
[   16.984330] dfc0: 01986c0c 01986bd0 0002334c 0000000b 00038568 01986bf4 0002334c 00024414
[   17.002689] dfe0: 00035f80 7e94499c 00014b90 76eb59ac
[   17.017813] Code: e7914002 e3540000 0a000067 e5963014 (e7943003) 
[   17.034151] ---[ end trace d3a36ee9aad6e446 ]---
[   17.049000] Kernel panic - not syncing: Fatal exception in interrupt
[   17.065539] CPU1: stopping
[   17.078322] CPU: 1 PID: 118 Comm: systemd-journal Tainted: G      D  C        4.19.57-v7+ #1244
[   17.097329] Hardware name: BCM2835
[   17.110824] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   17.128838] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   17.146409] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   17.164054] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   17.183342] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
[   17.202557] Exception stack(0xb9359ce0 to 0xb9359d28)
[   17.217893] 9ce0: 0000000b 00000000 00000021 00000000 bda9c0b0 b9359db8 b9359ec8 80d04d48
[   17.236473] 9d00: 809081e0 8021b44c 00000000 b9359d94 00000000 b9359d30 8021da1c 8021c2f8
[   17.255070] 9d20: 20000013 ffffffff
[   17.268769] [<801019bc>] (__irq_svc) from [<8021c2f8>] (___bpf_prog_run+0xcdc/0x14e4)
[   17.286977] [<8021c2f8>] (___bpf_prog_run) from [<8021da1c>] (__bpf_prog_run32+0x4c/0x68)
[   17.305567] [<8021da1c>] (__bpf_prog_run32) from [<801de320>] (seccomp_run_filters+0x70/0x16c)
[   17.324606] [<801de320>] (seccomp_run_filters) from [<801de7cc>] (__seccomp_filter+0x4c/0x27c)
[   17.343639] [<801de7cc>] (__seccomp_filter) from [<801df21c>] (__secure_computing+0x50/0x8c)
[   17.362513] [<801df21c>] (__secure_computing) from [<8010b6f4>] (syscall_trace_enter+0x104/0x190)
[   17.381874] [<8010b6f4>] (syscall_trace_enter) from [<80101160>] (__sys_trace+0xc/0x44)
[   17.400339] Exception stack(0xb9359fa8 to 0xb9359ff0)
[   17.415736] 9fa0:                   00000000 7ec729b0 004acfd4 00000000 00000424 01b48558
[   17.434399] 9fc0: 00000000 7ec729b0 76f94968 00000021 7ec70178 00000000 00000009 7ec701ec
[   17.453026] 9fe0: 004bfde4 7ec700c0 004a405c 76ec5b5c
[   17.468438] CPU2: stopping
[   17.481233] CPU: 2 PID: 146 Comm: systemd-udevd Tainted: G      D  C        4.19.57-v7+ #1244
[   17.500105] Hardware name: BCM2835
[   17.513685] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   17.531843] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   17.549548] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   17.567340] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   17.586759] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<80101d0c>] (__irq_usr+0x4c/0x60)
[   17.606111] Exception stack(0xb819dfb0 to 0xb819dff8)
[   17.621562] dfa0:                                     00c9cdf1 00558681 00000003 76f33000
[   17.640300] dfc0: 00000003 00c9cdf0 00c9cdf1 00564910 00558681 00000001 00caf2d8 7e8b9924
[   17.659025] dfe0: 0000014c 7e8b9880 0053196c 76e61cdc 60000010 ffffffff
[   17.676170] CPU3: stopping
[   17.689277] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D  C        4.19.57-v7+ #1244
[   17.707876] Hardware name: BCM2835
[   17.721707] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   17.740055] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   17.757901] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   17.775769] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   17.795207] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
[   17.814569] Exception stack(0xb9d51f38 to 0xb9d51f80)
[   17.829977] 1f20:                                                       80109a84 00000000
[   17.848620] 1f40: 40000093 40000093 80d04d70 b9d50000 80d04db8 00000008 80d8ed3e 410fd034
[   17.867310] 1f60: 00000000 b9d51f94 80d0517c b9d51f88 00000000 80109a88 40000013 ffffffff
[   17.885966] [<801019bc>] (__irq_svc) from [<80109a88>] (arch_cpu_idle+0x34/0x4c)
[   17.903841] [<80109a88>] (arch_cpu_idle) from [<80831eb4>] (default_idle_call+0x34/0x48)
[   17.922455] [<80831eb4>] (default_idle_call) from [<8015265c>] (do_idle+0xec/0x17c)
[   17.940612] [<8015265c>] (do_idle) from [<801529ac>] (cpu_startup_entry+0x28/0x2c)
[   17.958629] [<801529ac>] (cpu_startup_entry) from [<8010fd54>] (secondary_start_kernel+0x134/0x140)
[   17.978195] [<8010fd54>] (secondary_start_kernel) from [<0010270c>] (0x10270c)
[   17.995906] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[   17.991779] Unable to handle kernel paging request at virtual address 3ccb203c
[   18.010824] pgd = 82d99a41
[   18.025091] [3ccb203c] *pgd=00000000
[   18.040208] Internal error: Oops: 5 [#1] SMP ARM
[   18.056392] Modules linked in: bcm2835_codec(C) bcm2835_v4l2(C+) cfg80211(+) v4l2_mem2mem bcm2835_mmal_vchiq(C) rfkill videobuf2_dma_contig v4l2_common snd_bcm2835(C) videobuf2_vmalloc videobuf2_memops snd_pcm videobuf2_v4l2 snd_timer videobuf2_common snd videodev media vc_sm_cma(C) uio_pdrv_genirq uio fixed i2c_dev ip_tables x_tables ipv6
[   18.111658] CPU: 0 PID: 72 Comm: kworker/0:2 Tainted: G         C        4.19.57-v7+ #1244
[   18.132493] Hardware name: BCM2835
[   18.148268] Workqueue: events lan78xx_delayedwork
[   18.165373] PC is at dwc_otg_hcd_urb_dequeue+0x30/0x224
[   18.182985] LR is at dwc_otg_urb_dequeue+0x70/0xd8
[   18.200144] pc : [<80643cbc>]    lr : [<80647854>]    psr: 20000093
[   18.218930] sp : b9737cf0  ip : b9737d18  fp : b9737d14
[   18.236656] r10: b83b0c40  r9 : fffffffe  r8 : 00000000
[   18.254337] r7 : b9684000  r6 : b9684000  r5 : b91e1580  r4 : 3ccb2020
[   18.273338] r3 : b91e1594  r2 : fffffffe  r1 : b83b0c80  r0 : b9684000
[   18.292265] Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   18.311937] Control: 10c5383d  Table: 393f406a  DAC: 00000055
[   18.330157] Process kworker/0:2 (pid: 72, stack limit = 0x363ac59d)
[   18.348880] Stack: (0xb9737cf0 to 0xb9738000)
[   18.365618] 7ce0:                                     80d04d48 b91e1580 b965e400 b9684000
[   18.386192] 7d00: 00000000 fffffffe b9737d44 b9737d18 80647854 80643c98 a0000013 d1863631
[   18.406645] 7d20: b91e1580 b9658c00 b965e400 fffffffe b9737e18 00001388 b9737d6c b9737d48
[   18.426936] 7d40: 806184dc 806477f0 b9658c00 b91e1580 b9658c00 60000013 fffffffe b9737e18
[   18.447140] 7d60: b9737d8c b9737d70 806198bc 806184ac b91e1580 b91e158c 80d04d48 00000000
[   18.467332] 7d80: b9737dc4 b9737d90 8061ae44 80619868 80d04d48 b9737dc8 00000000 b9737e18
[   18.487354] 7da0: 00001388 d1863631 b9737dc4 b91e1580 80d04d48 b9737dc8 b9737e04 b9737dc8
[   18.507167] 7dc0: 8061b288 8061add8 00000000 00020002 b9737dd0 b9737dd0 00000000 d1863631
[   18.526855] 7de0: 80d04d48 b9658c00 00000000 00000000 00000000 00000001 b9737e4c b9737e08
[   18.546507] 7e00: 8061b370 8061b1b8 b9737e34 00000000 80000600 00000002 60000113 d1863631
[   18.566136] 7e20: b95d7698 00000002 00000000 b9658c00 ba354c00 00000000 b9d17580 00000000
[   18.585810] 7e40: b9737e84 b9737e50 8061c574 8061b2a8 00000000 00000002 00000000 00000000
[   18.605528] 7e60: 00001388 b9737e70 8061eb54 b9d1757c 80d04d48 b9d17500 b9737efc b9737e88
[   18.625060] 7e80: 80606918 8061c524 ba351e40 b803cb00 b8132c00 b8132c00 00000003 ba351e40
[   18.644426] 7ea0: b96c4b00 80d04d48 b9737f1c b9737eb8 8082c5f8 80c92ad8 8013bed8 ba351a40
[   18.663639] 7ec0: ba354c00 00000000 b9d17580 00000000 b9737efc d1863631 802018e8 b94ceb00
[   18.682818] 7ee0: b9d1757c ba351a40 ba354c00 00000000 b9737f34 b9737f00 8013beec 80606744
[   18.702002] 7f00: 40000193 ba351a40 ba351a40 ba351a40 b94ceb14 ba351a40 ba351a58 80d03d00
[   18.721078] 7f20: 00000008 b94ceb00 b9737f7c b9737f38 8013c230 8013bd88 b9737f5c b9737f48
[   18.740062] 7f40: 80832464 80d03d00 80d8ecfa b9736038 b9737f7c b9d07580 b94d4240 00000000
[   18.759021] 7f60: b94ceb00 8013c1d4 b9d0759c b9d31e74 b9737fac b9737f80 8014251c 8013c1e0
[   18.778000] 7f80: 80104378 b94d4240 801423e4 00000000 00000000 00000000 00000000 00000000
[   18.796996] 7fa0: 00000000 b9737fb0 801010ac 801423f0 00000000 00000000 00000000 00000000
[   18.815929] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   18.834876] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[   18.853770] [<80643cbc>] (dwc_otg_hcd_urb_dequeue) from [<80647854>] (dwc_otg_urb_dequeue+0x70/0xd8)
[   18.873733] [<80647854>] (dwc_otg_urb_dequeue) from [<806184dc>] (unlink1+0x3c/0x124)
[   18.892379] [<806184dc>] (unlink1) from [<806198bc>] (usb_hcd_unlink_urb+0x60/0x94)
[   18.910862] [<806198bc>] (usb_hcd_unlink_urb) from [<8061ae44>] (usb_kill_urb+0x78/0x104)
[   18.929944] [<8061ae44>] (usb_kill_urb) from [<8061b288>] (usb_start_wait_urb+0xdc/0xf0)
[   18.948589] [<8061b288>] (usb_start_wait_urb) from [<8061b370>] (usb_control_msg+0xd4/0x130)
[   18.967262] [<8061b370>] (usb_control_msg) from [<8061c574>] (usb_clear_halt+0x5c/0x7c)
[   18.985548] [<8061c574>] (usb_clear_halt) from [<80606918>] (lan78xx_delayedwork+0x1e0/0x500)
[   19.004467] [<80606918>] (lan78xx_delayedwork) from [<8013beec>] (process_one_work+0x170/0x458)
[   19.023619] [<8013beec>] (process_one_work) from [<8013c230>] (worker_thread+0x5c/0x5a4)
[   19.042185] [<8013c230>] (worker_thread) from [<8014251c>] (kthread+0x138/0x168)
[   19.060067] [<8014251c>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[   19.077521] Exception stack(0xb9737fb0 to 0xb9737ff8)
[   19.092504] 7fa0:                                     00000000 00000000 00000000 00000000
[   19.110677] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   19.128795] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   19.145257] Code: 0a00005c e5914004 e3540000 0a00005a (e594501c) 
[   19.161119] ---[ end trace c5d3497c2f34b489 ]---
[   14.106713] Unable to handle kernel paging request at virtual address f810aaa0
[   14.126061] pgd = 4fd3c64a
[   14.140622] [f810aaa0] *pgd=00000000
[   14.156006] Internal error: Oops: 5 [#1] SMP ARM
[   14.172415] Modules linked in: snd_pcm bcm2835_v4l2(C+) v4l2_mem2mem snd_timer v4l2_common bcm2835_mmal_vchiq(C) videobuf2_vmalloc snd videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media vc_sm_cma(C) uio_pdrv_genirq uio fixed i2c_dev ip_tables x_tables ipv6
[   14.223565] CPU: 0 PID: 150 Comm: systemd-udevd Tainted: G         C        4.19.57-v7+ #1244
[   14.244873] Hardware name: BCM2835
[   14.260854] PC is at __kmalloc_track_caller+0xac/0x288
[   14.278620] LR is at __kmalloc_track_caller+0x60/0x288
[   14.296280] pc : [<80297b6c>]    lr : [<80297b20>]    psr: a0000013
[   14.315184] sp : b82e5580  ip : b82e5580  fp : b82e55c4
[   14.333060] r10: ffffffff  r9 : b9c01b40  r8 : 806e63fc
[   14.350906] r7 : 00490220  r6 : b9c01b40  r5 : 00000180  r4 : f810aaa0
[   14.370073] r3 : 00000000  r2 : 80c97aa0  r1 : 396bd000  r0 : b82e5580
[   14.389200] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   14.408981] Control: 10c5383d  Table: 382e806a  DAC: 00000055
[   14.427326] Process systemd-udevd (pid: 150, stack limit = 0x6de1e732)
[   14.446518] Stack: (0xb82e5580 to 0xb82e6000)
[   14.463450] 5580: b9650000 80d04d48 b9d16000 00008952 b82e55ac b82e55a0 80126b88 b82e5603
[   14.484401] 55a0: 00480020 80d04d48 806e63fc 00000180 00480020 ffffffff b82e55ec b82e55c8
[   14.505156] 55c0: 806e6320 80297acc 8060763c b96509c0 000000a6 80d04d48 8060763c b9e42240
[   14.525776] 55e0: b82e5634 b82e55f0 806e63fc 806e62f8 b82e5603 b82e5600 802018e8 00000000
[   14.546235] 5600: 20000013 18d3b897 b9d16558 000000a6 00000001 b9d16558 00000001 b9d16500
[   14.566635] 5620: b9d16564 b9d16558 b82e567c b82e5638 8060763c 806e6378 80d04d48 b9d16538
[   14.586874] 5640: ba349298 00000000 00000006 18d3b897 b82e567c b9d16568 b9d1656c 00000000
[   14.606982] 5660: ba349298 00000000 00000006 00000040 b82e56a4 b82e5680 80126cf4 8060749c
[   14.626887] 5680: 80d03098 00000006 b82e4038 00000101 80d04d48 00000018 b82e56b4 b82e56a8
[   14.646642] 56a0: 80126dd4 80126c9c b82e571c b82e56b8 80102410 80126db8 b82e573c b82e56c8
[   14.666356] 56c0: 00000000 00400140 80d03d00 ffff9054 0000000a 80c94a00 80d8f278 80d04d70
[   14.686106] 56e0: 80c8c308 80d03080 b82e56b8 00000007 b9650000 60000093 b9650000 b9020200
[   14.705887] 5700: 0000009e 80d04d48 00000010 00000000 b82e5734 b82e5720 801266d0 8010228c
[   14.725738] 5720: b82e4000 00000200 b82e574c b82e5738 801267fc 80126668 18d3b897 00000000
[   14.745420] 5740: b82e5784 b82e5750 8075d7f8 80126734 8db2a8c0 18d3b897 782b26d9 b9650000
[   14.764927] 5760: 80d04d48 b97b09c0 00000000 80d7fc40 000005dc b9d16000 b82e57bc b82e5788
[   14.784309] 5780: 8075f4a4 8075d5d4 00000200 80d04d48 000004e9 b9650000 80d04d48 000004e9
[   14.803628] 57a0: 00000000 80d7fc40 b97b09c0 b9d16000 b82e580c b82e57c0 80760114 8075f36c
[   14.822979] 57c0: 8075f658 8075d544 807609c4 8070e998 b9f0b600 00000000 00000000 18d3b897
[   14.842235] 57e0: 8075e61c b9650000 b97b09c0 80d7fc40 0000007c 00000000 b82e5908 00000074
[   14.861376] 5800: b82e582c b82e5810 8075f78c 8076004c 00004ee7 b9650000 80d7fc40 b810b034
[   14.880502] 5820: b82e5844 b82e5830 80760a8c 8075f750 b9650000 b97b09c0 b82e586c b82e5848
[   14.899613] 5840: 8078e638 80760a74 b97b09c0 b9650000 b82e5908 80d04d48 b82e58e0 00000000
[   14.918739] 5860: b82e59b4 b82e5870 80791388 8078e518 0000007c 00000008 b82e58b8 b82e58b4
[   14.937895] 5880: b82e58e0 00004040 80d7fc40 8db2a8c0 00000000 00000000 00000108 8075f0c8
[   14.957081] 58a0: 00000000 00000000 b97b0c20 00000000 b9f31400 00000000 00000000 00000000
[   14.976279] 58c0: 00000000 00000000 8db2a8c0 00000000 00000000 ffff0000 00000000 00000000
[   14.995443] 58e0: 00000000 00000000 00000000 000005dc 0000007c 00000000 ffff0000 00000000
[   15.014523] 5900: 00000000 00000000 00000002 00000001 00000000 00110000 00000000 00000000
[   15.033523] 5920: 00000000 00000000 00000000 b9d1df38 e2b2a8c0 8db2a8c0 73030108 80201b60
[   15.052542] 5940: 000009e6 b9d1da00 b82e599c b82e5958 8014ddbc 80832434 00000000 a0000093
[   15.071226] 5960: b82e598c b82e5970 807ef3e0 18d3b897 b9195fe0 b9195284 b9eee100 b9c16400
[   15.089582] 5980: 00000000 18d3b897 b9f31400 b97b09c0 b9b9e540 00000000 18d3b897 00000000
[   15.107963] 59a0: b82e5ad8 00000000 b82e59d4 b82e59b8 8079c1c0 80790e74 bab8825c 00000000
[   15.126425] 59c0: b82e5a1c b9b9e540 b82e59e4 b82e59d8 806ddab8 8079c184 b82e5a04 b82e59e8
[   15.144864] 59e0: 806ddbf4 806ddaa0 00000074 ba374440 80d04d48 b905d804 b82e5a74 b82e5a08
[   15.163304] 5a00: 807e78fc 806ddbc0 00000074 006012c0 00000001 b905d804 00000074 b97a100c
[   15.181718] 5a20: 00000010 00000003 00000000 00000000 b82e5a1c 00000000 00000000 00000000
[   15.199894] 5a40: 00000000 00004040 00000000 18d3b897 b82e5a7c 00000001 00000074 b96eec04
[   15.217785] 5a60: b9b9e540 b9f0ba00 b82e5ac4 b82e5a78 807e799c 807e785c 00000000 00000000
[   15.235569] 5a80: b82e5ac4 b82e5a90 8019b964 806a7b48 29b1a68f 00000003 00000000 b96eec00
[   15.253323] 5aa0: 80d04d48 b97a1000 b96eec04 b9f0ba00 00000000 00000000 b82e5b04 b82e5ac8
[   15.271011] 5ac0: 807e7e54 807e7924 00000000 00000001 b82e5ad8 b97a123c 00000000 18d3b897
[   15.288701] 5ae0: b97a1000 b96eec00 00000080 b96eec74 b97a1268 a315e60e b82e5b44 b82e5b08
[   15.306386] 5b00: 807e4a70 807e7de4 b96eec04 00000000 00000000 b9f0ba00 b96eec00 b9f0ba00
[   15.324161] 5b20: b96eec00 00000080 b96eec00 02000000 a315e60e 807e172c b82e5b74 b82e5b48
[   15.341983] 5b40: 807e18a0 807e4a28 b82e5c64 b9f0ba00 b9f0ba00 b9f0ba00 00000000 b82e4038
[   15.359450] 5b60: 80d04d70 80d04d48 b82e5bbc b82e5b78 807ef8c8 807e1738 00000080 80d932b0
[   15.376520] 5b80: b82e5b9c b82e5b90 80168bb4 18d3b897 b82e5bbc b9f0ba00 b9f0ba00 00000000
[   15.393559] 5ba0: b9699200 80914ca4 b9707480 b97073c0 b82e5bdc b82e5bc0 807f0234 807ef848
[   15.410590] 5bc0: 00000002 b9f0ba00 b9699200 b82e5c04 b82e5bfc b82e5be0 807e0360 807f01bc
[   15.427642] 5be0: 00000000 b82e4000 80d04d48 b82e5c7c b82e5c3c b82e5c00 807e04bc 807e021c
[   15.444740] 5c00: 802018e8 00000000 b9699200 00000000 b82e5c7c 809567a8 00000000 00000000
[   15.461859] 5c20: 00000000 18d3b897 b82e4000 80dc40c0 b82e5c5c b82e5c40 803e8b94 807e0460
[   15.478985] 5c40: b8f18b48 80d04d48 b8f18c90 b97073c0 b82e5cb4 b82e5c60 803e8ea8 803e8b4c
[   15.496142] 5c60: 80d04b80 b8f18b48 b8e6d1c4 0000000d b9707240 b9707480 b97073c0 80914ca4
[   15.513346] 5c80: b82e5c64 b82e5c70 00000000 18d3b897 b8f18c90 00000000 b8e6d198 00000011
[   15.530620] 5ca0: 00000011 b9707480 b82e5cf4 b82e5cb8 803cd2e8 803e8e10 00000000 00000000
[   15.547890] 5cc0: b9dbd948 18d3b897 b82e5d1c b8e6d198 80d04d48 b8f18c90 b8eb0330 00000011
[   15.565130] 5ce0: b82e5e98 00000000 b82e5d3c b82e5cf8 802b6774 803cd1e8 b82e5d1c 8082d004
[   15.582353] 5d00: 00000000 b82e5d04 b82e5d04 18d3b897 8082d004 b8f18d10 b8eb0330 00000011
[   15.599583] 5d20: b82e5e98 b8eb0330 a96c3e06 00007fff b82e5d5c b82e5d40 802b688c 802b66e4
[   15.616790] 5d40: b82e5e90 80d04d48 00000000 00000003 b82e5da4 b82e5d60 802b9670 802b6858
[   15.633975] 5d60: 803cbfd0 803cbaf0 b8f18c90 00000081 00000000 18d3b897 b82e5da4 61c88647
[   15.651161] 5d80: 80d04d48 b9f8905c b82e5e90 b8eb0330 a96c3e06 00007fff b82e5dfc b82e5da8
[   15.668346] 5da0: 802b9910 802b9484 b82e5dd4 b82e5db8 00000000 0000000d 00020000 802b5e48
[   15.685503] 5dc0: b82e5e90 b9f89010 b82e5dfc 18d3b897 802b5e48 80d04d48 80d04d48 b82e5f50
[   15.702694] 5de0: b82e5f50 b8055f00 b82e5e90 00000142 b82e5e8c b82e5e00 802ba220 802b977c
[   15.719856] 5e00: 00000000 b82e5e08 b82e5db8 00000000 801b8f00 801682f0 b82e5e3c b82e4000
[   15.737052] 5e20: 80282ca0 b8246440 80d04d48 00000041 80d05688 8021d9dc b9f89000 00832628
[   15.754260] 5e40: 000a0000 00000000 00832628 76f4c968 00000142 40000028 76e7d464 18d3b897
[   15.771489] 5e60: ffffff9c 80d04d48 80d04d48 b82e5f50 b82e5e90 00000001 b82e4000 00000142
[   15.788724] 5e80: b82e5f44 b82e5e90 802bc48c 802ba188 b91de710 b8eb0330 8dd1d9e6 0000000d
[   15.805939] 5ea0: b9f8904e 006000c0 b91de710 b9b15990 b8f18c90 00000011 00000002 00000202
[   15.823169] 5ec0: 00000000 00000000 00000000 b82e5ed0 7ee350f8 18d3b897 000000c3 00000011
[   15.840362] 5ee0: b91ec400 b91ec418 00000000 00000400 000a0000 00000020 b82e5f34 b82e5f08
[   15.857550] 5f00: b9f89000 00000000 000a0000 00000000 ffffff9c ffffff9c b9f89000 18d3b897
[   15.874709] 5f20: b82e4000 00000011 80d04d48 ffffff9c b9f89000 801011c4 b82e5f94 b82e5f48
[   15.891912] 5f40: 802a7f08 802bc418 b82e4010 18d3b897 00020000 76f40000 00000004 00000100
[   15.909111] 5f60: 00000001 18d3b897 b82e5f84 00832628 76f4c968 00020000 00000142 801011c4
[   15.926294] 5f80: b82e4000 00000142 b82e5fa4 b82e5f98 802a8004 802a7dd4 00000000 b82e5fa8
[   15.943454] 5fa0: 80101198 802a7ff4 00832628 76f4c968 ffffff9c 00832628 000a0000 00000000
[   15.960630] 5fc0: 00832628 76f4c968 00020000 00000142 00832628 00821578 00835d28 7ee3617c
[   15.977846] 5fe0: 76f02000 7ee36118 76d9ed68 76e7d464 60000010 ffffff9c 00000000 00000000
[   15.995101] [<80297b6c>] (__kmalloc_track_caller) from [<806e6320>] (__kmalloc_reserve+0x34/0x80)
[   16.013119] [<806e6320>] (__kmalloc_reserve) from [<806e63fc>] (__alloc_skb+0x90/0x178)
[   16.030288] [<806e63fc>] (__alloc_skb) from [<8060763c>] (lan78xx_bh+0x1ac/0x81c)
[   16.046946] [<8060763c>] (lan78xx_bh) from [<80126cf4>] (tasklet_action_common+0x64/0xec)
[   16.064422] [<80126cf4>] (tasklet_action_common) from [<80126dd4>] (tasklet_action+0x28/0x30)
[   16.082360] [<80126dd4>] (tasklet_action) from [<80102410>] (__do_softirq+0x190/0x3f0)
[   16.099750] [<80102410>] (__do_softirq) from [<801266d0>] (do_softirq+0x74/0xcc)
[   16.116657] [<801266d0>] (do_softirq) from [<801267fc>] (__local_bh_enable_ip+0xd4/0xe4)
[   16.134301] [<801267fc>] (__local_bh_enable_ip) from [<8075d7f8>] (ip_finish_output2+0x230/0x470)
[   16.152775] [<8075d7f8>] (ip_finish_output2) from [<8075f4a4>] (ip_finish_output+0x144/0x20c)
[   16.170932] [<8075f4a4>] (ip_finish_output) from [<80760114>] (ip_output+0xd4/0x154)
[   16.188282] [<80760114>] (ip_output) from [<8075f78c>] (ip_local_out+0x48/0x4c)
[   16.205198] [<8075f78c>] (ip_local_out) from [<80760a8c>] (ip_send_skb+0x24/0xa4)
[   16.222330] [<80760a8c>] (ip_send_skb) from [<8078e638>] (udp_send_skb+0x12c/0x37c)
[   16.239642] [<8078e638>] (udp_send_skb) from [<80791388>] (udp_sendmsg+0x520/0x98c)
[   16.256968] [<80791388>] (udp_sendmsg) from [<8079c1c0>] (inet_sendmsg+0x48/0xe4)
[   16.274146] [<8079c1c0>] (inet_sendmsg) from [<806ddab8>] (sock_sendmsg+0x24/0x34)
[   16.291415] [<806ddab8>] (sock_sendmsg) from [<806ddbf4>] (kernel_sendmsg+0x40/0x48)
[   16.308835] [<806ddbf4>] (kernel_sendmsg) from [<807e78fc>] (xs_send_kvec+0xac/0xc8)
[   16.326253] [<807e78fc>] (xs_send_kvec) from [<807e799c>] (xs_sendpages+0x84/0x264)
[   16.343549] [<807e799c>] (xs_sendpages) from [<807e7e54>] (xs_udp_send_request+0x7c/0x140)
[   16.361441] [<807e7e54>] (xs_udp_send_request) from [<807e4a70>] (xprt_transmit+0x54/0x32c)
[   16.379410] [<807e4a70>] (xprt_transmit) from [<807e18a0>] (call_transmit+0x174/0x208)
[   16.396966] [<807e18a0>] (call_transmit) from [<807ef8c8>] (__rpc_execute+0x8c/0x3f4)
[   16.414369] [<807ef8c8>] (__rpc_execute) from [<807f0234>] (rpc_execute+0x84/0xfc)
[   16.431514] [<807f0234>] (rpc_execute) from [<807e0360>] (rpc_run_task+0x150/0x168)
[   16.448727] [<807e0360>] (rpc_run_task) from [<807e04bc>] (rpc_call_sync+0x68/0xf0)
[   16.465953] [<807e04bc>] (rpc_call_sync) from [<803e8b94>] (nfs3_rpc_wrapper.constprop.6+0x54/0xe4)
[   16.484683] [<803e8b94>] (nfs3_rpc_wrapper.constprop.6) from [<803e8ea8>] (nfs3_proc_lookup+0xa4/0x134)
[   16.503895] [<803e8ea8>] (nfs3_proc_lookup) from [<803cd2e8>] (nfs_lookup+0x10c/0x294)
[   16.521604] [<803cd2e8>] (nfs_lookup) from [<802b6774>] (__lookup_slow+0x9c/0x174)
[   16.538963] [<802b6774>] (__lookup_slow) from [<802b688c>] (lookup_slow+0x40/0x54)
[   16.556344] [<802b688c>] (lookup_slow) from [<802b9670>] (walk_component+0x1f8/0x2f8)
[   16.574001] [<802b9670>] (walk_component) from [<802b9910>] (link_path_walk+0x1a0/0x4d4)
[   16.591945] [<802b9910>] (link_path_walk) from [<802ba220>] (path_openat+0xa4/0x1024)
[   16.609584] [<802ba220>] (path_openat) from [<802bc48c>] (do_filp_open+0x80/0xec)
[   16.626854] [<802bc48c>] (do_filp_open) from [<802a7f08>] (do_sys_open+0x140/0x1f0)
[   16.644315] [<802a7f08>] (do_sys_open) from [<802a8004>] (sys_openat+0x1c/0x20)
[   16.661420] [<802a8004>] (sys_openat) from [<80101198>] (__sys_trace_return+0x0/0x10)
[   16.679054] Exception stack(0xb82e5fa8 to 0xb82e5ff0)
[   16.693851] 5fa0:                   00832628 76f4c968 ffffff9c 00832628 000a0000 00000000
[   16.711949] 5fc0: 00832628 76f4c968 00020000 00000142 00832628 00821578 00835d28 7ee3617c
[   16.730064] 5fe0: 76f02000 7ee36118 76d9ed68 76e7d464
[   16.744957] Code: e7914002 e3540000 0a000067 e5963014 (e7943003) 
[   16.760915] ---[ end trace dc9b8cd67b41e29c ]---
[   16.775336] Kernel panic - not syncing: Fatal exception in interrupt
[   16.791586] CPU1: stopping
[   16.804159] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D  C        4.19.57-v7+ #1244
[   16.822192] Hardware name: BCM2835
[   16.835440] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   16.853219] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   16.870567] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   16.887975] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   16.907017] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
[   16.926008] Exception stack(0xb9d4df38 to 0xb9d4df80)
[   16.941097] df20:                                                       80109a84 00000000
[   16.959416] df40: 40000093 40000093 80d04d70 b9d4c000 80d04db8 00000002 80d8ed3e 410fd034
[   16.977741] df60: 00000000 b9d4df94 80d0517c b9d4df88 00000000 80109a88 40000013 ffffffff
[   16.996071] [<801019bc>] (__irq_svc) from [<80109a88>] (arch_cpu_idle+0x34/0x4c)
[   17.013613] [<80109a88>] (arch_cpu_idle) from [<80831eb4>] (default_idle_call+0x34/0x48)
[   17.031896] [<80831eb4>] (default_idle_call) from [<8015265c>] (do_idle+0xec/0x17c)
[   17.049737] [<8015265c>] (do_idle) from [<801529ac>] (cpu_startup_entry+0x28/0x2c)
[   17.067441] [<801529ac>] (cpu_startup_entry) from [<8010fd54>] (secondary_start_kernel+0x134/0x140)
[   17.086686] [<8010fd54>] (secondary_start_kernel) from [<0010270c>] (0x10270c)
[   17.104081] CPU2: stopping
[   17.116647] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D  C        4.19.57-v7+ #1244
[   17.134684] Hardware name: BCM2835
[   17.147945] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   17.165736] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   17.183067] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   17.200400] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   17.219336] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
[   17.238242] Exception stack(0xb9d4ff38 to 0xb9d4ff80)
[   17.253207] ff20:                                                       80109a84 00000000
[   17.271449] ff40: 40000193 40000193 80d04d70 b9d4e000 80d04db8 00000004 80d8ed3e 410fd034
[   17.289732] ff60: 00000000 b9d4ff94 80d0517c b9d4ff88 00000000 80109a88 40000113 ffffffff
[   17.308030] [<801019bc>] (__irq_svc) from [<80109a88>] (arch_cpu_idle+0x34/0x4c)
[   17.325583] [<80109a88>] (arch_cpu_idle) from [<80831eb4>] (default_idle_call+0x34/0x48)
[   17.343835] [<80831eb4>] (default_idle_call) from [<8015265c>] (do_idle+0xec/0x17c)
[   17.361719] [<8015265c>] (do_idle) from [<801529ac>] (cpu_startup_entry+0x28/0x2c)
[   17.379508] [<801529ac>] (cpu_startup_entry) from [<8010fd54>] (secondary_start_kernel+0x134/0x140)
[   17.398868] [<8010fd54>] (secondary_start_kernel) from [<0010270c>] (0x10270c)
[   17.416400] CPU3: stopping
[   17.429242] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D  C        4.19.57-v7+ #1244
[   17.447559] Hardware name: BCM2835
[   17.461143] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
[   17.479300] [<8010d4b0>] (show_stack) from [<80814ea0>] (dump_stack+0xd4/0x118)
[   17.497007] [<80814ea0>] (dump_stack) from [<80110350>] (handle_IPI+0x388/0x3a8)
[   17.514770] [<80110350>] (handle_IPI) from [<801021f4>] (bcm2836_arm_irqchip_handle_irq+0xa0/0xa4)
[   17.534132] [<801021f4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
[   17.553382] Exception stack(0xb9d51f38 to 0xb9d51f80)
[   17.568669] 1f20:                                                       80109a84 00000000
[   17.587221] 1f40: 40000093 40000093 80d04d70 b9d50000 80d04db8 00000008 80d8ed3e 410fd034
[   17.605790] 1f60: 00000000 b9d51f94 80d0517c b9d51f88 00000000 80109a88 40000013 ffffffff
[   17.624356] [<801019bc>] (__irq_svc) from [<80109a88>] (arch_cpu_idle+0x34/0x4c)
[   17.642131] [<80109a88>] (arch_cpu_idle) from [<80831eb4>] (default_idle_call+0x34/0x48)
[   17.660640] [<80831eb4>] (default_idle_call) from [<8015265c>] (do_idle+0xec/0x17c)
[   17.678727] [<8015265c>] (do_idle) from [<801529ac>] (cpu_startup_entry+0x28/0x2c)
[   17.696727] [<801529ac>] (cpu_startup_entry) from [<8010fd54>] (secondary_start_kernel+0x134/0x140)
[   17.716281] [<8010fd54>] (secondary_start_kernel) from [<0010270c>] (0x10270c)
[   17.733992] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

Issue seems to be present with the 4.19.x kernels.
4.14.x works fine with same root file system.

@maxnet

This comment has been minimized.

Copy link
Contributor Author

commented Jul 12, 2019

Confirmed. It's broken. How can I revert to previous version?

If using piserver and "apt-get upgrade" gave you a 4.19.x kernel that bricked your system, use the following to downgrade to 4.14.x:

  • In Piserver GUI go to tab "software", select operating system, open chroot shell
cd /tmp
wget https://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-kernel_1.20190401-1_armhf.deb
dpkg -i r*kernel*.deb
@pelwell

This comment has been minimized.

Copy link
Contributor

commented Jul 12, 2019

This looks like memory corruption - something is probably writing off the end of a buffer, or continuing to write to it after our has been freed. @maxnet There are some kernel heap validation features of varying performance impact that can be enabled if you are prepared to build your own kernel.

Correction: "slub" debugging is a runtime option - try adding "slub_debug" to cmdline.txt.

@maxnet

This comment has been minimized.

Copy link
Contributor Author

commented Jul 12, 2019

@pelwell

slub_debug did not give anything useful.

But did notice that if for testing purposes I boot without modules (rename the /lib/modules folder), it does boot to X correctly.
So after that I have been playing a bit with blacklisting modules one by one. And the problem module seems to be bcm2835_v4l2

@maxnet

This comment has been minimized.

Copy link
Contributor Author

commented Jul 13, 2019

Two things I noticed.

  • With the default 3 seconds timeout the MMAL PORT_PARAMETER_GET command to query camera information times out (reply does arrive after all some seconds later)
[   16.692332] bcm2835_mmal_vchiq: >>> sync message type:COMPONENT_CREATE(4) length:160
[   16.709582] <<h: 00000000: 6c616d6d 00000004 00000000 00000001  mmal............
[   16.726864] <<h: 00000010: 00000000 b7cb18e0                    ........
[   16.743603] <<p: 00000000: 00000000 656d6163 695f6172 006f666e  ....camera_info.
[   16.760915] <<p: 00000010: 00000000 00000000 00000000 00000000  ................
[   16.778346] <<p: 00000020: 00000000 00000000 00000000 00000000  ................
[   16.795789] <<p: 00000030: 00000000 00000000 00000000 00000000  ................
[   16.813190] <<p: 00000040: 00000000 00000000 00000000 00000000  ................
[   16.830287] <<p: 00000050: 00000000 00000000 00000000 00000000  ................
[   16.847153] <<p: 00000060: 00000000 00000000 00000000 00000000  ................
[   16.864036] <<p: 00000070: 00000000 00000000 00000000 00000000  ................
[   16.880486] <<p: 00000080: 00000000 80159be8                    ........
[   16.901818] bcm2835_mmal_vchiq: <<< reply message type:COMPONENT_CREATE(4) length:44
[   16.918532] <<h: 00000000: 6c616d6d 00000004 3f2d6f04 00000001  mmal.....o-?....
[   16.934842] <<h: 00000010: 00000000 b7cb18e0                    ........
[   16.934862] <<p: 00000000: 00000000 00000040 00000000 00000000  ....@...........
[   16.966806] <<p: 00000010: 00000000                             ....
[   16.982145] bcm2835_mmal_vchiq: >>> sync message type:PORT_INFO_GET(8) length:36
[   16.997751] <<h: 00000000: 6c616d6d 00000008 00004003 00000001  mmal.....@......
[   17.013277] <<h: 00000010: 00000000 00000004                    ........
[   17.028356] <<p: 00000000: 00000040 00000001 00000000           @...........
[   17.028510] bcm2835_mmal_vchiq: <<< reply message type:PORT_INFO_GET(8) length:316
[   17.028531] <<h: 00000000: 6c616d6d 00000008 3f2d6f04 00000001  mmal.....o-?....
[   17.028544] <<h: 00000010: 00000000 00000004                    ........
[   17.028568] <<p: 00000000: 00000000 00000040 00000001 00000000  ....@...........
[   17.028586] <<p: 00000010: 00000000 00000000 3f4edae0 3f4edb80  ..........N?..N?
[   17.028603] <<p: 00000020: 00000001 00000000 00000000 3f4edbe0  ..............N?
[   17.028629] <<p: 00000030: 00000004 00000060 00000000 00000000  ....`...........
[   17.028646] <<p: 00000040: 00000000 00000004 00000060 3f4ed860  ........`...`.N?
[   17.028663] <<p: 00000050: 3f2db12c 00000000 00000001 00000000  ,.-?............
[   17.028678] <<p: 00000060: 00000000 3f4edc00 00000000 00000000  ......N?........
[   17.028694] <<p: 00000070: 00000000 00000000 00000000 00000000  ................
[   17.028710] <<p: 00000080: 00000000 00000000 00000000 00000000  ................
[   17.028732] <<p: 00000090: 00000000 00000000 00000000 00000000  ................
[   17.028758] <<p: 000000a0: 00000000 00000000 00000000 00000000  ................
[   17.028775] <<p: 000000b0: 00000000 00000000 00000000 00000000  ................
[   17.028791] <<p: 000000c0: 00000000 00000000 00000000 00000000  ................
[   17.028808] <<p: 000000d0: 00000000 00000000 00000000 00000000  ................
[   17.028824] <<p: 000000e0: 00000000 00000000 00000000 00000000  ................
[   17.028840] <<p: 000000f0: 00000000 00000000 00000000 00000000  ................
[   17.028857] <<p: 00000100: 00000000 00000000 00000000 00000000  ................
[   17.028877] <<p: 00000110: 00000000 00000000 00000000 00000000  ................
[   17.028887] <<p: 00000120: 00000000                             ....
[   17.034900] bcm2835_mmal_vchiq: >>> sync message type:PORT_PARAMETER_GET(15) length:40
[   17.378035] <<h: 00000000: 6c616d6d 0000000f 00000000 00000002  mmal............
[   17.378054] <<h: 00000010: 00000000 00000000                    ........
[   17.405463] <<p: 00000000: 00000040 00000000 0001001c 00000098  @...............
[   20.479224] bcm2835_mmal_vchiq: timed out waiting for sync completion
[   20.492217] Failed to get camera info
[   20.503996] bcm2835_mmal_vchiq: >>> sync message type:COMPONENT_DESTROY(5) length:28
[   20.518230] <<h: 00000000: 6c616d6d 00000005 ffffe000 00000002  mmal............
[   20.532081] <<h: 00000010: 00000000 80faf030                    ....0...
[   20.545115] <<p: 00000000: 00000040                             @...
[   23.411269] bcm2835_mmal_vchiq: <<< reply message type:PORT_PARAMETER_GET(15) length:180
[   23.425794] <<h: 00000000: 6c616d6d 0000000f 3f2d6f04 00000002  mmal.....o-?....
[   23.439509] <<h: 00000010: 00000000 00000000                    ........
[   23.452390] <<p: 00000000: 00000000 0001001c 00000098 00000000  ................
[   23.465959] <<p: 00000010: 00000002 00000000 00000000 00000000  ................
[   23.479494] <<p: 00000020: 00000000 00000000 00000000 00000000  ................
[   23.492986] <<p: 00000030: 00000000 00000000 00000000 00000000  ................
[   23.506385] <<p: 00000040: 00000000 00000000 00000000 00000000  ................
[   23.519777] <<p: 00000050: 00000000 00000000 00000000 00000000  ................
[   23.533017] <<p: 00000060: 00000000 00000000 00000000 00000000  ................
[   23.546100] <<p: 00000070: 00000000 00000000 00000000 00000000  ................
[   23.559110] <<p: 00000080: 00000000 00000000 00000000 00000000  ................
[   23.572023] <<p: 00000090: 00000000 00000001 00000001           ............
[   23.584560] bcm2835_mmal_vchiq: <<< reply message type:COMPONENT_DESTROY(5) length:28
[   23.598031] <<h: 00000000: 6c616d6d 00000005 3f2d6f04 00000002  mmal.....o-?....
[   23.598043] <<h: 00000010: 00000000 80faf030                    ....0...
[   23.598062] <<p: 00000000: 00000000                             ....

If I raise SYNC_MSG_TIMEOUT to say 10, it does not timeout.
But in both cases the system hangs (or panics) some seconds afterwards.

  • If I alter the kernel code so that the PORT_PARAMETER_GET command is not sent at all (but the other commands are), the system does not hang.
    Like there is something wrong with the firmware side implementation of that command.
    Does firmware also have the potential to corrupt memory or otherwise get the system to hang some seconds later if there is a bug there?
@kpfleming

This comment has been minimized.

Copy link

commented Jul 15, 2019

I noticed a number of errors with the bcm2835_v4l2 module as well when trying to boot the 4.19.57 kernel, and had to roll back to 4.19.42 to get a working system. I didn't have kernel crashes, but I'm also using nfsboot and the lan78xx driver failed to keep the link working after the bcm2835_v4l2 module failure messages (could be unrelated of course).

@6by9

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2019

We did have a memory corruption issue with bcm2835_v4l2 and PORT_PARAMETER_GET previously. Life got a little tortuous as upstream applied a different patch to "fix" it, so it's possible it's still partially broken.
I'll have a look.

I'm bemused that we get a timeout at 3 seconds - the VPU should be doing very little at that point. It is probing for the connected sensor, but that should take less time than that.

@maxnet

This comment has been minimized.

Copy link
Contributor Author

commented Jul 15, 2019

I'm bemused that we get a timeout at 3 seconds - the VPU should be doing very little at that point.

That also only happens when the Pi is being booted using a PXE network boot BTW.
If I put same kernel on SD card, even when having root file system still on NFS, the command does complete fast, and system boots properly.

[   17.087453] bcm2835_mmal_vchiq: >>> sync message type:PORT_PARAMETER_GET(15) length:40
[   17.087465] <<h: 00000000: 6c616d6d 0000000f 00000000 00000002  mmal............
[   17.087473] <<h: 00000010: 00000000 00000000                    ........
[   17.087483] <<p: 00000000: 00000040 00000000 0001001c 00000098  @...............
[   17.104555] bcm2835_mmal_vchiq: <<< reply message type:PORT_PARAMETER_GET(15) length:180
[   17.104586] <<h: 00000000: 6c616d6d 0000000f 3f2d6ec4 00000002  mmal.....n-?....
[   17.104604] <<h: 00000010: 00000000 00000000                    ........
[   17.104621] <<p: 00000000: 00000000 0001001c 00000098 00000000  ................
[   17.104639] <<p: 00000010: 00000002 00000000 00000000 00000000  ................
[   17.104656] <<p: 00000020: 00000000 00000000 00000000 00000000  ................
[   17.104673] <<p: 00000030: 00000000 00000000 00000000 00000000  ................
[   17.104688] <<p: 00000040: 00000000 00000000 00000000 00000000  ................
[   17.104706] <<p: 00000050: 00000000 00000000 00000000 00000000  ................
[   17.104725] <<p: 00000060: 00000000 00000000 00000000 00000000  ................
[   17.104742] <<p: 00000070: 00000000 00000000 00000000 00000000  ................
[   17.104761] <<p: 00000080: 00000000 00000000 00000000 00000000  ................
[   17.104776] <<p: 00000090: 00000000 00000001 00000001           ............
@kpfleming

This comment has been minimized.

Copy link

commented Jul 15, 2019

Just a note: Pis do not support PXE, at all. They can boot off of a network (I have four that do), but they don't use PXE and it's really unfortunate that the community has adopted this terminology. PXE is a specific protocol and configuration, it's not just another name for 'network boot' :-)

@maxnet

This comment has been minimized.

Copy link
Contributor Author

commented Jul 15, 2019

Just a note: Pis do not support PXE, at all.

If a device spits out DHCP packets that say "PXEClient" I call it a PXE boot.
Regardless of the level of protocol conformity.

@kpfleming

This comment has been minimized.

Copy link

commented Jul 15, 2019

Indeed, a decent part of the blame falls on the RPi team which decided to use PXEClient as an identifier in the DHCP packets for a non-PXE device. In my network I don't test for it, and don't include it in responses back to the devices (and I don't even check for the magic vendor identifier either) and it all works fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.